lighteval
lighteval copied to clipboard
Initial proposal for model lazy loading
This is a first try for issue #496.
However, we need the docs which currently in turn depend on the model being initialized. The model itself is not actually needed, but the tokenizer would be enough (in lighteval_task.create_requests_from_tasks).