lighteval
lighteval copied to clipboard
launch lighteval using `lighteval --args`
- removes uneeded script files to launch lighteval
- moves parsers function to
lighteval.commands.parsers - creates a lighteval cli executable with 3 subcommands (list-tasks, nanotron, accelerate)
how to use:
with nanotron
nothing changes except for the use of the cli executable
lighteval nanotron {nanotron args}
with accelerate
accelerate args now need to be passed to the lighteval script.
accelerate launch --multi_gpu --num_processes=8 -m \
lighteval accelerate \
--model_args "pretrained=gpt2,dtype=float16" \
--tasks "original|mmlu:anatomy|0|0" \
--override_batch_size 1 \
--output_dir="tmp/" \
--max_samples 1 \
--use_chat_template