lighteval icon indicating copy to clipboard operation
lighteval copied to clipboard

launch lighteval using `lighteval --args`

Open NathanHB opened this issue 1 year ago • 0 comments

  • removes uneeded script files to launch lighteval
  • moves parsers function to lighteval.commands.parsers
  • creates a lighteval cli executable with 3 subcommands (list-tasks, nanotron, accelerate)

how to use:

with nanotron

nothing changes except for the use of the cli executable

 lighteval nanotron  {nanotron args}

with accelerate

accelerate args now need to be passed to the lighteval script.

accelerate launch --multi_gpu --num_processes=8 -m \
lighteval accelerate \
--model_args "pretrained=gpt2,dtype=float16" \
--tasks "original|mmlu:anatomy|0|0" \
--override_batch_size 1  \
--output_dir="tmp/" \
--max_samples 1 \
--use_chat_template

NathanHB avatar Apr 10 '24 08:04 NathanHB