lighteval
lighteval copied to clipboard
Adds RULER benchmark
https://github.com/NVIDIA/RULER
Available context size: 4096, 8192, 16384, 32768, 65536, 131072
uv run lighteval vllm "model_name=meta-llama/Llama-3.1-8B,dtype=bfloat16,max_model_length=131072" "lighteval|ruler_{context size}|0|0"
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Has this PR been abandoned? Is there any plan to include RULER in lighteval?
hi @Jeronymous ! I answered on #1001