lighteval icon indicating copy to clipboard operation
lighteval copied to clipboard
verified publisher icon huggingface

How to evaluate MMLU-Pro

Open qhz991029 opened this issue 3 months ago • 2 comments

Hi,

Thank you for the wonderful work!

I just want to ask how to perform the evaluation on MMLU-Pro, as I don't see any related code besides the README.

qhz991029 avatar Oct 24 '25 20:10 qhz991029

There is code related MMLU-Pro in Pr 835, but it was merged to branch model_release, not sure why.

https://github.com/huggingface/lighteval/pull/835

https://github.com/huggingface/lighteval/tree/model_release/src/lighteval/tasks/extended

colinzuo avatar Oct 25 '25 02:10 colinzuo

hey ! we are adding mmlu_pro in the main tasks here #1031

NathanHB avatar Nov 04 '25 10:11 NathanHB