colinzuo

As @mrjbq7 guessed, it's indeed caused by using TA_IS_ZERO, which is defined as #define TA_IS_ZERO(v) (((-0.00000001)

Sure, let me have a try

There is code related MMLU-Pro in Pr 835, but it was merged to branch model_release, not sure why. https://github.com/huggingface/lighteval/pull/835 https://github.com/huggingface/lighteval/tree/model_release/src/lighteval/tasks/extended

In LightevalTaskConfig:__str__, asdict(self) changed MetricGrouping to dict, so > adding an __str__ method to the MetricGrouping won't work To stay consistent with the existing code, I added another level of...

How about **sorted**(self.registry.task_to_configs[f"{task_suite}|{task_name}"]), currently comparison operator is not defined for LightevalTaskConfig, so this place will fail if more than one config is met, any thought?