lighteval
lighteval copied to clipboard
[BUG] Cache management failed for tasks using MetricGrouping
Describe the bug
SampleCache:_get_task_hash uses LightevalTaskConfig:str to compute task hash, but LightevalTaskConfig:str doesn't handle tasks using MetricGrouping correctly, for MetricGrouping field corpus_level_fn is a dict, so another level of loop is needed.
To Reproduce
Cache doesn't work for "leaderboard|truthfulqa:mc|0"
Expected behavior
Cache should work
Version info
0.11.0
Hm this one could be fixed fast by adding an __str__ method to the MetricGrouping I think - do you think you'd have the time for a PR?
Sure, let me have a try