lighteval icon indicating copy to clipboard operation
lighteval copied to clipboard

[BUG] Cache management failed for tasks using MetricGrouping

Open colinzuo opened this issue 3 months ago • 2 comments

Describe the bug

SampleCache:_get_task_hash uses LightevalTaskConfig:str to compute task hash, but LightevalTaskConfig:str doesn't handle tasks using MetricGrouping correctly, for MetricGrouping field corpus_level_fn is a dict, so another level of loop is needed.

To Reproduce

Cache doesn't work for "leaderboard|truthfulqa:mc|0"

Expected behavior

Cache should work

Version info

0.11.0

colinzuo avatar Oct 21 '25 08:10 colinzuo

Hm this one could be fixed fast by adding an __str__ method to the MetricGrouping I think - do you think you'd have the time for a PR?

clefourrier avatar Oct 21 '25 09:10 clefourrier

Sure, let me have a try

colinzuo avatar Oct 21 '25 09:10 colinzuo