lighteval icon indicating copy to clipboard operation
lighteval copied to clipboard
verified publisher icon huggingface

[BUG] LightevalTaskConfig doesn't support comparison, but Cache management uses sorted on it

Open colinzuo opened this issue 3 months ago • 2 comments

Describe the bug

Specify two tasks as below: helm|boolq|0 helm|boolq:contrastset|0 The current logic will assume "helm|boolq" is a group and will add "helm|boolq:contrastset" again, then "helm|boolq:contrastset" will appear twice, later at SampleCache:_get_task_hash sorted is used to sort these two, and an error "<" is not supported on LightevalTaskConfig is raised

To Reproduce

Specify two tasks as below in yaml: helm|boolq|0 helm|boolq:contrastset|0

Expected behavior

Run should succeed

Version info

0.11.0

colinzuo avatar Oct 21 '25 08:10 colinzuo

Oh cool edge case, good catch! Here we would actually rename boolq to boolq:default so that a task does not also carry a group name.

clefourrier avatar Oct 21 '25 09:10 clefourrier

How about sorted(self.registry.task_to_configs[f"{task_suite}|{task_name}"]), currently comparison operator is not defined for LightevalTaskConfig, so this place will fail if more than one config is met, any thought?

colinzuo avatar Oct 21 '25 09:10 colinzuo