lighteval icon indicating copy to clipboard operation
lighteval copied to clipboard

[BUG] Cache management failed for tasks using sample_params to update function attrs

Open colinzuo opened this issue 3 months ago • 1 comments

Describe the bug

Metric:call used "&".join(f"{k}={v}" for k, v in sample_params.items()) to update metric name, but the value might be function, which will change for each run.

To Reproduce

Task that use sample_params, such as: leaderboard:gsm8k:0

metrics=[
    Metrics.exact_match(sample_params={"normalize_gold": gsm8k_normalizer, "normalize_pred": gsm8k_normalizer})
],

Expected behavior

Cache should work

Version info

main branch

colinzuo avatar Oct 22 '25 06:10 colinzuo

hey! thanks for the issue. We won't have time to focus on this for now as we are integrating inspect-ai for cache!

NathanHB avatar Nov 04 '25 11:11 NathanHB