lighteval
lighteval copied to clipboard
[BUG] Cache management failed for tasks using sample_params to update function attrs
Describe the bug
Metric:call used "&".join(f"{k}={v}" for k, v in sample_params.items()) to update metric name, but the value might be function, which will change for each run.
To Reproduce
Task that use sample_params, such as: leaderboard:gsm8k:0
metrics=[
Metrics.exact_match(sample_params={"normalize_gold": gsm8k_normalizer, "normalize_pred": gsm8k_normalizer})
],
Expected behavior
Cache should work
Version info
main branch
hey! thanks for the issue. We won't have time to focus on this for now as we are integrating inspect-ai for cache!