colinzuo

Fixes #1023: add custom processing logic for MetricGrouping

2

add another level of loop to dict fields in MetricGrouping instance

[BUG] Cache management failed for tasks using sample_params to update function attrs

1

## Describe the bug Metric:__call__ used "&".join(f"{k}={v}" for k, v in sample_params.items()) to update metric name, but the value might be function, which will change for each run. ## To...

[BUG] Cache management failed for tasks using MetricGrouping

2

## Describe the bug SampleCache:_get_task_hash uses LightevalTaskConfig:__str__ to compute task hash, but LightevalTaskConfig:__str__ doesn't handle tasks using MetricGrouping correctly, for MetricGrouping field corpus_level_fn is a dict, so another level of...

[BUG] LightevalTaskConfig doesn't support comparison, but Cache management uses sorted on it

2

## Describe the bug Specify two tasks as below: helm|boolq|0 helm|boolq:contrastset|0 The current logic will assume "helm|boolq" is a group and will add "helm|boolq:contrastset" again, then "helm|boolq:contrastset" will appear twice,...