lighteval icon indicating copy to clipboard operation
lighteval copied to clipboard

Add `ligheval --list-tasks`

Open clefourrier opened this issue 1 year ago • 4 comments

This command should list all the available tasks (in extended tasks and the task table)

clefourrier avatar Feb 26 '24 10:02 clefourrier

Hello,

I would like to work on this. Do you need special formatting for the list?

DimbyTa avatar Mar 30 '24 22:03 DimbyTa

Hi, thanks for your interest! I think providing the tasks grouped by suite would be nice, for example

- lighteval: all the tasks available, sorted alphabetically
- community: all the tasks available, sorted alphabetically
...

clefourrier avatar Apr 02 '24 07:04 clefourrier

We have a lot of tasks so it would be better to have them in a list in that format, this makes reading easier as you can ligheval --list-tasks | grep mmlu for example.

- lighteval:
    - task 1
    - task 2
- community:
    - task 1 
    - task 2

NathanHB avatar Apr 02 '24 09:04 NathanHB

I understand. I'm working on it now.

DimbyTa avatar Apr 02 '24 16:04 DimbyTa