AgentBench icon indicating copy to clipboard operation
AgentBench copied to clipboard

[Assistance] Number of problems in the OS dataset

Open deema-A opened this issue 2 years ago • 2 comments

Hi, I have counted the number of data samples or problems in the 'os_interaction' folder, and my count shows a total of 191 samples. However, the table that provides statistics reports a different number of samples, specifically 170 samples. Not sure if I was looking at the correct folder. Appreciate your help. thanks!

deema-A avatar Nov 08 '23 20:11 deema-A

Thank you for your interest.

  1. Could you provide the detail of the split whose count is wrong?
  2. File data/os_interaction/data/6-backup.json is deprecated and we don't contain it in our dataset. Details are shown in main/configs/tasks/os.yaml.

Longin-Yu avatar Nov 09 '23 11:11 Longin-Yu

thank you @Longin-Yu apart form the 6-backup the total is 182, including 26 dev. test 156 dev 26 but the stat here shows the test with 144 samples? Screenshot 2023-11-09 at 7 58 18 AM

deema-A avatar Nov 09 '23 13:11 deema-A