worldbench
Corruption JSON files solely have question_type": "robust_qas"
In https://huggingface.co/datasets/drive-bench/arena
the json file corresponding to corruptions only have the question_type: robust_qas
This is a serious issue as the eval.py expects these question types: self.results = { "perception": { "MCQ": {"gpt": [], "accuracy": []}, "VQA": {"gpt": [], "language": []} }, "prediction": { "VQA": {"gpt": [], "language": []} }, "planning": { "VQA": {"gpt": [], "language": []} }, "behavior": { "MCQ": {"gpt": [], "accuracy": []}} }
The json files for corruptions need to be updated.
There are two ways to use the corrupted images.
- Corrupted images with original text input. Corresponding to all the other numbers reported in the paper, except Table 5.
- Corrupted image with specific text input for each corruption (the screenshot here), corresponding to Table 5 in the paper.
The current evaluation script only supports the first one. We will provide the script to support the second one soon.
For the 1st way, "Corrupted images with original text input. Corresponding to all the other numbers reported in the paper, except Table 5.". In drivebench-test.json, "image_path" point to the uncorrupted images, so do I need to modify drivebench-test.json manually to point to the corrupted images?
Yes, since those questions are also used for clean answers evaluated in the paper, we use the original image path by default. The path change is also in our released script.
By "The path change is also in our released script.", which script are you referring to? Could you give step-by-step instructions on how to run the 1st way, "Corrupted images with original text input. Corresponding to all the other numbers reported in the paper, except Table 5."
Sorry for the confusion. You can refer to the code here. Please let me know if you have further questions!