DriveBench

DriveBench copied to clipboard

Hi, thank you for making this project available! I’ve been encountering versioning issues with the Python packages needed here when trying to use this codebase, which has limited my ability...

jcboone

Google Drive scripts: incorrect iteration

2

Observe the structure of an entry in the dataset for a json file: https://huggingface.co/datasets/drive-bench/arena [ { "scene_token": "da41ecbc644b4915b84bb732e35ebf8c", "frame_token": "7e4c3282bc2a4402b5d1d6705f9eb844", "question_type": "robust_qas", "question": "What is the current type of corruption?:...

SM20sam

Run Inference Using Oryx

1

Hello, may you please provide inference script for Oryx or at least the version of transformers you are using? Receiving this error when attempting to run inference using Oryx: Value...

SM20sam

Request for Finetuning Code

1

In your paper, you mention in 4.2.3. Fine-Tuned VLM Models, "Regarding DriveLM [59], we further investigate how the model benefits from in-distribution fine-tuning in the following section". Are the fine...

SM20sam

Corruption JSON files solely have question_type": "robust_qas"

5

In https://huggingface.co/datasets/drive-bench/arena the json file corresponding to corruptions only have the question_type: robust_qas This is a serious issue as the eval.py expects these question types: self.results = { "perception": {...

SM20sam

Qwen2-VL-7B MCQ accuracy on perception cannot be reproduced.

5

I used Qwen2-VL-7B-Instruct to evaluate perception MCQs. The model output "going ahead" for most questions, making the accuracy ~50%, while the score reported in the paper is 59%. Did you...

xuan-li

Issue: Ray Library Fails to Handle Irregular Paths with Missing Views

2

## Description The Ray library is unable to handle irregular paths that contain missing views (e.g., six-view paths with missing perspectives). This results in a runtime error during processing, specifically...

curryqka

Issue: Inference Script's `--data` Argument Should Support Dynamic Input Lists

1

## Description The current inference script (`inference/llava1.5.py`) uses a fixed JSON file path for the `--data` argument. However, this argument should ideally support dynamic input lists, such as a collection...

curryqka

Issue: eval.py Does Not Support All Question Types for Robutness Evaluation

1

## Description The `eval.py` script currently only supports specific question types (perception, prediction, and planning) for fog evaluation. The `self.results` dictionary only includes these categories, which causes issues when evaluating...

curryqka

ActorDiedError Caused by ValueError in VLLM Initialization

2

**Description**: I encountered an `ActorDiedError` exception when running my Ray-based application. The error occurs during the initialization of the `LLM` class from the `vllm` library, specifically when configuring the model....

curryqka