opencompass icon indicating copy to clipboard operation
opencompass copied to clipboard

[Bug] piqa or lambada dataset is not available

Open msz12345 opened this issue 7 months ago • 0 comments

Prerequisite

Type

I'm evaluating with the officially supported tasks/models/datasets.

Environment

I use python run.py --datasets piqa_gen --hf-type base --hf-path /root/autodl-tmp/deepseek-Qwen-1.5B --debug but get error other dataset like aime or math or HellaSwag or WinoGrande is avaliable, piqa and lambada dataset get error

Reproduces the problem - code/configuration sample

python run.py --datasets piqa_gen --hf-type base --hf-path /root/autodl-tmp/deepseek-Qwen-1.5B --debug

Reproduces the problem - command or script

python run.py --datasets piqa_gen --hf-type base --hf-path /root/autodl-tmp/deepseek-Qwen-1.5B --debug

Reproduces the problem - error message

(quarot) root@autodl-container-fee8478cb9-083f90fa:~/autodl-tmp/opencompass# python run.py --datasets piqa_gen --hf-type base --hf-path /root/autodl-tmp/deepseek-Qwen-1.5B --debug --batch-size 50

07/08 18:19:02 - OpenCompass - INFO - Loading piqa_gen: /root/autodl-tmp/opencompass/opencompass/configs/./datasets/piqa/piqa_gen.py 07/08 18:19:02 - OpenCompass - INFO - Loading example: /root/autodl-tmp/opencompass/opencompass/configs/./summarizers/example.py 07/08 18:19:02 - OpenCompass - INFO - Current exp folder: outputs/default/20250708_181902 07/08 18:19:02 - OpenCompass - WARNING - SlurmRunner is not used, so the partition argument is ignored. 07/08 18:19:02 - OpenCompass - INFO - ./data/piqa does not exist!Start Download data automatically!If you have downloaded the data before,You can specific COMPASS_DATA_CACHE to avoid downloading~ Traceback (most recent call last): File "/root/autodl-tmp/opencompass/run.py", line 5, in main() File "/root/autodl-tmp/opencompass/opencompass/cli/main.py", line 330, in main tasks = partitioner(cfg) File "/root/autodl-tmp/opencompass/opencompass/partitioners/base.py", line 83, in call tasks = self.partition(**model_and_dataset_args, File "/root/autodl-tmp/opencompass/opencompass/partitioners/num_worker.py", line 69, in partition dataset_size = self.get_size(dataset) File "/root/autodl-tmp/opencompass/opencompass/partitioners/num_worker.py", line 141, in get_size dataset = build_dataset_from_cfg(dataset) File "/root/autodl-tmp/opencompass/opencompass/utils/build.py", line 12, in build_dataset_from_cfg return LOAD_DATASET.build(dataset_cfg) File "/root/miniconda3/envs/quarot/lib/python3.10/site-packages/mmengine/registry/registry.py", line 570, in build return self.build_func(cfg, *args, **kwargs, registry=self) File "/root/miniconda3/envs/quarot/lib/python3.10/site-packages/mmengine/registry/build_functions.py", line 121, in build_from_cfg obj = obj_cls(**args) # type: ignore File "/root/autodl-tmp/opencompass/opencompass/datasets/base.py", line 17, in init dataset = self.load(**kwargs) File "/root/autodl-tmp/opencompass/opencompass/datasets/piqa.py", line 83, in load path = get_data_path(path) File "/root/autodl-tmp/opencompass/opencompass/utils/datasets.py", line 52, in get_data_path return download_dataset(local_path, cache_dir) File "/root/autodl-tmp/opencompass/opencompass/utils/datasets.py", line 90, in download_dataset assert dataset_name, f'No valid url for {data_path}!\n' +
AssertionError: No valid url for ./data/piqa! Please make sure ./data/piqa is correct

Other information

No response

msz12345 avatar Jul 08 '25 10:07 msz12345