load_dataset("livecodebench/code_generation_lite", version_tag="release_v2") TypeError: 'NoneType' object is not callable

[dshwei]

Describe the bug

from datasets import load_dataset lcb_codegen = load_dataset("livecodebench/code_generation_lite", version_tag="release_v2")

or configs = get_dataset_config_names("livecodebench/code_generation_lite", trust_remote_code=True)

both error: Traceback (most recent call last): File "", line 1, in File "/workspace/miniconda/envs/grpo/lib/python3.10/site-packages/datasets/load.py", line 2131, in load_dataset builder_instance = load_dataset_builder( File "/workspace/miniconda/envs/grpo/lib/python3.10/site-packages/datasets/load.py", line 1888, in load_dataset_builder builder_instance: DatasetBuilder = builder_cls( TypeError: 'NoneType' object is not callable

Steps to reproduce the bug

from datasets import get_dataset_config_names

configs = get_dataset_config_names("livecodebench/code_generation_lite", trust_remote_code=True) OR lcb_codegen = load_dataset("livecodebench/code_generation_lite", version_tag="release_v2")

Expected behavior

load datasets livecodebench/code_generation_lite

Environment info

import datasets version '3.3.2'

[zwxandy]

datasets

Hi, have you solved this bug? Today I also met the same problem about livecodebench/code_generation_lite when evaluating the Open-R1 repo. I am looking forward to your reply!

[Serzhanov]

Hey guys, I tried to reproduce the issue and it works fine. I used google colab as enviroment.

[zwxandy]

Hey guys, I tried to reproduce the issue and it works fine. I used google colab as enviroment.

Thanks for your kind reply! I wonder which Python version do you use? My Python version is 3.11.11 and datasets version is 3.3.2 but I still met this bug.

[Serzhanov]

@zwxandy It's Python 3.11.11

[FightingZhen]

@Serzhanov @zwxandy I have met the same problem, have this problem be solved?

[FightingZhen]

@Serzhanov @zwxandy I have met the same problem, have this problem be solved?

I try to downgrade datasets version to 2.20.0，and it works for me @Serzhanov @dshwei , hope this work for you too :)

[yfanDai]

datasets

Hi, have you solved this bug? Today I also met the same problem about livecodebench/code_generation_lite when evaluating the Open-R1 repo. I am looking forward to your reply!

Hi, have you resolved this problem? I meet the same bug when evaluating the ’Open-R1’, too. Looking forward to your reply!

[zwxandy]

@Serzhanov @zwxandy I have met the same problem, have this problem be solved?

I try to downgrade datasets version to 2.20.0，and it works for me @Serzhanov @dshwei , hope this work for you too :)

I still met the same bug after downgrading datasets version to 2.20.0. Moreover, it is not friendly to Open-R1 since there can be another bug: open-r1 0.1.0.dev0 requires datasets>=3.2.0 with datasets==2.20.0

[zwxandy]

datasets

Hi, have you solved this bug? Today I also met the same problem about livecodebench/code_generation_lite when evaluating the Open-R1 repo. I am looking forward to your reply!

Hi, have you resolved this problem? I meet the same bug when evaluating the ’Open-R1’, too. Looking forward to your reply!

Hi, I still cannot solve this bug introduced from datasets version. Downgrading datasets version to 2.20.0 cannot work for me and it introduces another problem open-r1 0.1.0.dev0 requires datasets>=3.2.0 in Open-R1.

Luckily, there is a tricky way to enable you to run Open-R1. You can remove or comment the code related to lcb in ~/anaconda3/envs/openr1/lib/python3.11/site-packages/lighteval/tasks/extended/__init__.py. I have reproduce the results of DeepSeek-R1-Distill-Qwen-1.5B and 7B on MATH-500, GPQA, and AIME24.

You can have a try~

[qingfengyuhuoda]

The issue was resolved . because the filelivecodebench/code_generation_lite/code_generation_lite.pywas not downloaded. Manually downloading it fixed the problem.