opencompass
opencompass copied to clipboard
[Bug] LongBench v2 dataset config is incorrect
Prerequisite
- [x] I have searched Issues and Discussions but cannot get the expected help.
- [x] The bug has not been fixed in the latest version.
Type
I'm evaluating with the officially supported tasks/models/datasets.
Environment
Not relevant
Reproduces the problem - code/configuration sample
opencompass:https://github.com/open-compass/opencompass/blob/main/opencompass/configs/datasets/longbenchv2/longbenchv2_gen_75fbba.py#L19
prompt='Please read the following text and answer the questions below.\n <text> \n {context} \n </text> \n \n What is the correct answer to this question: {question} \n \n Choices: \n (A) {choice_A} \n (B) {choice_B} \n (C) {choice_C} \n (D) {choice_D} \n Let’s think step by step. Based on the above, what is the single, most likely answer choice? Format your response as follows: "The correct answer is (insert answer here)'
longbenchv2:https://github.com/THUDM/LongBench/blob/main/prompts/0shot_cot_ans.txt
Please read the following text and answer the questions below.
The text is too long and omitted here.
What is the correct answer to this question: $Q$
Choices:
(A) $C_A$
(B) $C_B$
(C) $C_C$
(D) $C_D$
Let’s think step by step: $COT$
Based on the above, what is the single, most likely answer choice? Format your response as follows: "The correct answer is (insert answer here)".
OpenCompasse code Missing “.
Reproduces the problem - command or script
Not relevant
Reproduces the problem - error message
Not relevant
Other information
No response