opencompass icon indicating copy to clipboard operation
opencompass copied to clipboard

[Bug] LongBench v2 dataset config is incorrect

Open sserdoubleh opened this issue 8 months ago • 0 comments

Prerequisite

Type

I'm evaluating with the officially supported tasks/models/datasets.

Environment

Not relevant

Reproduces the problem - code/configuration sample

opencompass:https://github.com/open-compass/opencompass/blob/main/opencompass/configs/datasets/longbenchv2/longbenchv2_gen_75fbba.py#L19

prompt='Please read the following text and answer the questions below.\n <text> \n {context} \n </text> \n \n What is the correct answer to this question: {question} \n \n Choices: \n (A) {choice_A} \n (B) {choice_B} \n (C) {choice_C} \n (D) {choice_D} \n Let’s think step by step. Based on the above, what is the single, most likely answer choice? Format your response as follows: "The correct answer is (insert answer here)'

longbenchv2:https://github.com/THUDM/LongBench/blob/main/prompts/0shot_cot_ans.txt

Please read the following text and answer the questions below.

The text is too long and omitted here.

What is the correct answer to this question: $Q$
Choices:
(A) $C_A$
(B) $C_B$
(C) $C_C$
(D) $C_D$

Let’s think step by step: $COT$

Based on the above, what is the single, most likely answer choice? Format your response as follows: "The correct answer is (insert answer here)".

OpenCompasse code Missing “.

Reproduces the problem - command or script

Not relevant

Reproduces the problem - error message

Not relevant

Other information

No response

sserdoubleh avatar Jun 02 '25 19:06 sserdoubleh