InfiniteBench icon indicating copy to clipboard operation
InfiniteBench copied to clipboard

bug in computing scores for longdialogue_qa_eng

Open Xianchao-Wu opened this issue 1 year ago • 1 comments

https://github.com/OpenBMB/InfiniteBench/blob/main/src/compute_scores.py#L238

  1. only one reference label is used for comparison, better loop around each answer in label, e.g., label=['ECKER', 'COMMANDER BILL ECKER'];
  2. prediction phrase is splitted into words for comparison, which means: pred=STAMP PAID, label=['STAMP PAID'], score=0.0, data_name=longdialogue_qa_eng

and

pred=ADRIAN, label=['ADRIAN'], score=1.0, data_name=longdialogue_qa_eng

Xianchao-Wu avatar Jul 17 '24 06:07 Xianchao-Wu

We have fixed this bug in the latest commit. Thank you for your notification.

tuantuanzhang avatar Aug 08 '24 06:08 tuantuanzhang