lighteval icon indicating copy to clipboard operation
lighteval copied to clipboard
verified publisher icon huggingface

`apply_target_perplexity_metric` pops only the first response

Open sadra-barikbin opened this issue 1 year ago • 0 comments

Hi there.

apply_target_perplexity_metric seems to pop only the first related response not all of them.

https://github.com/huggingface/lighteval/blob/ad42e43bcc3bd50fdba68936999bf553bf53b9e4/src/lighteval/metrics/init.py#L31-L36

While its related responses are as much as the number of gold choices.

sadra-barikbin avatar May 20 '24 16:05 sadra-barikbin