simple-evals
simple-evals copied to clipboard
Fix MMLU answer extraction regex for repeated "Answer: LETTER" pattern
Description
This pull request addresses issue https://github.com/openai/simple-evals/issues/33 by fixing the regular expression used to extract answers from model outputs with MMLU.
Solution
The existing regex fails to handle cases where the "Answer: LETTER" pattern appears multiple times. This is resolved by:
- Using
re.findall: Instead ofre.search,re.findallis used to find all occurrences of the answer pattern. - Selecting the last match: The last match from the
re.findallresults is taken as the correct answer. - Allowing overlapping matches: The regex pattern is adjusted to allow overlapping matches, using a capturing group inside a lookahead.