math icon indicating copy to clipboard operation
math copied to clipboard

typo in `_strip_string`?

Open sieu-n opened this issue 1 year ago • 1 comments

Description

There appears to be a typo in the _strip_string function of the math_equivalence.py file, where an unnecessary escape character is used with the percent sign. The code uses string.replace("\%", "") when it seems to me that string.replace("%", "") was intended. I'm not 100% certain whether this behavior is unintentional but currently is_equiv("50%", "50") == False. The current code also throws a warning for python 3.11.

https://github.com/hendrycks/math/blob/357963a7f5501a6c1708cf3f3fb0cdf525642761/modeling/math_equivalence.py#L104

Suggestion

string = string.replace("\%", "")
->
string = string.replace("%", "")

sieu-n avatar Feb 17 '24 07:02 sieu-n

Just did some research for 5 mins

Everything in the data should be correctly normalized since they all seem to be formatted like 50\%.

The only difference due to this is when the model predicts 50% instead of 50 or 50\% because is_equiv("50\\%", "50%") == False. There are 51 % signs in the test set(~1%), and the results are only impacted if the model acts in a certain way.

edit: typo

sieu-n avatar Feb 17 '24 08:02 sieu-n