Corrupted MathVista testmini data

Open LeoDu0314 opened this issue 1 year ago • 2 comments

I found that the MathVista testmini data used in VLMEvalkit(https://opencompass.openxlab.space/utils/VLMEval/MathVista_MINI.tsv) is not identical to that provided by the MathVista offical huggingface repo(https://huggingface.co/datasets/AI4Math/MathVista)

The VLMEvalkit version of MathVista testmini data has the following problem:

Images are compressed
Misuse of \r\n in some questions.

Could you provide an updated version of MathVista testmini?

Nov 05 '24 13:11 LeoDu0314

Hi, @LeoDu0314 , During our pre-processing, we have converted images to JPEG format. We will try to build another version that do not perform the JPEG conversion and do the evaluation again to see the gap.

Nov 06 '24 08:11 kennymckormick

Hello, I wonder if there is a version that do not perform the JPEG conversion now?

Mar 06 '25 13:03 kydxh