Inputs without JPEG conversion

Open kydxh opened this issue 11 months ago • 2 comments

Hello, I've notice that the input images used in VLMEvalKit are converted to JPEG format. I wonder if there is version that use the original images instead of JPEG format as input?

Mar 10 '25 06:03 kydxh

Hi @kydxh, we do not provide evaluation sets using original images, but the image quality before and after conversion should not change much, and the impact on the evaluation results is quite small. Which dataset do you want to evaluate?

Mar 12 '25 06:03 FangXinyu-0913

Thank you. I want to evaluate some VQA datasets, such as ScienceQA, MMMU, Mathvista and TextVQA. Besides, I want to ask that, if the bugs in evaluating TextVQA dataset mentioned before has been modified?

Mar 12 '25 14:03 kydxh