Support for Vidore-v3 benchmark

[StupidBuluchacha]

Hi Vidore Team, thanks for releasing vidore-v3 benchmark.

I modified the pipeline of vidore-benchmark codebase before (to fit my own research needs), so I am trying to evaluate the recently released vidore-v3 on vidore-benchmark. But it turns out that I got "DatasetGenerationError: An error occurred while generating the dataset". This error did not occur when I ran other benchmarks such vidore-v1 and v2 successfully. So I am wondering if you can check on this, or is there any different setting we should know if we still use vidore-benchmark to evaluate?

Below is my example script for vidore-v3: vidore-benchmark evaluate-retriever \ --model-class colpali \ --model-name vidore/colpali-v1.3 \ --dataset-name vidore/vidore_v3_hr \ --dataset-format beir \ --split test

Then I got the following the error:

[QuentinJGMace]

Hello @StupidBuluchacha !

what's your transformers and datasets versions ? can you try to update them ? A first test would be to try to load the datasets in a notebook as I think this error can only happen at dataset loading.

However when this is fixed, you should still encounter an error due to format mismatch between v2 and v3, you can change the corpus_id_column and query_id_column to "corpus_id" and "query_id" here (and not "corpus-id" and "query-id")

In general I would advise using mteb (they already support most publicly available visual retrievers), and this repo should not be maintained much in the future.