Márton Kardos
Márton Kardos
Okay, so I have fixed the cases where the results are present, but in the external results folder. On the other hand, for some models, like voyage2-large-instruct we are missing...
So, in conclusion, it is a bug with the old leaderboard, and the only way to go about fixing it is for us to run MSMARCO's dev split on these...
Sure thing @wissam-sib! By all means go ahead
@Muennighoff Can you help us out with it? Some models don't have MSMARCO results at all on the dev split, and we might need to run them.
We should probably annotate this somehow in the model metadata. Perhaps add it as a framework? Btw I might be using the filters wrong, but it doesn't seem to me...
Yeah but I imagine you could add `frameworks=["Sentence Transformers", "Cross Encoders"]` and then you would know that they are sentence-transformer compatible. `is_cross_encoder` is not a bad idea either, we could...
Alright, we do it that way then
Yes, if we can get an estimate on the number of tokens we need, Tengyu agreed to provide them for us.
How would MIEB fit into this?
Here's a fix: #2221