Evaluating LLava: batch size > 1?

[kahnchana]

Good job on the really insightful and useful paper! The quantitative metrics are really useful when working with these models.

Question: When evaluating the LLava baseline (Table 4 in paper), did you do batch-wise inference?

I'm trying to replicate numbers of that Table, but I'm following default LLava setup (using per sample, i.e. batch size of 1) which is extremely slow.

[daoyuan98]

Yes, we have met the same issue... We currentlt use batch size of 1 to evaluate all models.