Zafir Stojanovski

@amyeroberts Doesn't `idefics2` already handle this? https://github.com/huggingface/transformers/blob/cf7bed98325a0be9d195cb6b66c6a0bef9fccbc8/src/transformers/models/idefics2/modeling_idefics2.py#L139-L149 For example, the following sample script: ```py import torch import requests from PIL import Image from transformers import Idefics2Processor, Idefics2ForConditionalGeneration device = torch.device("cuda"...

Opened a PR (#30722) addressing this issue for the BLIP family of models (BLIP, BLIP2, InstructBLIP).

@amyeroberts good point! the tests now also have a check for the textual content.

@haileyschoelkopf I believe this issue should be closed now.

@amyeroberts I have now force-pushed only my changes 👍

@amyeroberts thank you for the constructive feedback. I am currently experiencing some weird behavior when I integrate those changes, perhaps I am not 100% familiar with the internals of the...

Hi @amyeroberts @younesbelkada With the implementation you proposed, for the following sample code: ```py import torch from transformers import Idefics2ForConditionalGeneration model = Idefics2ForConditionalGeneration.from_pretrained( "HuggingFaceM4/idefics2-8b", torch_dtype=torch.bfloat16, attn_implementation="flash_attention_2", ) print("Perceiver model flash...

Moreover, when using the vision tower with Flash Attention, I get this exception: ``` The input hidden states seems to be silently casted in float32, this might be related to...

Hi @haileyschoelkopf I have started looking into this. I am confused as to why even though the `Task` class has a `download` method: https://github.com/EleutherAI/lm-evaluation-harness/blob/86319a9b14ddae2030bc6e0fdddd47fc7d0bb525/lm_eval/api/task.py#L236-L240 the `ConfigurableTask` class is the only...