Issue with Text Encoder Output Dimensions in Fine-Tuned CLIP Model When Using with Stable Diffusion

[QXGeraldMo]

I'm encountering an issue with the dimensions of the text encoder output in a fine-tuned CLIP model. The fine-tuning output of my CLIP model based on RN50 is (1, 1024), whereas the output from CLIPTextModel in transformers is (1, 77, 768). This causes difficulties when integrating with Stable Diffusion models, as the encoder_hidden_states parameter in U-Net requires an embedding of shape (x, x, 768).

Can someone help me deal with this problem? Would truncating the output of encode_text to (1, 1, 768) potentially resolve this issue?

[supersonicMaclaurin]

same question

[99991]

CLIP has many models with different sizes. See also here for HuggingFace CLIP models.

That aside, if you finetune a vision encoder like ResNet50 without also finetuning the text encoder at the same time, their embeddings might become desynchronized and you get bad results. But truncating would definitely not work at all unless the model is specifically trained for this task (see e.g. Matryoshka embeddings).