Error occurred during BLIP2-demo execution

[Kim-DKyu]

Hi, everyone. I encountered the following errors during the execution of BLIP2-demo of huggingface. I executed the following code.

import os
os.environ['CUDA_DEVICE_ORDER'] = 'PCI_BUS_ID'
os.environ['CUDA_VISIBLE_DEVICES'] = "3"

from PIL import Image
import requests
from transformers import Blip2Processor, Blip2ForConditionalGeneration
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"

processor = Blip2Processor.from_pretrained("Salesforce/blip2-opt-2.7b")
model = Blip2ForConditionalGeneration.from_pretrained(
    "Salesforce/blip2-opt-2.7b", torch_dtype=torch.float16
)
model.to(device)
url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

prompt = "Question: how many cats are there? Answer:"
inputs = processor(images=image, text=prompt, return_tensors="pt").to(device, torch.float16)

generated_ids = model.generate(**inputs)
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0].strip()
print(generated_text)

---

ValueError                                Traceback (most recent call last)
/tmp/ipykernel_43434/3245279263.py in <module>
     21 inputs = processor(images=image, text=prompt, return_tensors="pt").to(device, torch.float16)
     22 
---> 23 generated_ids = model.generate(**inputs)
     24 generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0].strip()
     25 print(generated_text)

~/.local/lib/python3.8/site-packages/torch/autograd/grad_mode.py in decorate_context(*args, **kwargs)
     25         def decorate_context(*args, **kwargs):
     26             with self.clone():
---> 27                 return func(*args, **kwargs)
     28         return cast(F, decorate_context)
     29 

~/.local/lib/python3.8/site-packages/transformers/models/blip_2/modeling_blip_2.py in generate(self, pixel_values, input_ids, attention_mask, **generate_kwargs)
   1828         inputs_embeds = torch.cat([language_model_inputs, inputs_embeds.to(language_model_inputs.device)], dim=1)
   1829 
-> 1830         outputs = self.language_model.generate(
   1831             inputs_embeds=inputs_embeds,
   1832             attention_mask=attention_mask,

~/.local/lib/python3.8/site-packages/torch/autograd/grad_mode.py in decorate_context(*args, **kwargs)
     25         def decorate_context(*args, **kwargs):
     26             with self.clone():
---> 27                 return func(*args, **kwargs)
     28         return cast(F, decorate_context)
     29 

~/.local/lib/python3.8/site-packages/transformers/generation/utils.py in generate(self, inputs, generation_config, logits_processor, stopping_criteria, prefix_allowed_tokens_fn, synced_gpus, assistant_model, streamer, negative_prompt_ids, negative_prompt_attention_mask, **kwargs)
   1469                 self._setup_cache(cache_cls, max_batch_size=batch_size, max_cache_len=generation_config.max_length)
   1470 
-> 1471         self._validate_generated_length(generation_config, input_ids_length, has_default_max_length)
   1472 
   1473         # 7. determine generation mode

~/.local/lib/python3.8/site-packages/transformers/generation/utils.py in _validate_generated_length(self, generation_config, input_ids_length, has_default_max_length)
   1187         if input_ids_length >= generation_config.max_length:
   1188             input_ids_string = "decoder_input_ids" if self.config.is_encoder_decoder else "input_ids"
-> 1189             raise ValueError(
   1190                 f"Input length of {input_ids_string} is {input_ids_length}, but `max_length` is set to"
   1191                 f" {generation_config.max_length}. This can lead to unexpected behavior. You should consider"

ValueError: Input length of input_ids is 0, but `max_length` is set to -23. This can lead to unexpected behavior. You should consider increasing `max_length` or, better yet, setting `max_new_tokens`.

I executed the following code, but even when providing a prompt, the input_ids length is 0, and max_length is negative. Have you ever experienced a similar error? Thank you in advance.

[Yunski]

you can set max_new_tokens like:

model.generate(**inputs, max_new_tokens=10)