Quantization

[IceTea42]

Hi, thank you for this work. How to quantize it to use int8? Any comments are appreciated.

[QuangHyperGAI]

You can quantize our model at run time using bitsandbyte. Please refer to the steps below.

1. Install bitsandbyte

pip install bitsandbyte

2. Modify the vlmeval/vlm/hpt.pt file as follows.

from transformers import BitsAndBytesConfig

# this part should go at line 53
quant_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type='nf4',
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,)

llm = AutoModelForCausalLM.from_pretrained(global_model_path,
    quantization_config=quant_config,
    subfolder = 'llm',
    trust_remote_code=True,
    use_safetensors=True,
    torch_dtype=torch_dtype,
    device_map='cuda')
self.llm = llm

3. Then you can proceed to run the demo

python demo/demo.py --image_path demo/einstein.jpg  --text 'Question: What is unusual about this image?\nAnswer:'  --model hpt-air-demo-local

This is the output on my end.

The unusual aspect of this image is the depiction of a famous scientist, such as Albert Einstein, holding a cell phone. It is not common to see a renowned scientist using a cell phone, as they are often associated with more traditional communication methods or research. The painting or drawing of Einstein holding a cell phone is an interesting and unexpected representation of the scientist's daily life or activities.