xchen
xchen
目前不支持internvl，只支持qwen2-vl，进行中的有qwen2-audio。
@j4yan Hi, I updated [matmul.cpp](https://github.com/x574chen/CUDALibrarySamples/blob/master/cuSPARSELt/matmul/matmul_example.cpp) and got the same error. Log: [cusparse_error.log](https://github.com/user-attachments/files/17041725/cusparse_error.log) Which docker are you using? If it is public, I could try it.
The latest version solved the multithreading issue. Closing it
@kzjeef dashinfer_vlm_serve --model /mnt/ssd/xchen/workspace/Qwen2.5-VL-3B-Instruct/ --vision_engine tensorrt 2>&1 | tee /tmp/vl2_5_trt.log [vl2_5_trt.log](https://github.com/user-attachments/files/21500244/vl2_5_trt.log) transformers==4.52.3 TensorRT-10.5.0.18
@kzjeef no update on transformers for vit dashinfer_vlm_serve --model /mnt/ssd/xchen/workspace/Qwen2.5-VL-3B-Instruct/ --vision_engine transformers --parallel_size 1 2>&1 | tee /tmp/vl2_5_transformers.log [vl2_5_transformers.log](https://github.com/user-attachments/files/21522916/vl2_5_transformers.log) transformers==4.52.3
@kzjeef GPTQ, a8w8, a16w4, fp8都支持了 https://github.com/modelscope/dash-infer/blob/main/multimodal/dashinfer_vlm/vl_inference/utils/model_loader.py#L145
dashinfer_vlm_serve 启动加上对应的--quant-type