aabbccddwasd

is there any quantized models for Qwen2.5-VL

15

FP16 version is impossible to run on local GPU, I hope there will be GPTQ and AWQ versions, please!!!

[Usage]: how to tune a dense fp8_w8a8 model

2

### Your current environment ```text The output of `python collect_env.py` ``` ### How would you like to use vllm (EngineCore_0 pid=20659) WARNING 08-11 05:36:17 [fp8_utils.py:593] Using default W8A8 Block FP8...

多显卡+cpu推理满血deepseek

1

假设我使用4张rtx pro 6000加双路epyc 9005推理满血的deepseek 685B，性能上会领先于单路吗？