Xianjie Qiao

icon indicating an email [email protected]

icon location Beijing icon location A software developer focus on MLSys, 10+ years of experience in CUDA programming

Results 3 comments of Xianjie Qiao

Support W8A8 inference in vllm

Hi, Is this per-token quantization patch only support single card? I tested this patch on A10 with llama2-7b, there is no problem if I run with single card. But if...

H20 infer Qwen14B fp8 results in a divide-by-zero error.

pip install nvidia-cublas-cu12==12.3.4.1

[None][fix] Fix on-disk cache and revise logger/statistics for AutoTuner.

/bot run