Batch inference

[VietDunghacker]

How to perform batch inference with swift? I don't see it mentioned anywhere in the docs and I cannot find it in the code either.

[Jintao-Huang]

Using the infer_backend vllm allows for batch inference.

https://github.com/modelscope/swift/blob/main/docs/source/LLM/VLLM%E6%8E%A8%E7%90%86%E5%8A%A0%E9%80%9F%E4%B8%8E%E9%83%A8%E7%BD%B2.md

The "inference_vllm" can take a "request_list" as input.

[VietDunghacker]

Thank you.

[VietDunghacker]

@Jintao-Huang vllm is great, but unfortunately vllm does not support all models in this repo. For instance, Phi-3 Vision is supported in their Github repo but not in the official pip version. I really think it will be helpful if the feature is implemented natively in swift instead of relying on vllm.

[tastelikefeet]

Thanks for you suggestion! We have added batch inference for pytorch native to our todo list. This requirement will be accomplished in one sprint

[gouqi666]

Using the infer_backend vllm allows for batch inference.

https://github.com/modelscope/swift/blob/main/docs/source/LLM/VLLM%E6%8E%A8%E7%90%86%E5%8A%A0%E9%80%9F%E4%B8%8E%E9%83%A8%E7%BD%B2.md

The "inference_vllm" can take a "request_list" as input.

hello, @Jintao-Huang , the page is not found

[Jintao-Huang]

supported

https://swift.readthedocs.io/zh-cn/latest/Instruction/%E6%8E%A8%E7%90%86%E5%92%8C%E9%83%A8%E7%BD%B2.html