bugzyz comments

Results 7 comments of


                                            bugzyz

Inference performance drop 22X on GPU hardware with optimum[onnxruntime-gpu] (compared with transformer)

Also, we did some nsight GPU profiling on these execution and got some strange finding. We hope these will help. no.1: optimum took lots of time on data copy optimum...

Inference performance drop 22X on GPU hardware with optimum[onnxruntime-gpu] (compared with transformer)

Hi @JingyaHuang, Thanks for the new feature! Sure, we want to have a try on this to see onnx/optimum's performance with this enhancement. we will try the new release today...

Inference performance drop 22X on GPU hardware with optimum[onnxruntime-gpu] (compared with transformer)

@JingyaHuang Thanks for the new feature. I validate the latest optimum with previous code. The optimum ran faster than original transformer one. (latency: 2.73s vs 2.13s) 🆘🆘🆘But there's a warning...

Inference performance drop 22X on GPU hardware with optimum[onnxruntime-gpu] (compared with transformer)

Thanks @JingyaHuang for the confirmation. One more minor found from us is that we used to create device by `device = torch.device('cuda')`. But it leads to this error after using...

Inference performance drop 22X on GPU hardware with optimum[onnxruntime-gpu] (compared with transformer)

The feature is very valuable for user like us to use optimum. Big thanks! 🎉 @JingyaHuang

Pytorch member variable not working after converting to onnx format

@BowenBao Hi there, any luck on this issue?

Streamable HTTP

And also need support on stateless mode. We tried some app client like mcp-inspector but it hasn't supported the stateless mode well.