microsoft
hi ，when the ZeroQuant inference will be released?
Hi,
The engine of ZeroQuant inference is not released yet. The code example in DeepSpeed-Example is only to help verify the accuracy of ZeroQuant.
The kernel/engine released is on our calendar and we are actively working on it to make it compatible for various models. Please stay tuned.
For LKD, we will also release it soon.
For the last question, the code for training or accuracy testing is different than the final inference engine. Here, everything is simulated, so we can do quantization aware training or other things
Originally posted by @yaozhewei in https://github.com/microsoft/DeepSpeed/issues/2207#issuecomment-1212355792
hi ，when the ZeroQuant inference (for GPT model) will be released?
Reza wraps up this https://github.com/microsoft/DeepSpeed/pull/2217 which answers some part of your questions, such as the model size reduction. Regarding the kernels, we are working on a plan to release it soon so that you can give it a try. Thanks,
