quantizer documentation

[patelprateek]

Thanks for the amazing work. I am reading the documentation and it says : 8-bit quantization: Quantile, Linear, and Dynamic quantization In the code i also observe block wise quantization. Can you please share some documentation or pointer to these quantizers . Also curios if these quantizer preserve not just multiplication of matrices but also distance metrics like euclidean , ip, cosine . This would be helpful for ANN (approx nearest neighbour ) techniques as well.

[TimDettmers]

You can read more about these quantization methods in the 8-bit optimizer paper. You can find quantile quantization in the appendix.

None of the block-wise quantization methods preserve distances or multiplication of matrices since they are block-wise dependent.

You can use bnb.functional.double_quant for this which performs linear quantization for the full row and column of an input 2D tensor. As such, linear operations like distance or matrix multiplication preserve their values if you dequantize them after the operation.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.