Broken GPU inference kernel output on Turing

[etasnadi]

Dear All,

I tested the code on a GeForce 2060 GPU and seems that the W2A8 matmul output is wrong.

The same code works properly with an RTX A5000 GPU (Ampere).

Has anyone else experienced this issue on Turing?

Thanks

Reference:

$ python3 test.py 
custom == np False
Shape(2560, 2560), W2A8: 10.90us, torch BF16: 39.58us
custom == np False
Shape(3840, 2560), W2A8: 11.37us, torch BF16: 53.93us
custom == np False
Shape(13824, 2560), W2A8: 10.97us, torch BF16: 173.78us
custom == np False
Shape(2560, 6912), W2A8: 10.91us, torch BF16: 92.06us
custom == np False
Shape(3200, 3200), W2A8: 11.00us, torch BF16: 52.27us
custom == np False
Shape(4800, 3200), W2A8: 11.03us, torch BF16: 75.66us
custom == np False
Shape(3200, 10240), W2A8: 10.82us, torch BF16: 157.68us
custom == np False
Shape(20480, 3200), W2A8: 10.98us, torch BF16: 304.58us