[FEA] Has CUTLASS considered supporting Zero-points and block-wise scaling in Hoppr Mixed Grouped Gemm recently?

[mengsoso]

For the Hopper architecture, the mixed_dtype_grouped_gemm currently only supports row-wise scaling. However, for the AWQ quantization, the precision loss is still quite significant.

Will CUTLASS support the Zero-points and block-wise scaling of AWQ (W4A16 / W4A8) for MoE models?

Thanks~

[mnicely]

Hi @mengsoso, this feature is not on our roadmap, but we welcome community contributions!

[mnicely]

I'd like to more about your use case and model

[hwu36]

cc += @alexsamardzic @IwakuraRein

[alexsamardzic]

I'd say CUTLASS is more about providing tools to implement such kind of functionality, and for the requested functionality itself you may want to track the development, or open an issue, in a project like torchao.

[github-actions[bot]]

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.

[ankutalev]

Hello! I create a PR with this functionality

Can you review it, please?

cc @mnicely @hwu36

[github-actions[bot]]

This issue has been labeled inactive-90d due to no recent activity in the past 90 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed.