Float FMA vs Integer DP4A & DPX Instructions

[ashvardanian]

CUDA natively supports Fused-Multiply-Accumulate operations for every float type, including f16 and bf16. It also provides DP4A instructions for 8-bit integer dot-products with 32-bit accumulators and umul24 instructions for 24-bit integer multiplication. Starting with Hopper, Dynamic Programming eXtensitons (DPX) were added for combinatorial problems that can be used to implement Algebraic Graph Theory algorithms using matrix multiplications over alternative semirings.

How do those instructions stack up, and how much performance can we expect from recent State-of-the-Art GPUs like the Nvidia H200?

• f64 FMA: 4.5 T
• i64 FMA: 3.1 T
• f32 FMA: 22 T
• i32 FMA: 15.5 T ...so we should always prefer 32-bit ops
• u8u32 DP4A: 39.3 T
• u24u32 UMUL: 13.4 T ...not really better than i32 FMA
• f16 FMA on Volta: 12.2 T
• bf16 FMA on Ampere: 12.2 T
• DPX for Floyd-Warshall algorithm with u16 and u32 on Hopper: 11 T
• DPX for Needleman-Wunsch algorithm with i16 and i32 on Hopper: 11 T
• DPX for Smith-Waterman algorithm with i32 on Hopper: 27 T

Check the code and inline comments for more details!