lightning-thunder
lightning-thunder copied to clipboard
Lightning-AI
Enable cat for nvfuser >= 0.1.7
Benchmarks are neutral.
Before:
--------------------------------------------------------------------------------------------------------------------- benchmark: 24 tests ----------------------------------------------------------------------------------------------------------------------
Name (time in us) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_nanogpt_sdpa_fwd[thunder] 85.9729 (1.0) 123.5039 (1.0) 90.1970 (1.0) 9.0070 (1.45) 87.2805 (1.0) 1.0572 (1.0) 2;3 11,086.8437 (1.0) 20 1
test_nanogpt_layer_norm_fwd[thunder] 98.7879 (1.15) 236.1580 (1.91) 108.8600 (1.21) 30.0711 (4.83) 101.3121 (1.16) 3.2765 (3.10) 1;2 9,186.1104 (0.83) 20 1
test_nanogpt_gelu_fwd[thunder] 110.0181 (1.28) 217.0510 (1.76) 115.9017 (1.28) 16.7485 (2.69) 112.1076 (1.28) 3.1011 (2.93) 1;3 8,628.0009 (0.78) 40 1
test_nanogpt_gelu_grad[thunder] 293.2760 (3.41) 706.9011 (5.72) 324.0489 (3.59) 69.9223 (11.23) 299.2521 (3.43) 15.2696 (14.44) 2;9 3,085.9543 (0.28) 40 1
test_nanogpt_csa_fwd[thunder] 312.5530 (3.64) 433.2711 (3.51) 320.6752 (3.56) 26.5578 (4.26) 314.2810 (3.60) 2.4339 (2.30) 1;1 3,118.4206 (0.28) 20 1
test_llama2_7b_rmsnorm_grad[thunder] 413.3140 (4.81) 725.0749 (5.87) 494.8684 (5.49) 73.1271 (11.74) 511.1340 (5.86) 127.2761 (120.39) 20;0 2,020.7393 (0.18) 40 1
test_nanogpt_sdpa_grad[thunder] 465.0408 (5.41) 629.3731 (5.10) 550.4548 (6.10) 57.3874 (9.22) 584.9096 (6.70) 97.4401 (92.17) 8;0 1,816.6797 (0.16) 20 1
test_nanogpt_mlp_fwd[thunder] 474.1389 (5.51) 635.3639 (5.14) 480.8633 (5.33) 25.2371 (4.05) 476.1979 (5.46) 1.9741 (1.87) 1;4 2,079.5930 (0.19) 40 1
test_nanogpt_cross_entropy_fwd[thunder] 617.4808 (7.18) 705.7891 (5.71) 626.5095 (6.95) 18.7819 (3.02) 622.5256 (7.13) 3.8975 (3.69) 1;1 1,596.1449 (0.14) 20 1
test_nanogpt_block_fwd[thunder] 788.3550 (9.17) 1,079.0159 (8.74) 811.1224 (8.99) 63.5831 (10.21) 795.4536 (9.11) 11.8630 (11.22) 1;2 1,232.8595 (0.11) 20 1
test_nanogpt_csa_grad[thunder] 1,125.9848 (13.10) 1,563.9060 (12.66) 1,389.8655 (15.41) 159.5754 (25.63) 1,458.0046 (16.70) 286.0019 (270.54) 6;0 719.4941 (0.06) 20 1
test_nanogpt_mlp_grad[thunder] 1,340.7920 (15.60) 1,653.8859 (13.39) 1,429.6559 (15.85) 87.8710 (14.11) 1,405.1094 (16.10) 48.2167 (45.61) 6;5 699.4690 (0.06) 40 1
test_nanogpt_cross_entropy_grad[thunder] 1,869.2850 (21.74) 1,897.7679 (15.37) 1,875.7246 (20.80) 6.2271 (1.0) 1,874.9150 (21.48) 6.9386 (6.56) 2;1 533.1273 (0.05) 20 1
test_nanogpt_block_grad[thunder] 2,490.2120 (28.97) 3,805.0050 (30.81) 2,805.9399 (31.11) 349.0571 (56.05) 2,594.0740 (29.72) 456.2305 (431.56) 3;1 356.3868 (0.03) 20 1
test_nanogpt_gpt2_fwd[thunder] 6,656.4621 (77.43) 8,000.5289 (64.78) 6,937.3684 (76.91) 594.5807 (95.48) 6,672.7730 (76.45) 367.6593 (347.78) 1;1 144.1469 (0.01) 5 1
test_nanogpt_gpt2_grad[thunder] 23,743.2919 (276.17) 27,975.8458 (226.52) 25,283.1272 (280.31) 1,732.2901 (278.18) 24,580.3180 (281.62) 2,475.7239 (>1000.0) 1;0 39.5521 (0.00) 5 1
test_nanogpt_gpt2xl_fwd[thunder] 35,379.9469 (411.52) 38,557.2209 (312.19) 37,649.9868 (417.42) 1,292.4774 (207.56) 38,157.8330 (437.19) 1,098.8793 (>1000.0) 1;1 26.5604 (0.00) 5 1
test_llama2_7b_sdpa_grad[thunder] 53,339.5000 (620.42) 54,730.9991 (443.15) 53,902.8342 (597.61) 402.7068 (64.67) 53,828.8241 (616.73) 643.8149 (609.00) 15;0 18.5519 (0.00) 40 1
test_nanogpt_gpt2xl_grad[thunder] 125,010.8320 (>1000.0) 127,470.0339 (>1000.0) 125,858.3189 (>1000.0) 986.0469 (158.35) 125,817.0409 (>1000.0) 1,205.6009 (>1000.0) 1;0 7.9454 (0.00) 5 1
test_llama2_causal_self_attention_7b_grad[thunder] 183,797.7511 (>1000.0) 188,489.4371 (>1000.0) 186,973.9573 (>1000.0) 919.1014 (147.60) 186,991.5576 (>1000.0) 1,137.5169 (>1000.0) 10;1 5.3483 (0.00) 40 1
test_llama2_mlp_7b_grad[thunder] 236,751.7750 (>1000.0) 242,391.4201 (>1000.0) 239,621.8158 (>1000.0) 1,236.3999 (198.55) 239,807.0445 (>1000.0) 1,620.5839 (>1000.0) 10;0 4.1732 (0.00) 40 1
test_open_llama_7b_fwd[thunder] 1,134,035.7491 (>1000.0) 1,137,398.5051 (>1000.0) 1,135,597.6682 (>1000.0) 1,525.8574 (245.03) 1,135,803.1400 (>1000.0) 2,821.9949 (>1000.0) 3;0 0.8806 (0.00) 5 1
test_llama_2_7b_grad[thunder] 1,771,630.3240 (>1000.0) 1,812,210.8860 (>1000.0) 1,794,236.1877 (>1000.0) 11,941.4007 (>1000.0) 1,795,424.9046 (>1000.0) 19,348.8440 (>1000.0) 8;0 0.5573 (0.00) 20 1
test_llama_2_7b_hf_fwd[thunder] 2,362,912.0821 (>1000.0) 2,380,006.6570 (>1000.0) 2,372,338.9194 (>1000.0) 6,423.7285 (>1000.0) 2,373,629.3749 (>1000.0) 8,465.9495 (>1000.0) 2;0 0.4215 (0.00) 5 1
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
After:
--------------------------------------------------------------------------------------------------------------------- benchmark: 24 tests ----------------------------------------------------------------------------------------------------------------------
Name (time in us) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_nanogpt_sdpa_fwd[thunder] 85.2009 (1.0) 124.4859 (1.0) 88.2313 (1.0) 8.6566 (1.59) 85.8374 (1.0) 0.7616 (1.0) 1;3 11,333.8463 (1.0) 20 1
test_nanogpt_layer_norm_fwd[thunder] 96.7828 (1.14) 226.2490 (1.82) 106.2040 (1.20) 28.4880 (5.23) 98.4970 (1.15) 5.3196 (6.98) 1;1 9,415.8393 (0.83) 20 1
test_nanogpt_gelu_fwd[thunder] 109.7580 (1.29) 220.9090 (1.77) 115.8981 (1.31) 18.3704 (3.37) 111.4970 (1.30) 1.8084 (2.37) 2;4 8,628.2722 (0.76) 40 1
test_nanogpt_gelu_grad[thunder] 281.1830 (3.30) 498.5251 (4.00) 327.3759 (3.71) 48.0104 (8.81) 343.1151 (4.00) 64.7875 (85.07) 2;2 3,054.5923 (0.27) 40 1
test_nanogpt_csa_fwd[thunder] 310.6389 (3.65) 421.8400 (3.39) 318.4893 (3.61) 24.3820 (4.48) 312.7330 (3.64) 2.3684 (3.11) 1;1 3,139.8229 (0.28) 20 1
test_llama2_7b_rmsnorm_grad[thunder] 321.2700 (3.77) 556.2950 (4.47) 427.4230 (4.84) 64.2681 (11.80) 406.9416 (4.74) 91.8243 (120.57) 17;0 2,339.6027 (0.21) 40 1
test_nanogpt_sdpa_grad[thunder] 418.2830 (4.91) 596.5910 (4.79) 497.8039 (5.64) 41.0411 (7.53) 483.9625 (5.64) 21.3555 (28.04) 4;4 2,008.8230 (0.18) 20 1
test_nanogpt_mlp_fwd[thunder] 475.6320 (5.58) 641.9369 (5.16) 481.3083 (5.46) 26.0898 (4.79) 476.8439 (5.56) 1.2020 (1.58) 1;6 2,077.6705 (0.18) 40 1
test_nanogpt_cross_entropy_fwd[thunder] 618.6331 (7.26) 918.3811 (7.38) 636.9287 (7.22) 66.2812 (12.17) 621.8040 (7.24) 3.5827 (4.70) 1;1 1,570.0344 (0.14) 20 1
test_nanogpt_block_fwd[thunder] 787.0132 (9.24) 1,084.0050 (8.71) 806.6165 (9.14) 65.5276 (12.03) 789.6619 (9.20) 7.1742 (9.42) 1;1 1,239.7466 (0.11) 20 1
test_nanogpt_csa_grad[thunder] 1,164.3781 (13.67) 1,623.8189 (13.04) 1,377.5621 (15.61) 127.9834 (23.50) 1,330.9990 (15.51) 172.1865 (226.09) 8;0 725.9201 (0.06) 20 1
test_nanogpt_mlp_grad[thunder] 1,330.7731 (15.62) 1,913.2278 (15.37) 1,416.6653 (16.06) 128.9129 (23.67) 1,366.1454 (15.92) 97.8445 (128.47) 6;4 705.8830 (0.06) 40 1
test_nanogpt_cross_entropy_grad[thunder] 1,867.4510 (21.92) 1,888.5611 (15.17) 1,874.1238 (21.24) 5.4472 (1.0) 1,872.5011 (21.81) 6.8379 (8.98) 7;1 533.5827 (0.05) 20 1
test_nanogpt_block_grad[thunder] 2,575.9540 (30.23) 4,624.9500 (37.15) 3,135.8434 (35.54) 394.9543 (72.51) 3,112.7925 (36.26) 27.5585 (36.19) 3;6 318.8935 (0.03) 20 1
test_nanogpt_gpt2_fwd[thunder] 6,583.1831 (77.27) 7,109.2511 (57.11) 6,706.5494 (76.01) 226.2149 (41.53) 6,622.6891 (77.15) 167.7495 (220.26) 1;1 149.1080 (0.01) 5 1
test_nanogpt_gpt2_grad[thunder] 24,254.6930 (284.68) 27,537.9689 (221.21) 25,215.8198 (285.79) 1,416.5809 (260.06) 24,382.2648 (284.05) 1,813.9615 (>1000.0) 1;0 39.6576 (0.00) 5 1
test_nanogpt_gpt2xl_fwd[thunder] 36,717.4961 (430.95) 38,749.9100 (311.28) 38,145.8928 (432.34) 861.6864 (158.19) 38,645.1820 (450.21) 1,051.7663 (>1000.0) 1;0 26.2151 (0.00) 5 1
test_llama2_7b_sdpa_grad[thunder] 53,971.0042 (633.46) 55,367.9529 (444.77) 54,619.9200 (619.05) 403.7169 (74.11) 54,645.0875 (636.61) 733.2700 (962.82) 16;0 18.3083 (0.00) 40 1
test_nanogpt_gpt2xl_grad[thunder] 127,010.2949 (>1000.0) 129,932.8168 (>1000.0) 128,392.6164 (>1000.0) 1,271.2247 (233.37) 128,682.6360 (>1000.0) 2,229.3555 (>1000.0) 2;0 7.7886 (0.00) 5 1
test_llama2_causal_self_attention_7b_grad[thunder] 180,725.4220 (>1000.0) 185,217.8711 (>1000.0) 183,897.6699 (>1000.0) 912.1753 (167.46) 183,977.0535 (>1000.0) 1,265.6110 (>1000.0) 12;1 5.4378 (0.00) 40 1
test_llama2_mlp_7b_grad[thunder] 238,051.7379 (>1000.0) 243,403.6671 (>1000.0) 240,681.5872 (>1000.0) 1,245.0415 (228.57) 240,967.0515 (>1000.0) 1,819.9860 (>1000.0) 11;0 4.1549 (0.00) 40 1
test_open_llama_7b_fwd[thunder] 1,083,434.2770 (>1000.0) 1,085,535.2401 (>1000.0) 1,084,531.6276 (>1000.0) 809.0349 (148.52) 1,084,406.5200 (>1000.0) 1,167.9122 (>1000.0) 2;0 0.9221 (0.00) 5 1
test_llama_2_7b_grad[thunder] 1,759,139.5450 (>1000.0) 1,798,621.3120 (>1000.0) 1,781,164.2416 (>1000.0) 11,620.1781 (>1000.0) 1,782,893.3335 (>1000.0) 20,682.9539 (>1000.0) 8;0 0.5614 (0.00) 20 1
test_llama_2_7b_hf_fwd[thunder] 2,260,459.6941 (>1000.0) 2,275,246.8339 (>1000.0) 2,268,166.2148 (>1000.0) 5,800.4606 (>1000.0) 2,268,238.2530 (>1000.0) 8,989.1682 (>1000.0) 2;0 0.4409 (0.00) 5 1
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Benchmarked on the following GPU:
$ nvidia-smi -L
GPU 0: NVIDIA A100 80GB PCIe (UUID: GPU-af301991-abb4-a8db-ec48-be7abb9c59b3)
I don't understand why "docs-make / make-docs" is still pending, but this should be ready to merge.