[BUG] Build failed with nvcc "-G" on H100

[fei-xx]

Describe the bug The build of examples/54_hopper_fp8_warp_specialized_gemm/54_hopper_fp8_warp_specialized_gemm.cu fail when adding -G to nvcc; but without -G, the build works well.

Steps/Code to reproduce bug Follow this guide http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports to craft a minimal bug report. This helps us reproduce the issue you're having and resolve the issue more quickly.

nvcc -t=0 \
-lineinfo -g -G \
-DCUTLASS_ENABLE_TENSOR_CORE_MMA=1 -DCUTLASS_DEBUG_TRACE_LEVEL=1 \
-w -gencode=arch=compute_90a,code=[sm_90a,compute_90a] \
-O1 -std=c++17 --expt-relaxed-constexpr \
-Xcompiler=-fPIC -Xcompiler=-fno-strict-aliasing -Xcompiler -fvisibility=hidden \
-Xcompiler=-Wconversion \
-I/home/fff/git/cutlass/include \
-I/home/fff/git/cutlass/tools/library/include \
-I/home/fff/git/cutlass/tools/library/src \
-I/home/fff/git/cutlass/tools/util/include \
-I/home/fff/git/cutlass/examples/common/ \
-I/usr/local/cuda-12.1/targets/x86_64-linux/include/ \
-L/usr/local/cuda-12.1/lib64 -L/usr/local/cuda-12.1/lib64/stubs -lcuda -lcudart \
-o test_54.out /home/fff/git/cutlass/examples/54_hopper_fp8_warp_specialized_gemm/54_hopper_fp8_warp_specialized_gemm.cu

Expected behavior

nvcc warning : '--device-debug (-G)' overrides '--generate-line-info (-lineinfo)'
ptxas /tmp/tmpxft_0006f883_00000000-6_54_hopper_fp8_warp_specialized_gemm.ptx, line 24246; error   : Illegal operand type to instruction 'ld'
ptxas /tmp/tmpxft_0006f883_00000000-6_54_hopper_fp8_warp_specialized_gemm.ptx, line 24246; error   : Unknown symbol '__unnamed_1_param_0'
ptxas fatal   : Ptx assembly aborted due to errors

Environment details (please complete the following information): Azure H100 VM

Additional context

• gcc

gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

• nvcc

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Wed_Nov_22_10:17:15_PST_2023
Cuda compilation tools, release 12.3, V12.3.107
Build cuda_12.3.r12.3/compiler.33567101_0

• nvidia-smi

NVIDIA-SMI 535.154.05             Driver Version: 535.154.05   CUDA Version: 12.2  

[hwu36]

Would you please double check if you are using nvcc 12.3? This issue is supposed to be fixed in 12.3 as confirmed by two github users already.

[fei-xx]

@hwu36 Confirmed it uses nvcc 12.3. I'm the user at this issue. I could compile the code there with -G, but not the above example code.

[hwu36]

thank you. i can reproduce and it is reported to nvcc team now.

[github-actions[bot]]

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.

[fei-xx]

thank you. i can reproduce and it is reported to nvcc team now.

@hwu36 kindly follow up with any updates?

[hwu36]

We have a compiler engineer actively working on it.

[Saitama10000]

same issue with nvcc 12.4 with -G, without -G it compiles and runs just fine.

ptxas /tmp/tmpxft_0000cddb_00000000-7_iota.compute_89.ptx, line 7157; error   : Illegal operand type to instruction 'ld'
ptxas /tmp/tmpxft_0000cddb_00000000-7_iota.compute_89.ptx, line 7157; error   : Unknown symbol '__unnamed_1_param_0'

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Thu_Mar_28_02:18:24_PDT_2024
Cuda compilation tools, release 12.4, V12.4.131
Build cuda_12.4.r12.4/compiler.34097967_0

EDIT: Found the issue with my case: I was running Cuda Toolkit 12.4 with cuda driver 535. Installed 550 and not it compiles without issues.

[github-actions[bot]]

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.