GPU-Puzzles icon indicating copy to clipboard operation
GPU-Puzzles copied to clipboard

Collab version mismatch

Open JaLnYn opened this issue 1 year ago • 5 comments

ERROR:numba.cuda.cudadrv.driver:Call to cuLinkAddData results in CUDA_ERROR_UNSUPPORTED_PTX_VERSION
---------------------------------------------------------------------------
CudaAPIError                              Traceback (most recent call last)
[/usr/local/lib/python3.11/dist-packages/numba_cuda/numba/cuda/cudadrv/driver.py](https://localhost:8080/#) in add_ptx(self, ptx, name)
   2806         try:
-> 2807             driver.cuLinkAddData(self.handle, enums.CU_JIT_INPUT_PTX,
   2808                                  ptxbuf, len(ptx), namebuf, 0, None, None)

12 frames
CudaAPIError: [222] Call to cuLinkAddData results in CUDA_ERROR_UNSUPPORTED_PTX_VERSION

During handling of the above exception, another exception occurred:

LinkerError                               Traceback (most recent call last)
[/usr/local/lib/python3.11/dist-packages/numba_cuda/numba/cuda/cudadrv/driver.py](https://localhost:8080/#) in add_ptx(self, ptx, name)
   2808                                  ptxbuf, len(ptx), namebuf, 0, None, None)
   2809         except CudaAPIError as e:
-> 2810             raise LinkerError("%s\n%s" % (e, self.error_log))
   2811 
   2812     def add_file(self, path, kind):

LinkerError: [222] Call to cuLinkAddData results in CUDA_ERROR_UNSUPPORTED_PTX_VERSION
ptxas application ptx input, line 9; fatal   : Unsupported .version 8.5; current version is '8.4'

:(

JaLnYn avatar Jan 30 '25 12:01 JaLnYn

Yes, I got this error too. Besides, it worked fine just yesterday. It looks like the numba is upgraded to use PTX 8.5 while the Tesla T4 GPU (Compute compatibility 7.5) doesn't support PTX 8.5

PuppyQ08 avatar Jan 31 '25 05:01 PuppyQ08

It is a temporary issue in Colab. Please see this ticket: https://github.com/googlecolab/colabtools/issues/5081 When we use "numba -s", it showed the CUDA driver is 12.4 while the CUDA runtime version is 12.5. So it is not compatible. Solution: type "Use fallback runtime version" in the command palette would switch back to the old version with CUDA runtime 12.2. It works for me now.

PuppyQ08 avatar Feb 02 '25 20:02 PuppyQ08

It is a temporary issue in Colab. Please see this ticket: https://github.com/googlecolab/colabtools/issues/5081 When we use "numba -s", it showed the CUDA driver is 12.4 while the CUDA runtime version is 12.5. So it is not compatible. Solution: type "Use fallback runtime version" in the command palette would switch back to the old version with CUDA runtime 12.2. It works for me now.

It doesn't work for me anymore. It gives message like “The fallback runtime version is unavailable at this time.“

Now this command works for me !uv pip install -q --system numba-cuda==0.4.0. I saw it from https://github.com/googlecolab/colabtools/issues/5081#issuecomment-2629611179

yolk-pie-L avatar Mar 18 '25 07:03 yolk-pie-L

It is a temporary issue in Colab. Please see this ticket: https://github.com/googlecolab/colabtools/issues/5081 When we use "numba -s", it showed the CUDA driver is 12.4 while the CUDA runtime version is 12.5. So it is not compatible. Solution: type "Use fallback runtime version" in the command palette would switch back to the old version with CUDA runtime 12.2. It works for me now.

It doesn't work for me anymore. It gives message like “The fallback runtime version is unavailable at this time.“

Now this command works for me !uv pip install -q --system numba-cuda==0.4.0. I saw it from googlecolab/colabtools#5081 (comment)

I also had to paste in this line and run, as it says in the issue you linked. from numba import config config.CUDA_ENABLE_PYNVJITLINK = 1

JacobLBaum avatar Mar 25 '25 04:03 JacobLBaum

Add !uv pip install -q --system numba-cuda==0.4.0 in first cell. from numba import config config.CUDA_ENABLE_PYNVJITLINK = 1 in imports cell. Should solve it

NiharKod avatar Mar 31 '25 00:03 NiharKod