NVIDIA
BUG: migrating from 12.6 to 12.8 breaks Python import
On Linux Ubuntu 22.08, I've recently updated the cuda-python package from 12.6 to 12.8.0 in my Python project and when I'm now trying to do:
from cuda import cuda, cudart, nvrtc
I'm getting the error message:
ImportError: cannot import name 'cuda' from 'cuda' (unknown location)
From the examples in the cuda.bindings package, I see that the same import command is still used.
Since, I'm using RDMA, I haven't tried this on Windows.
I cannot reproduce it (obviously, since our CI definitely got this tested), so...
- How did you install cuda-python? (pip? conda? other means?)
- Could you show the output of
pip list? And if you use conda environment, alsoconda list?
I have an existing virtual environment with cuda-python installed and I simply did a:
pip install cuda-python==12.8
The pip list command outputs:
Package Version
----------------- -----------
contourpy 1.3.1
cuda-bindings 12.8.0
cuda-core 0.1.1
cuda-python 12.8.0
cupy-cuda12x 13.3.0
cycler 0.12.1
fastrlock 0.8.2
flexcache 0.3
flexparser 0.4
fonttools 4.55.0
freetype-py 2.5.1
h5py 3.13.0
hsluv 5.0.4
kiwisolver 1.4.7
matplotlib 3.9.2
numpy 2.2.3
packaging 24.2
pillow 11.0.0
Pint 0.24.4
pip 22.0.2
platformdirs 4.3.6
pyparsing 3.2.0
PyQt5 5.15.11
PyQt5-Qt5 5.15.15
PyQt5_sip 12.15.0
python-dateutil 2.9.0.post0
scipy 1.14.1
setuptools 59.6.0
six 1.16.0
spcm 1.5.2
spcm_core 1.0.2
typing_extensions 4.12.2
vispy 0.14.3
Forgot to ask, what's the Python version?
Also, with this broken environment, could you check what happens if you do
from cuda.bindings import driver as cuda
? (Same for cudart and nvrtc)
The Python version is 3.10.12 and with the command from cuda.bindings import driver as cuda I get:
Exception has occurred: ModuleNotFoundError No module named 'cuda.bindings' File "...", line 13, in
from cuda.bindings import driver as cuda ModuleNotFoundError: No module named 'cuda.bindings'
ps. small correction I'm using Ubuntu LTS 22.04
Okay, I've narrowed it down. It has to do with the migration from 12.6 to 12.8, because when setting up a new Virtual Environment with the same packages both imports work:
from cuda import cuda
from cuda.bindings import driver as cuda2
Here is the result from the pip list command in the new environment:
Package Version
----------------- -----------
contourpy 1.3.1
cuda-bindings 12.8.0
cuda-python 12.8.0
cupy-cuda12x 13.3.0
cycler 0.12.1
fastrlock 0.8.3
flexcache 0.3
flexparser 0.4
fonttools 4.56.0
freetype-py 2.5.1
h5py 3.13.0
hsluv 5.0.4
kiwisolver 1.4.8
matplotlib 3.10.0
numpy 2.2.3
packaging 24.2
pillow 11.1.0
Pint 0.24.4
pip 25.0.1
platformdirs 4.3.6
pyparsing 3.2.1
PyQt5 5.15.11
PyQt5-Qt5 5.15.16
PyQt5_sip 12.17.0
python-dateutil 2.9.0.post0
scipy 1.15.2
six 1.17.0
spcm 1.5.2
spcm_core 1.0.2
typing_extensions 4.12.2
vispy 0.14.3
It seems there is a weird cleanup issue going on. With the following steps I hit an error, though different from yours
- Create a fresh env
-
pip install "cuda-python==12.6.*" -
pip install "cuda-python==12.8"
$ python
Python 3.12.9 | packaged by conda-forge | (main, Feb 14 2025, 08:00:06) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from cuda import cuda, cudart, nvrtc
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'cuda'
@spec-benno could you try a complete removal and then reinstall?
-
pip uninstall -y cuda-python cuda-bindings cuda-core -
pip install cuda-python==12.8 cuda-core
and see if it is fixed?
we were thinking the same thing. Setting up a new environment does the trick, however migrating is still a problem somehow
It seems there is a weird cleanup issue going on. With the following steps I hit an error, though different from yours
- Create a fresh env
pip install "cuda-python==12.6.*"pip install "cuda-python==12.8"$ python Python 3.12.9 | packaged by conda-forge | (main, Feb 14 2025, 08:00:06) [GCC 13.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> from cuda import cuda, cudart, nvrtc Traceback (most recent call last): File "<stdin>", line 1, in <module> ModuleNotFoundError: No module named 'cuda'@spec-benno could you try a complete removal and then reinstall?
pip uninstall -y cuda-python cuda-bindings cuda-corepip install cuda-python==12.8 cuda-coreand see if it is fixed?
I also did the uninstall and reinstall and this works as well, for both import methods. The error you're seeing I've seen as well.
- Create a fresh env
pip install "cuda-python==12.6.*"pip install "cuda-python==12.8"
Reproducer is to change the step 2 above to pip install "cuda-python==12.6", which pip interprets as installing 12.6.0 (instead of 12.6.2.post1). Then I hit the same error:
$ python
Python 3.12.9 | packaged by conda-forge | (main, Feb 14 2025, 08:00:06) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from cuda import cuda, cudart, nvrtc
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: cannot import name 'cuda' from 'cuda' (unknown location)
This seems like another unexpected issue due to layout change that caused some trouble earlier (ex: #226). I don't think there's any fix we can do but I do want to know what caused package updates dirty (since the old version is uninstalled first before the new version is installed)...
@bdice reports offline that this still happens when updating from 12.6.2.post1 to 12.9.0... 😵💫
Marking this issue as a "won't fix." Unfortunately there does not seem to exist any workaround to our knowledge, other than kindly asking users to do a clean installation (uninstall and then install) instead of updating. I'll add this to known issues shortly.