cuda-python icon indicating copy to clipboard operation
cuda-python copied to clipboard

`cuda.cudart.getLocalRuntimeVersion()` raises `RuntimeError: Failed to dlopen libcudart.so.12`

Open Matt711 opened this issue 1 year ago • 6 comments

Is this a bug? getLocalRuntimeVersion() fails for me in cuda 11.8 environment. I'm asking because I see that the API call is in the cuda-python 11.8 release notes.

In the source code, we're hard coding libcudart.so.12. Is that right?

Repro

In [1]: from cuda import cudart

In [2]: cudart.getLocalRuntimeVersion()
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[2], line 1
----> 1 cudart.getLocalRuntimeVersion()

File ~/.conda/envs/rapids/lib/python3.11/site-packages/cuda/cudart.pyx:24961, in cuda.cudart.getLocalRuntimeVersion()

File ~/.conda/envs/rapids/lib/python3.11/site-packages/cuda/ccudart.pyx:2365, in cuda.ccudart.getLocalRuntimeVersion()

File ~/.conda/envs/rapids/lib/python3.11/site-packages/cuda/_lib/ccudart/ccudart.pyx:2121, in cuda._lib.ccudart.ccudart._getLocalRuntimeVersion()

RuntimeError: Failed to dlopen libcudart.so.12

Matt711 avatar Sep 10 '24 15:09 Matt711

xref rmm/1675

Matt711 avatar Sep 10 '24 15:09 Matt711

It seems to be a backport mistake that we should fix: https://github.com/NVIDIA/cuda-python/blob/64cc9ae081ba405972b0472cd1fd35b919c455fc/cuda/_lib/ccudart/ccudart.pyx.in#L2451-L2457 @Matt711 how urgent is this?

leofang avatar Sep 10 '24 15:09 leofang

@Matt711 how urgent is this?

Not urgent. We already have a workaround using numba.cuda. I also don't mind working on this @leofang, if you could point me in the right direction.

Matt711 avatar Sep 10 '24 16:09 Matt711

Thanks, @Matt711. The offending code that I linked to above is from the 11.8.x branch, so ideally we can just fix the lines referencing libcudart.so.12 to .11. But we're transitioning to a new development/release process so let me check with @vzhurba01 later today first, and get back to you later.

leofang avatar Sep 10 '24 16:09 leofang

@Matt711 we discussed and will try to get a new 11.8.x release out next week, with this bug fixed and perhaps also #75 backported.

leofang avatar Sep 10 '24 21:09 leofang

Thanks @leofang

Matt711 avatar Sep 11 '24 04:09 Matt711

@leofang, @vzhurba01 did this backport/release occur?

wence- avatar Sep 30 '24 09:09 wence-

Not yet. The wheels and conda packages are currently going through pre-release validation. I'll update this issue once posting is complete.

vzhurba01 avatar Sep 30 '24 20:09 vzhurba01

FYI I've updated the repo with the fix under the patch release 11.8.4 (tag v11.8.4).

I created new issue #139 to track the wheels/conda uploads for this patch release. I'm thinking of keeping this current issue open though until they are uploaded, and then give a notice before closing.

vzhurba01 avatar Oct 07 '24 19:10 vzhurba01

https://pypi.org/project/cuda-python/11.8.4/ https://anaconda.org/nvidia/cuda-python/files?version=11.8.4

Issue #139 is now complete as both PYPI and Conda (nvidia channel) are now updated with the 11.8.4 patch. Thank you all for your patience.

vzhurba01 avatar Nov 04 '24 22:11 vzhurba01