alphageometry
alphageometry copied to clipboard
google-deepmind
Could not find TensorRT, resulting inference without GPU
After installation, I was able to run inference in CPU mode, but it fails to run with GPU,
- I have tried to install
tensorrtwith pip, - I have added its path to
LD_LIBARARY_PATH, - I have try to link the library with other name,
but I still fails to run tensorflow with GPU, not sure what am I missing here.
Here are the settings for my setup, I am using A100 with CUDA 11.8
(gpugeo) $ python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
2024-04-04 20:13:57.014520: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-04-04 20:13:58.875746: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-04-04 20:14:03.631155: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1960] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
[]
$ cat /etc/os-release
NAME="Rocky Linux"
VERSION="8.6 (Green Obsidian)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="8.6"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Rocky Linux 8.6 (Green Obsidian)"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:rocky:rocky:8:GA"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
ROCKY_SUPPORT_PRODUCT="Rocky Linux"
ROCKY_SUPPORT_PRODUCT_VERSION="8"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8"
(gpugeo) [APAC-TY@b05u17g geo]$ nvidia-smi
Thu Apr 4 20:14:14 2024
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 520.61.05 Driver Version: 520.61.05 CUDA Version: 11.8 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA A100-SXM... On | 00000000:26:00.0 Off | 0 |
| N/A 24C P0 60W / 400W | 31MiB / 81920MiB | 0% E. Process |
| | | Disabled |
+-------------------------------+----------------------+----------------------+
| 1 NVIDIA A100-SXM... On | 00000000:2C:00.0 Off | 0 |
| N/A 31C P0 206W / 400W | 3041MiB / 81920MiB | 46% E. Process |
| | | Disabled |
+-------------------------------+----------------------+----------------------+
| 2 NVIDIA A100-SXM... On | 00000000:66:00.0 Off | 0 |
| N/A 37C P0 353W / 400W | 54879MiB / 81920MiB | 89% E. Process |
| | | Disabled |
+-------------------------------+----------------------+----------------------+
| 3 NVIDIA A100-SXM... On | 00000000:6B:00.0 Off | 0 |
| N/A 24C P0 61W / 400W | 0MiB / 81920MiB | 0% E. Process |
| | | Disabled |
+-------------------------------+----------------------+----------------------+
| 4 NVIDIA A100-SXM... On | 00000000:A4:00.0 Off | 0 |
| N/A 23C P0 60W / 400W | 0MiB / 81920MiB | 0% Default |
| | | Disabled |
+-------------------------------+----------------------+----------------------+
| 5 NVIDIA A100-SXM... On | 00000000:A9:00.0 Off | 0 |
| N/A 24C P0 59W / 400W | 0MiB / 81920MiB | 0% Default |
| | | Disabled |
+-------------------------------+----------------------+----------------------+
| 6 NVIDIA A100-SXM... On | 00000000:E1:00.0 Off | 0 |
| N/A 23C P0 61W / 400W | 0MiB / 81920MiB | 0% Default |
| | | Disabled |
+-------------------------------+----------------------+----------------------+
| 7 NVIDIA A100-SXM... On | 00000000:E7:00.0 Off | 0 |
| N/A 23C P0 61W / 400W | 0MiB / 81920MiB | 0% Default |
| | | Disabled |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 3812720 C nvidia-cuda-mps-server 28MiB |
| 1 N/A N/A 3744900 M+C lmp_mpi 3010MiB |
| 1 N/A N/A 3744976 C nvidia-cuda-mps-server 28MiB |
| 2 N/A N/A 3772436 M+C python 54848MiB |
| 2 N/A N/A 3772443 C nvidia-cuda-mps-server 28MiB |
+-----------------------------------------------------------------------------+
(gpugeo) $ python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
2024-04-04 20:14:20.371825: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-04-04 20:14:22.177861: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-04-04 20:14:26.929802: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1960] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
(gpugeo) $ conda list
# packages in environment at /work/APAC-TY/miniconda3/envs/gpugeo:
#
# Name Version Build Channel
_libgcc_mutex 0.1 main defaults
_openmp_mutex 5.1 1_gnu defaults
absl-py 1.4.0 pypi_0 pypi
array-record 0.4.1 pypi_0 pypi
astunparse 1.6.3 pypi_0 pypi
babel 2.13.0 pypi_0 pypi
beautifulsoup4 4.12.2 pypi_0 pypi
bzip2 1.0.8 h5eee18b_5 defaults
ca-certificates 2024.3.11 h06a4308_0 defaults
cached-property 1.5.2 pypi_0 pypi
cachetools 5.3.1 pypi_0 pypi
certifi 2023.7.22 pypi_0 pypi
charset-normalizer 3.3.0 pypi_0 pypi
chex 0.1.7 pypi_0 pypi
click 8.1.7 pypi_0 pypi
clu 0.0.7 pypi_0 pypi
colorama 0.4.6 pypi_0 pypi
commonmark 0.9.1 pypi_0 pypi
contextlib2 21.6.0 pypi_0 pypi
contourpy 1.1.1 pypi_0 pypi
cuda 11.8.0 0 nvidia/label/cuda-11.8.0
cuda-cccl 11.8.89 0 nvidia/label/cuda-11.8.0
cuda-command-line-tools 11.8.0 0 nvidia/label/cuda-11.8.0
cuda-compiler 11.8.0 0 nvidia/label/cuda-11.8.0
cuda-cudart 11.8.89 0 nvidia/label/cuda-11.8.0
cuda-cudart-dev 11.8.89 0 nvidia/label/cuda-11.8.0
cuda-cuobjdump 11.8.86 0 nvidia/label/cuda-11.8.0
cuda-cupti 11.8.87 0 nvidia/label/cuda-11.8.0
cuda-cuxxfilt 11.8.86 0 nvidia/label/cuda-11.8.0
cuda-demo-suite 11.8.86 0 nvidia/label/cuda-11.8.0
cuda-documentation 11.8.86 0 nvidia/label/cuda-11.8.0
cuda-driver-dev 11.8.89 0 nvidia/label/cuda-11.8.0
cuda-gdb 11.8.86 0 nvidia/label/cuda-11.8.0
cuda-libraries 11.8.0 0 nvidia/label/cuda-11.8.0
cuda-libraries-dev 11.8.0 0 nvidia/label/cuda-11.8.0
cuda-memcheck 11.8.86 0 nvidia/label/cuda-11.8.0
cuda-nsight 11.8.86 0 nvidia/label/cuda-11.8.0
cuda-nsight-compute 11.8.0 0 nvidia/label/cuda-11.8.0
cuda-nvcc 11.8.89 0 nvidia/label/cuda-11.8.0
cuda-nvdisasm 11.8.86 0 nvidia/label/cuda-11.8.0
cuda-nvml-dev 11.8.86 0 nvidia/label/cuda-11.8.0
cuda-nvprof 11.8.87 0 nvidia/label/cuda-11.8.0
cuda-nvprune 11.8.86 0 nvidia/label/cuda-11.8.0
cuda-nvrtc 11.8.89 0 nvidia/label/cuda-11.8.0
cuda-nvrtc-dev 11.8.89 0 nvidia/label/cuda-11.8.0
cuda-nvtx 11.8.86 0 nvidia/label/cuda-11.8.0
cuda-nvvp 11.8.87 0 nvidia/label/cuda-11.8.0
cuda-profiler-api 11.8.86 0 nvidia/label/cuda-11.8.0
cuda-runtime 11.8.0 0 nvidia/label/cuda-11.8.0
cuda-sanitizer-api 11.8.86 0 nvidia/label/cuda-11.8.0
cuda-toolkit 11.8.0 0 nvidia/label/cuda-11.8.0
cuda-tools 11.8.0 0 nvidia/label/cuda-11.8.0
cuda-visual-tools 11.8.0 0 nvidia/label/cuda-11.8.0
cycler 0.12.1 pypi_0 pypi
dm-tree 0.1.8 pypi_0 pypi
docstring-parser 0.15 pypi_0 pypi
editdistance 0.6.2 pypi_0 pypi
etils 1.5.1 pypi_0 pypi
filelock 3.12.4 pypi_0 pypi
flatbuffers 23.5.26 pypi_0 pypi
flax 0.5.3 pypi_0 pypi
fonttools 4.43.1 pypi_0 pypi
fsspec 2023.9.2 pypi_0 pypi
future 0.18.3 pypi_0 pypi
gast 0.4.0 pypi_0 pypi
gdown 4.7.1 pypi_0 pypi
gds-tools 1.4.0.31 0 nvidia/label/cuda-11.8.0
gin 0.1.6 pypi_0 pypi
gin-config 0.5.0 pypi_0 pypi
google-auth 2.23.3 pypi_0 pypi
google-auth-oauthlib 1.0.0 pypi_0 pypi
google-pasta 0.2.0 pypi_0 pypi
googleapis-common-protos 1.61.0 pypi_0 pypi
grpcio 1.59.0 pypi_0 pypi
h5py 3.10.0 pypi_0 pypi
huggingface-hub 0.17.3 pypi_0 pypi
idna 3.4 pypi_0 pypi
immutabledict 3.0.0 pypi_0 pypi
importlib-resources 6.1.0 pypi_0 pypi
jax 0.4.6 pypi_0 pypi
jaxlib 0.4.6 pypi_0 pypi
joblib 1.3.2 pypi_0 pypi
keras 2.13.1 pypi_0 pypi
kiwisolver 1.4.5 pypi_0 pypi
ld_impl_linux-64 2.38 h1181459_1 defaults
libclang 16.0.6 pypi_0 pypi
libcublas 11.11.3.6 0 nvidia/label/cuda-11.8.0
libcublas-dev 11.11.3.6 0 nvidia/label/cuda-11.8.0
libcufft 10.9.0.58 0 nvidia/label/cuda-11.8.0
libcufft-dev 10.9.0.58 0 nvidia/label/cuda-11.8.0
libcufile 1.4.0.31 0 nvidia/label/cuda-11.8.0
libcufile-dev 1.4.0.31 0 nvidia/label/cuda-11.8.0
libcurand 10.3.0.86 0 nvidia/label/cuda-11.8.0
libcurand-dev 10.3.0.86 0 nvidia/label/cuda-11.8.0
libcusolver 11.4.1.48 0 nvidia/label/cuda-11.8.0
libcusolver-dev 11.4.1.48 0 nvidia/label/cuda-11.8.0
libcusparse 11.7.5.86 0 nvidia/label/cuda-11.8.0
libcusparse-dev 11.7.5.86 0 nvidia/label/cuda-11.8.0
libffi 3.3 he6710b0_2 defaults
libgcc-ng 11.2.0 h1234567_1 defaults
libgomp 11.2.0 h1234567_1 defaults
libnpp 11.8.0.86 0 nvidia/label/cuda-11.8.0
libnpp-dev 11.8.0.86 0 nvidia/label/cuda-11.8.0
libnvjpeg 11.9.0.86 0 nvidia/label/cuda-11.8.0
libnvjpeg-dev 11.9.0.86 0 nvidia/label/cuda-11.8.0
libstdcxx-ng 11.2.0 h1234567_1 defaults
libuuid 1.41.5 h5eee18b_0 defaults
lxml 4.9.3 pypi_0 pypi
markdown 3.5 pypi_0 pypi
markupsafe 2.1.3 pypi_0 pypi
matplotlib 3.7.0 pypi_0 pypi
mesh-tensorflow 0.1.21 pypi_0 pypi
ml-collections 0.1.1 pypi_0 pypi
msgpack 1.0.7 pypi_0 pypi
ncurses 6.4 h6a678d5_0 defaults
nltk 3.8.1 pypi_0 pypi
nsight-compute 2022.3.0.22 0 nvidia/label/cuda-11.8.0
numpy 1.23.5 pypi_0 pypi
nvidia-cublas-cu12 12.4.5.8 pypi_0 pypi
nvidia-cuda-runtime-cu12 12.4.127 pypi_0 pypi
nvidia-cudnn-cu12 9.0.0.312 pypi_0 pypi
oauthlib 3.2.2 pypi_0 pypi
openssl 1.1.1w h7f8727e_0 defaults
opt-einsum 3.3.0 pypi_0 pypi
optax 0.1.7 pypi_0 pypi
packaging 23.2 pypi_0 pypi
pandas 2.1.1 pypi_0 pypi
pillow 10.0.1 pypi_0 pypi
pip 24.0 pypi_0 pypi
portalocker 2.8.2 pypi_0 pypi
promise 2.3 pypi_0 pypi
protobuf 3.20.3 pypi_0 pypi
psutil 5.9.5 pypi_0 pypi
pyasn1 0.5.0 pypi_0 pypi
pyasn1-modules 0.3.0 pypi_0 pypi
pyglove 0.4.3 pypi_0 pypi
pygments 2.16.1 pypi_0 pypi
pyparsing 3.1.1 pypi_0 pypi
pysocks 1.7.1 pypi_0 pypi
python 3.10.0 h12debd9_5 defaults
python-dateutil 2.8.2 pypi_0 pypi
pytz 2023.3.post1 pypi_0 pypi
pyyaml 6.0.1 pypi_0 pypi
readline 8.2 h5eee18b_0 defaults
regex 2023.10.3 pypi_0 pypi
requests 2.31.0 pypi_0 pypi
requests-oauthlib 1.3.1 pypi_0 pypi
rich 11.2.0 pypi_0 pypi
rouge-score 0.1.2 pypi_0 pypi
rsa 4.9 pypi_0 pypi
sacrebleu 2.3.1 pypi_0 pypi
safetensors 0.4.0 pypi_0 pypi
scikit-learn 1.3.1 pypi_0 pypi
scipy 1.10.0 pypi_0 pypi
sentencepiece 0.1.99 pypi_0 pypi
seqio 0.0.18 pypi_0 pypi
seqio-nightly 0.0.17.dev20231013 pypi_0 pypi
setuptools 68.2.2 py310h06a4308_0 defaults
six 1.16.0 pypi_0 pypi
soupsieve 2.5 pypi_0 pypi
sqlite 3.41.2 h5eee18b_0 defaults
t5 0.9.4 pypi_0 pypi
tabulate 0.9.0 pypi_0 pypi
tensorboard 2.13.0 pypi_0 pypi
tensorboard-data-server 0.7.1 pypi_0 pypi
tensorflow 2.13.0 pypi_0 pypi
tensorflow-datasets 4.9.3 pypi_0 pypi
tensorflow-estimator 2.13.0 pypi_0 pypi
tensorflow-hub 0.15.0 pypi_0 pypi
tensorflow-io-gcs-filesystem 0.34.0 pypi_0 pypi
tensorflow-metadata 1.14.0 pypi_0 pypi
tensorflow-text 2.13.0 pypi_0 pypi
tensorrt-libs 8.6.1 pypi_0 pypi
tensorstore 0.1.45 pypi_0 pypi
termcolor 2.3.0 pypi_0 pypi
tfds-nightly 4.9.2.dev202308090034 pypi_0 pypi
threadpoolctl 3.2.0 pypi_0 pypi
tk 8.6.12 h1ccaba5_0 defaults
tokenizers 0.14.1 pypi_0 pypi
toml 0.10.2 pypi_0 pypi
toolz 0.12.0 pypi_0 pypi
tqdm 4.66.1 pypi_0 pypi
transformers 4.34.0 pypi_0 pypi
typing-extensions 4.5.0 pypi_0 pypi
tzdata 2023.3 pypi_0 pypi
urllib3 2.0.6 pypi_0 pypi
werkzeug 3.0.0 pypi_0 pypi
wheel 0.41.2 py310h06a4308_0 defaults
wrapt 1.15.0 pypi_0 pypi
xz 5.4.6 h5eee18b_0 defaults
zipp 3.17.0 pypi_0 pypi
zlib 1.2.13 h5eee18b_0 defaults
[]
Happens to me as well, seems no trivial solution because the package dependency on cuda is pretty outdated
Can you please provide the version of tensor-RT used for the model?
You can use the GPU version of jax to call the GPU, but ddar can only run on the CPU.
Hi, I also encountered the "no GPU" issue, but I discovered that it was related to JAX and jaxlib. To resolve this, you need to visit https://storage.googleapis.com/jax-releases/jax_cuda_releases.html . From there, locate the version of jaxlib that matches your CUDA, cuDNN, and Python versions (the file descriptions indicate the Python version, such as cp310). Only with the correct version of jaxlib will your GPU be properly utilized!
Make sure to carefully check the compatibility between your system's CUDA, cuDNN, and Python versions to ensure everything works as expected.
