"/usr/local/nvidia/bin" is not available in Docker image

[somum]

In my machine I tried the command :

1. docker run -it --gpus all nvidia/cuda:11.4.0-devel-ubuntu20.04 nvidia-smi

output I got:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.57.02    Driver Version: 470.57.02    CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  On   | 00000000:0C:00.0 Off |                  N/A |
|  0%   50C    P8    32W / 350W |     19MiB / 24265MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce ...  On   | 00000000:0D:00.0 Off |                  N/A |
|  0%   44C    P8    29W / 350W |      5MiB / 24268MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+

But when I am using the docker image. Dockerfile:

FROM nvidia/cuda:11.4.0-devel-ubuntu20.04

RUN apt-get -y update
RUN apt-get -y install python3-pip

# Set the working directory to /app
WORKDIR /app

# Copy the current directory contents into the container at /app 
ADD . /app

# Install the dependencies
RUN pip install -r requirements.txt

# run the command to start uWSGI
CMD ["uwsgi", "app.ini"]

The app is running but its not using the GPU. Showing the following error:

ERROR:

flask    | 2022-01-27 08:40:08.729983: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
flask    | 2022-01-27 08:40:08.730033: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcusolver.so.10'; dlerror: libcusolver.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
flask    | 2022-01-27 08:40:08.730043: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
flask    | 2022-01-27 08:40:08.730069: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudnn.so.8'; dlerror: libcudnn.so.8: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
flask    | 2022-01-27 08:40:08.730075: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1757] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.

[hoanghonn]

Same issue here. Any updates on this?

[elezar]

The nvidia/cuda:11.4.0-devel-ubuntu20.04 does not include libcusolver or libcudnn.

You could use the cudnn images: https://hub.docker.com/r/nvidia/cuda/tags?page=1&name=11.4.0-cudnn

As a matter of interest, would the tensorflow images work directly? If not, consider checking how they install the required dependencies: https://github.com/tensorflow/tensorflow/blob/5dcfc51118817f27fad5246812d83e5dccdc5f72/tensorflow/tools/dockerfiles/dockerfiles/gpu.Dockerfile