"Could not load tynycudann: Unknown compute capability"

[alepot55]

Describe the bug It seems that driver version 470.199.02 can't support cuda 11.8, I tried everything except updating driver (I can't)

user@85876e8bbb9e:/workspace$ nvidia-smi
Sat Mar  9 11:38:01 2024       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.199.02   Driver Version: 470.199.02   CUDA Version: 11.8     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0  On |                  N/A |
| 26%   35C    P8    13W / 250W |    434MiB /  7979MiB |      2%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+
user@85876e8bbb9e:/workspace$ ns-train nerfacto
/home/user/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:107: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 804: forward compatibility was attempted on non supported HW (Triggered internally at ../c10/cuda/CUDAFunctions.cpp:109.)
  return torch._C._cuda_getDeviceCount() > 0
Could not load tinycudann: Unknown compute capability. Ensure PyTorch with CUDA support is installed.

I'm using docker, did anyone found a way to make it work withoud updating driver?

[NicolasBurny]

Hi, for anyone having this issue, check the following : https://github.com/NVIDIA/nvidia-container-toolkit/issues/520 It might help you solve the issue.