Cuda execution failed with error 2

[tuan97anton]

I run : ./tools/scripts/train.sh train1 I get error: /spconv/src/spconv/indice.cu 125 cuda execution failed with error 2

My PC have only 1 GPU NVIDIA GeForce RTX 2080 SUPER - 8GB My software : CUDA Version 10.0.130 cudnn : 7603 pytorch : 1.3.0 spconv : 1.0 Cmake: cmake version 3.21.3 ninjia: 1.9.0 Python 3.6.8

I installed without docker or with docker , I got the same error /spconv/src/spconv/indice.cu 125 cuda execution failed with error 2 DO you have idea to resolve error above ? Thank you !

[tuan97anton]

My Docker that I installes Det3D, I use extention Remote Development of VisualCode : link

1. Dockerfile

FROM pytorch/pytorch:1.1.0-cuda10.0-cudnn7.5-devel
RUN apt-get update && apt-get install -y libboost-all-dev\
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*
RUN pip install --upgrade pip

RUN pip install Pillow==6.1
RUN apt remove cmake
RUN pip install cmake --upgrade
RUN pip install setuptools==39.1.0
RUN pip install torch==1.3.0 torchvision==0.4.1

# spconv 
WORKDIR /root
RUN git clone https://github.com/poodarchu/spconv.git --recursive /spconv
RUN cd /spconv &&  python setup.py bdist_wheel
RUN cd /spconv/dist && pip install *

# nuscenes-devkit
RUN pip install nuscenes-devkit

# Det3D
RUN git clone https://github.com/poodarchu/Det3D.git /Det3D
WORKDIR /Det3D
RUN pip install -r requirements.txt
# RUN python setup.py build develop

RUN apt-get update && apt-get install -y libgl1 libgtk2.0-dev

2. devcontainer.json

{
	"name": "det3d",
	"build": {
		"dockerfile": "Dockerfile",
		"context": "..",
	},

	// Set *default* container specific settings.json values on container create.
	"settings": { 
		"terminal.integrated.profiles.linux": {
			"bash": {
				"path": "/bin/bash"
			}
		},
		"terminal.integrated.defaultProfile.linux": "bash",
		"python.pythonPath": "/usr/local/bin/python",
		"python.languageServer": "Pylance",
		"python.linting.enabled": true,
		"python.linting.pylintEnabled": true,
		"python.formatting.autopep8Path": "/usr/local/py-utils/bin/autopep8",
		"python.formatting.blackPath": "/usr/local/py-utils/bin/black",
		"python.formatting.yapfPath": "/usr/local/py-utils/bin/yapf",
		"python.linting.banditPath": "/usr/local/py-utils/bin/bandit",
		"python.linting.flake8Path": "/usr/local/py-utils/bin/flake8",
		"python.linting.mypyPath": "/usr/local/py-utils/bin/mypy",
		"python.linting.pycodestylePath": "/usr/local/py-utils/bin/pycodestyle",
		"python.linting.pydocstylePath": "/usr/local/py-utils/bin/pydocstyle",
		"python.linting.pylintPath": "/usr/local/py-utils/bin/pylint"
	},

	"runArgs": [
		"--name", "my_det3d",
		"--gpus=all", 
		"--shm-size=5G",
		"-v", "/home/ub18-2/Documents/kitti/:/workspaces/my_det3d/data_kitti/",
		"-v", "/home/ub18-2/Documents/nuscenes/v1.0-mini/:/workspaces/my_det3d/data_nusc/"
	],
	
	"extensions": [
		"ms-python.python",
		"ms-python.vscode-pylance"
	],

}

[poodarchu]

can you try to compile with pytorch 1.6+?

[www04576]

i got the same problem. have you solved it? thank you.

[ShaohuaDong2021]

Same question, have you solved it?

[lacie-life]

i have the same problem. have you solved it? thank you.

Edited: I fixed this problem by using spconv 1.2.1