NeRF-SLAM CUDA error: an illegal memory access was encountered

I run the code in server with a rtx A6000 48GB.But i met a error.So i debug the code.I found when run the line in corr.py and get that error.Can someone help me?Thanks!

Dec 01 '22 09:12 Wjt-shift

It seems tiny-cuda-nn also have some errors: RuntimeError: CUDA error: an illegal memory access was encountered Calling frontend dtor... Could not free memory: /home/wjt/code/NeRF-SLAM/thirdparty/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:141 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/wjt/code/NeRF-SLAM/thirdparty/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:141 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/wjt/code/NeRF-SLAM/thirdparty/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:141 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/wjt/code/NeRF-SLAM/thirdparty/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:141 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/wjt/code/NeRF-SLAM/thirdparty/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:141 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/wjt/code/NeRF-SLAM/thirdparty/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:141 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/wjt/code/NeRF-SLAM/thirdparty/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:141 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/wjt/code/NeRF-SLAM/thirdparty/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:141 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/wjt/code/NeRF-SLAM/thirdparty/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:141 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/wjt/code/NeRF-SLAM/thirdparty/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:141 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/wjt/code/NeRF-SLAM/thirdparty/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:141 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/wjt/code/NeRF-SLAM/thirdparty/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:141 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/wjt/code/NeRF-SLAM/thirdparty/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:141 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/wjt/code/NeRF-SLAM/thirdparty/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:141 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/wjt/code/NeRF-SLAM/thirdparty/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:141 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/wjt/code/NeRF-SLAM/thirdparty/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:141 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/wjt/code/NeRF-SLAM/thirdparty/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:141 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/wjt/code/NeRF-SLAM/thirdparty/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:141 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/wjt/code/NeRF-SLAM/thirdparty/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:141 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/wjt/code/NeRF-SLAM/thirdparty/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:141 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/wjt/code/NeRF-SLAM/thirdparty/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:141 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/wjt/code/NeRF-SLAM/thirdparty/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:141 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/wjt/code/NeRF-SLAM/thirdparty/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:141 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/wjt/code/NeRF-SLAM/thirdparty/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:141 cudaFree(rawptr) failed with error an illegal memory access was encountered

Dec 01 '22 11:12 Wjt-shift

do you still see that error?

Jan 04 '23 19:01 ToniRV

do you still see that error?

I still have this error,and I don't know how to sloved it.I try to run code in A6000(48G memory) server and 3090(24G memory),have the same problem.And I found your image used in code is resolution of 680x1200(Replica),i resize the image with the resolution of 480x680.And i got another error:NANs in L_inv!! #18

Jan 05 '23 02:01 Wjt-shift

It is the error information.It happens in the begin of the code.I use the A6000 server run your code.Thanks for your help! ic| os.environ['CUDA_VISIBLE_DEVICES']: '0' 10:14:17 SUCCESS NGP Device set to: 0 10:14:17 INFO Loading network config from: /home/wjt/code/NeRF-SLAM/thirdparty/instant-ngp/configs/nerf/base.json 10:14:17 INFO GridEncoding: Nmin=16 b=1.51572 F=2 T=2^19 L=16 10:14:17 INFO Density model: 3--[HashGrid]-->32--[FullyFusedMLP(neurons=64,layers=3)]-->1 10:14:17 INFO Color model: 3--[Composite]-->16+16--[FullyFusedMLP(neurons=64,layers=4)]-->3 10:14:17 INFO total_encoding_params=13074912 total_network_params=10240 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 0 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 0 viz_index is empty, nothing to visualize Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 1 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 1 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 1 viz_index is empty, nothing to visualize Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 2 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 2 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 2 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 2 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 2 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 2 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 2 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 2 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 2 viz_index is empty, nothing to visualize Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 3 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 3 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 3 viz_index is empty, nothing to visualize Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 4 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 4 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 4 viz_index is empty, nothing to visualize Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 5 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 5 viz_index is empty, nothing to visualize Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 6 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 6 viz_index is empty, nothing to visualize Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 7 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 7 Traceback (most recent call last): File "/home/wjt/code/NeRF-SLAM/./examples/slam_demo.py", line 200, in run(args) File "/home/wjt/code/NeRF-SLAM/./examples/slam_demo.py", line 179, in run and (not slam or slam_module.spin())
File "/home/wjt/code/NeRF-SLAM/./examples/../pipeline/pipeline_module.py", line 101, in spin output = self.spin_once(input); File "/home/wjt/code/NeRF-SLAM/./examples/../slam/slam_module.py", line 11, in spin_once output = self.slam(input) File "/home/wjt/software/anaconda3/envs/nerfslam/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, **kwargs) File "/home/wjt/code/NeRF-SLAM/./examples/../slam/meta_slam.py", line 30, in forward output = self._frontend(batch["data"], self.state, self.delta) File "/home/wjt/code/NeRF-SLAM/./examples/../slam/vio_slam.py", line 114, in _frontend x0_visual, visual_factors, viz_out = self.visual_frontend(batch) # TODO: currently also calls BA, and global BA File "/home/wjt/software/anaconda3/envs/nerfslam/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, **kwargs) File "/home/wjt/code/NeRF-SLAM/./examples/../slam/visual_frontends/visual_frontend.py", line 326, in forward self.__initialize() File "/home/wjt/code/NeRF-SLAM/./examples/../slam/visual_frontends/visual_frontend.py", line 647, in __initialize self.add_neighborhood_factors(kf0=0, kf1=self.kf_idx, radius=3) File "/home/wjt/code/NeRF-SLAM/./examples/../slam/visual_frontends/visual_frontend.py", line 708, in add_neighborhood_factors self.add_factors(ii[keep], jj[keep]) File "/home/wjt/software/anaconda3/envs/nerfslam/lib/python3.9/site-packages/torch/amp/autocast_mode.py", line 12, in decorate_autocast return func(*args, **kwargs) File "/home/wjt/code/NeRF-SLAM/./examples/../slam/visual_frontends/visual_frontend.py", line 857, in add_factors target, _, _ = self.reproject(ii, jj) File "/home/wjt/code/NeRF-SLAM/./examples/../slam/visual_frontends/visual_frontend.py", line 916, in reproject pops.projective_transform(Gs, self.cam0_idepths[None], self.cam0_intrinsics[None], ii, jj, cam_T_body=cam_T_body, jacobian=jacobian) File "/home/wjt/code/NeRF-SLAM/./examples/../networks/geom/projective_ops.py", line 105, in projective_transform X0, Jz = iproj(depths[:,ii], intrinsics[:,ii], jacobian=jacobian) File "/home/wjt/code/NeRF-SLAM/./examples/../networks/geom/projective_ops.py", line 26, in iproj torch.arange(ht).to(disps.device).float(), RuntimeError: CUDA error: an illegal memory access was encountered CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Calling frontend dtor... Could not free memory: /home/wjt/code/NeRF-SLAM/thirdparty/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:141 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/wjt/code/NeRF-SLAM/thirdparty/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:141 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/wjt/code/NeRF-SLAM/thirdparty/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:141 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/wjt/code/NeRF-SLAM/thirdparty/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:141 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/wjt/code/NeRF-SLAM/thirdparty/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:141 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/wjt/code/NeRF-SLAM/thirdparty/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:141 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/wjt/code/NeRF-SLAM/thirdparty/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:141 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/wjt/code/NeRF-SLAM/thirdparty/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:141 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/wjt/code/NeRF-SLAM/thirdparty/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:141 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/wjt/code/NeRF-SLAM/thirdparty/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:141 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/wjt/code/NeRF-SLAM/thirdparty/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:141 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/wjt/code/NeRF-SLAM/thirdparty/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:141 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/wjt/code/NeRF-SLAM/thirdparty/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:141 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/wjt/code/NeRF-SLAM/thirdparty/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:141 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/wjt/code/NeRF-SLAM/thirdparty/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:141 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/wjt/code/NeRF-SLAM/thirdparty/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:141 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/wjt/code/NeRF-SLAM/thirdparty/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:141 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/wjt/code/NeRF-SLAM/thirdparty/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:141 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/wjt/code/NeRF-SLAM/thirdparty/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:141 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/wjt/code/NeRF-SLAM/thirdparty/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:141 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/wjt/code/NeRF-SLAM/thirdparty/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:141 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/wjt/code/NeRF-SLAM/thirdparty/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:141 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/wjt/code/NeRF-SLAM/thirdparty/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:141 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/wjt/code/NeRF-SLAM/thirdparty/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:141 cudaFree(rawptr) failed with error an illegal memory access was encountered

Jan 05 '23 02:01 Wjt-shift