RuntimeError: CUDA error: out of memory

[away-back]

May I ask which kind of GPU for project training? I use the NVIDIA RTX A6000 and attache 47.5 memory,but show out of memory.

The follow is my running error detail： [2025-02-18 16:41:01,068][smart_tree.model.train][INFO] - Train Dataset Size: 480 [2025-02-18 16:41:01,068][smart_tree.model.train][INFO] - Validation Dataset Size: 60 [2025-02-18 16:41:01,068][smart_tree.model.train][INFO] - Test Dataset Size: 60 /home/huangyongchang/OpenProject/smart-tree/smart_tree/model/train.py:214: FutureWarning: torch.cuda.amp.autocast(args...) is deprecated. Please use torch.amp.autocast('cuda', args...) instead. amp_ctx = torch.cuda.amp.autocast() if cfg.fp16 else contextlib.nullcontext() /home/huangyongchang/OpenProject/smart-tree/smart_tree/model/train.py:215: FutureWarning: torch.cuda.amp.GradScaler(args...) is deprecated. Please use torch.amp.GradScaler('cuda', args...) instead. scaler = torch.cuda.amp.grad_scaler.GradScaler() Epoch: 0%| | 0/1 [00:27<?, ?it/s] Error executing job with overrides: [] Traceback (most recent call last): File "/home/huangyongchang/OpenProject/smart-tree/smart_tree/model/train.py", line 232, in main val_tracker = eval_epoch( File "/root/anaconda3/envs/smart-tree/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context return func(*args, **kwargs) File "/home/huangyongchang/OpenProject/smart-tree/smart_tree/model/train.py", line 74, in eval_epoch for sp_input, targets, mask, fn in tqdm( File "/root/anaconda3/envs/smart-tree/lib/python3.10/site-packages/tqdm/std.py", line 1181, in iter for obj in iterable: File "/home/huangyongchang/OpenProject/smart-tree/smart_tree/model/helper.py", line 14, in get_batch for (feats, target_feats), coords, mask, filenames in dataloader: File "/root/anaconda3/envs/smart-tree/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 701, in next data = self._next_data() File "/root/anaconda3/envs/smart-tree/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 757, in _next_data data = self._dataset_fetcher.fetch(index) # may raise StopIteration File "/root/anaconda3/envs/smart-tree/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 52, in fetch data = [self.dataset[idx] for idx in possibly_batched_index] File "/root/anaconda3/envs/smart-tree/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 52, in data = [self.dataset[idx] for idx in possibly_batched_index] File "/home/huangyongchang/OpenProject/smart-tree/smart_tree/dataset/dataset.py", line 74, in getitem cld = self.load(filename) File "/home/huangyongchang/OpenProject/smart-tree/smart_tree/dataset/dataset.py", line 68, in load self.cache[filename] = self.load_cloud(filename).pin_memory() File "/home/huangyongchang/OpenProject/smart-tree/smart_tree/data_types/cloud.py", line 112, in pin_memory rgb = self.rgb.pin_memory() if self.rgb is not None else None RuntimeError: CUDA error: out of memory CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1 Compile with TORCH_USE_CUDA_DSA to enable device-side assertions. `