RuntimeError: CUDA out of memory

[hongrui16]

Traceback (most recent call last): File "/home/hongrui/anaconda3/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 59, in _wrap fn(i, *args) File "/home/hongrui/project/metro_pro/instance-segmentation-pytorch/code/train_engine.py", line 60, in main_worker worker.validation(epoch) File "/home/hongrui/project/metro_pro/instance-segmentation-pytorch/code/train_worker.py", line 219, in validation mb_out_metrics, loss, outputs = self.forward( File "/home/hongrui/project/metro_pro/instance-segmentation-pytorch/code/train_worker.py", line 399, in forward disc_cost = self.criterion_discriminative( File "/home/hongrui/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl return forward_call(*input, **kwargs) File "/home/hongrui/project/metro_pro/instance-segmentation-pytorch/code/lib/losses/discriminative.py", line 180, in forward return discriminative_loss(input, target, n_objects, max_n_objects, File "/home/hongrui/project/metro_pro/instance-segmentation-pytorch/code/lib/losses/discriminative.py", line 147, in discriminative_loss cluster_means = calculate_means( File "/home/hongrui/project/metro_pro/instance-segmentation-pytorch/code/lib/losses/discriminative.py", line 20, in calculate_means pred_masked = pred_repeated * gt_expanded RuntimeError: CUDA out of memory. Tried to allocate 6.00 GiB (GPU 0; 39.59 GiB total capacity; 27.21 GiB already allocated; 2.05 GiB free; 35.53 GiB reserved in total by PyTorch)

Error description is shown as above. the error emerges during validation after training is completed.