CUDA Out of Memory

[schicksa]

While fine-tuning the network (on 4 V100 GPUs) for text to SC, I always face the issue of CUDA OUT OF MEMORY. If the batch size is reduced to 2 (from 16), this issue is faced again.

I am currently using -

pytorch 1.10.2 transformers 3.4.0 wandb==0.12.16 huggingface-hub==0.6.0 batch_size: 2 num_workers: 8

E.g.: RuntimeError: CUDA out of memory. Tried to allocate 194.00 MiB (GPU 5; 23.59 GiB total capacity; 22.18 GiB already allocated;111.19 MiB free; 23.37 GiB reserved in total by PyTorch). Did you face any of these issues during fine-tuning?

[cameroncvpr]

Is this problem solved?