sei-framework icon indicating copy to clipboard operation
sei-framework copied to clipboard

CUDA out of memory.

Open tzhu-bio opened this issue 2 years ago • 1 comments

Very good work. I'm using your data to train the model following your tutorials and I'm running out of memory on the GPU, how can I modify the train.yml file? I don't have a GPU cluster, only a 24GB Nvidia 3090. hope you can give me a new train.yml file to help to finish the training.

tzhu-bio avatar Feb 25 '23 06:02 tzhu-bio

Often times this is an issue where the batch size is too large for the GPU, you can try modifying the YAML with a very small batch size to see if it's better (like batch size 2 or 4).

kathyxchen avatar Mar 15 '23 19:03 kathyxchen