stable-diffusion-webui icon indicating copy to clipboard operation
stable-diffusion-webui copied to clipboard

Memory not freed after hypernetwork training OOM exception

Open papuSpartan opened this issue 3 years ago • 0 comments

Describe the bug When training a hypernetwork I can fairly often encounter my GPU's dedicated memory being kept from release if there is an OOM error during training. This means that I have to fully restart the program and have to go through the entire loading process again instead of just being able to send another request through the webui since the client-side web ui is unaffected.

To Reproduce Steps to reproduce the behavior:

  1. Open hypernetwork training menu
  2. Set batch value so high it will cause an OOM error on your GPU
  3. Start training
  4. See the OOM error in the console
  5. Check gpu memory usage using nvtop or task manager and see it is still allocated

Expected behavior Exception is handled and memory is freed of memory allocated for the network training.

Desktop (please complete the following information):

  • OS: Windows 11 25217.1010
  • Browser firefox but should be irrelevant
  • Commit revision 7432b6f4d2c3001895fc75411a34afae1810c1a2

Additional context This (I think) also happened a couple of times when I was training a textual inversion model.

papuSpartan avatar Oct 18 '22 09:10 papuSpartan