stable-diffusion-webui
stable-diffusion-webui copied to clipboard
Memory not freed after hypernetwork training OOM exception
Describe the bug When training a hypernetwork I can fairly often encounter my GPU's dedicated memory being kept from release if there is an OOM error during training. This means that I have to fully restart the program and have to go through the entire loading process again instead of just being able to send another request through the webui since the client-side web ui is unaffected.
To Reproduce Steps to reproduce the behavior:
- Open hypernetwork training menu
- Set batch value so high it will cause an OOM error on your GPU
- Start training
- See the OOM error in the console
- Check gpu memory usage using nvtop or task manager and see it is still allocated
Expected behavior Exception is handled and memory is freed of memory allocated for the network training.
Desktop (please complete the following information):
- OS: Windows 11 25217.1010
- Browser firefox but should be irrelevant
- Commit revision 7432b6f4d2c3001895fc75411a34afae1810c1a2
Additional context This (I think) also happened a couple of times when I was training a textual inversion model.