comfyanonymous
Memory is not release after each generation
Custom Node Testing
- [ ] I have tried disabling custom nodes and the issue persists (see how to disable custom nodes if you need help)
Expected Behavior
The memory should be deposed after each iteration.
Actual Behavior
The memory doesn't get dispose.
Steps to Reproduce
I used normal workflow from comfyui website
Debug Logs
The process is killed.
Other
No response
Hi, I have occasionally observed fp8 workflows leaking RAM that doesnt respond to --cache-none or the cache clear function. If you are 100% reproducing a RAM leak can I get your specific workflow pointer?
How much RAM do you have on your system?
I have been having the same issue. ComfyUI (the python process) always eats a huge chunk of system RAM even after clearing node cache/unloading models. It does not happen with my MacBook where CPU and GPU share memory but only happens on my servers with CUDA GPUs.
It seems to be some kind of hidden cache, because repeated loading/unloading of the same model does not cause RAM to increase, however if I load many many different models, it will eventually cause the system to freeze because the memory is getting filled to maximum.
It's happening to me when I'm not even loading models - this happens on a basic load image -> resize -> save image workflow. Memory gets allocated, then only partially released, then it fills the whole 128GB to full and starts swapping, then eventually crashing.
Only a restart fixes it.
It's happening to me when I'm not even loading models - this happens on a basic load image -> resize -> save image workflow. Memory gets allocated, then only partially released, then it fills the whole 128GB to full and starts swapping, then eventually crashing. Only a restart fixes it.
This is one of my RAM test basics and currently I can't reproduce any ramping leak like what you show. Can you give me specifics on this test?
Usually load -> resize -> save repeatedly will just cache hit the output and not run anything at all, so I tend to add noise to the image to force a rerun. Are you doing anything like that? Any cache settings on Comfy startup?
I ran comfy and the browser on separate machines (so a LAN IP instead of localhost), and spammed generations with adding noise to the image (so it doesnt just cache hit). RAM is returning to baseline:
I did however note if I queue 10 tasks straight from the browser I can ramp up RAM a little on the browser machine (this machine is not running comfy at all):
I did have one run where it crashed my browser (only 16GB RAM on this system).
check your task-manager processes tab to see if the RAM is coming from python or your web browser. If its the browser we can find some front-end people to take a look.
This is one of my RAM test basics and currently I can't reproduce any ramping leak like what you show. Can you give me specifics on this test?
There is nothing much else to it, it was extremely simple, I just threw a handful of nodes together by hand just like your example. I don't have that anymore but I can still reliably reproduce the effect by throwing a handful of node together again, because I have to restart comfy because of this every 2-3 videos I upscale. By the way, said workflow was definitely fine maybe some weeks ago, I reize things regularly, and it only became noticeable recently, maybe somewhere around the fast cancel changes?
Usually load -> resize -> save repeatedly will just cache hit the output and not run anything at all, so I tend to add noise to the image to force a rerun. Are you doing anything like that? Any cache settings on Comfy startup?
I have no cache settings; I do have pinned memory disabled on a previous recommendation on this issue, which didn't really help, but I left it that way.
Current parameters are --windows-standalone-build --listen 127.0.0.1 --disable-auto-launch --disable-pinned-memory
Previous was --windows-standalone-build --listen 127.0.0.1 --disable-auto-launch --fast pinned_memory --async-offload
I did however note if I queue 10 tasks straight from the browser I can ramp up RAM a little on the browser machine (this machine is not running comfy at all):
I didn't mention that my main use case where I found this was resizing batches of rather substantial sizes such as 3072x2048x145 down to 1536x1024x145. The memory ramping becomes much more noticeable with size.
check your task-manager processes tab to see if the RAM is coming from python or your web browser. If its the browser we can find some front-end people to take a look.
It's definitely from python. The image previews do have a massive CPU impact from the browser, but it should be unrelated to this. More to the point, I specifically have to restart comfy to fix this memory bloat, restarting the browser doesn't help at all.