automatic
automatic copied to clipboard
vladmandic
[Issue]: Generation controls become unresponsive after long-running tasks
Issue Description
After long-running tasks are performed, most replicable after making a big grid, the generate button becomes unresponsive, and backend is stuck with trace showing current:task(task_id).
This issue is filed separately from #3950, as it's happening on completely local installs as well.
The log shows a generation of a grid completing, with a few traces showing the task being stuck on UI, and even loading another model not fixing the problem, while changing the trace a bit.
Page has to be reloaded in order to allow further generation, which varies from being a small bother to a potentially very confusing situation, when paired with #4280 for example.
Version Platform Description
16:36:21-719618 INFO Python: version=3.12.3 platform=Linux bin="/home/ohiom/sdnext/venv/bin/python3" venv="/home/ohiom/sdnext/venv" 16:36:21-749012 INFO Version: app=sd.next updated=2025-10-19 hash=f05c2917 branch=dev url=https://github.com/vladmandic/sdnext.git/tree/dev ui=dev 16:36:22-184138 INFO Updating main repository 16:36:23-542861 INFO Repository upgraded: b627617d Sun Oct 19 17:25:02 2025 +0300 16:36:23-546181 INFO Platform: arch=x86_64 cpu=x86_64 system=Linux release=6.6.87.2-microsoft-standard-WSL2 python=3.12.3 locale=('C', 'UTF-8') docker=False
Relevant log output
17:36:56-364783 INFO XYZ grid complete: images=40 results=41 size=(8960, 5382) time=1400.55 save=0.65
17:36:56-366625 DEBUG XYZ grid: grid=1 subgrids=0 images=40 total=41
17:36:56-620692 INFO Save: grid="outputs/grids/flux.1-dev-SRPO/grid-00054-2025-10-19-flux.1-dev-SRPO.jpg" type=JPEG width=8960 height=5382 size=2499142
17:36:58-831109 TRACE Server: alive=True requests=573 memory=26.6/94.3 status='running' task='UI' timestamp='20251019173656' current='task(4tdh9hm30s3sk29)' id='6c685fafc6ad45f' job=0 jobs=1 total=382 step=0 steps=0 queued=0 uptime=3616 elapsed=1.88 eta=None progress=0
17:38:58-950995 TRACE Server: alive=True requests=591 memory=26.59/94.3 status='running' task='UI' timestamp='20251019173656' current='task(4tdh9hm30s3sk29)' id='6c685fafc6ad45f' job=0 jobs=1 total=382 step=0 steps=0 queued=0 uptime=3736 elapsed=122.0 eta=None progress=0
17:40:59-049499 TRACE Server: alive=True requests=593 memory=26.59/94.3 status='running' task='UI' timestamp='20251019173656' current='task(4tdh9hm30s3sk29)' id='6c685fafc6ad45f' job=0 jobs=1 total=382 step=0 steps=0 queued=0 uptime=3856 elapsed=242.1 eta=None progress=0
17:41:51-388722 INFO Load model: select="Diffusers/Tiwaz/CenKreChro [b7a5f36b42]"
17:41:51-392655 DEBUG Cache clear
17:41:51-394130 DEBUG Current model: {'ram': {'total': 94.3, 'rss': 21.85, 'used': 26.59, 'free': 66.12, 'avail': 66.65, 'buffers': 0.0, 'cached': 1.58}, 'gpu': {'used': 6.95, 'total': 24.0, 'active': 0.43, 'peak': 12.21, 'retries': 0, 'oom': 0, 'swap': 0}, 'job': 'Load model'}
17:41:54-207679 DEBUG GC: current={'gpu': 1.64, 'ram': 8.04, 'oom': 0} prev={'gpu': 6.95, 'ram': 8.04} load={'gpu': 7, 'ram': 9} gc={'gpu': 5.31, 'py': 694} fn=reload_model_weights:unload_model_weights why=unload time=1.05
17:41:54-209852 DEBUG Unload model: {'ram': {'total': 94.3, 'rss': 3.29, 'used': 8.04, 'free': 84.68, 'avail': 85.2, 'buffers': 0.0, 'cached': 1.58}, 'gpu': {'used': 1.64, 'total': 24.0, 'active': 0.03, 'peak': 12.21, 'retries': 0, 'oom': 0, 'swap': 0}, 'job': 'Load model'}
after
17:41:54-219612 INFO Autodetect model: detect="FLUX" class=FluxPipeline file="/home/ohiom/database/models/Diffusers/models--Tiwaz--CenKreChro/snapshots/b7a5f36b42a682d2d3bcd2af7a498c160b947372"
17:41:54-220736 DEBUG Cache clear
17:41:54-493440 DEBUG Load model: type=Flux repo="Tiwaz/CenKreChro" cls=FluxPipeline offload=balanced dtype=torch.bfloat16 args={'torch_dtype': torch.bfloat16}
17:41:54-496035 DEBUG Quantization: module="Model" type=sdnq mode=pre dtype=int8 matmul=True group_size=0 svd_rank=32 svd_steps=8 use_svd=False quant_conv=False matmul_conv=False dequantize_fp32=False quantize_with_gpu=True quantization_device=cuda return_device=cpu
device_map=default offload_mode=balanced non_blocking=False modules_to_not_convert=['single_transformer_blocks.0.norm.linear.weight', '.time_text_embed', '.context_embedder', '.x_embedder', '.proj_out', '.norm_out', 'pos_embed',
'single_transformer_blocks.0.norm.linear.weight', '.time_text_embed', '.context_embedder', '.x_embedder', '.proj_out', '.norm_out', 'pos_embed'] modules_dtype_dict={}
17:41:54-498807 DEBUG Load model: transformer="Tiwaz/CenKreChro" cls=FluxTransformer2DModel subfolder=transformer quant="SDNQConfig" args={'torch_dtype': torch.bfloat16}
Progress 5384.22it/s █████████████████████ 100% 3/3 00:00 00:00 Fetching 3 files
Progress ?it/s 0% 0/3 00:00 ? Loading checkpoint shards17:41:56-093578 TRACE Monitor: {'ram': {'total': 94.3, 'rss': 3.96, 'used': 8.25, 'free': 83.98, 'avail': 84.99, 'buffers': 0.01, 'cached': 2.06}, 'gpu': {'used': 2.06, 'total': 24.0, 'active': 0.2,
'peak': 12.21, 'retries': 0, 'oom': 0, 'swap': 0}, 'job': 'Load DiT'}
Progress 16.74s/it ████▋ 33% 1/3 00:16 00:33 Loading checkpoint shards17:42:16-106109 TRACE Monitor: {'ram': {'total': 94.3, 'rss': 10.8, 'used': 13.67, 'free': 67.81, 'avail': 79.55, 'buffers': 0.01, 'cached': 12.8}, 'gpu': {'used': 2.49, 'total': 24.0, 'active': 0.05, 'peak': 12.21, 'retries': 0, 'oom': 0, 'swap': 0}, 'job': 'Load DiT'}
Progress 16.85s/it █████████▎ 67% 2/3 00:33 00:16 Loading checkpoint shards17:42:31-115572 TRACE Monitor: {'ram': {'total': 94.3, 'rss': 14.28, 'used': 17.93, 'free': 64.4, 'avail': 75.29, 'buffers': 0.0, 'cached': 11.96}, 'gpu': {'used': 2.49, 'total': 24.0, 'active': 0.1, 'peak': 12.21, 'retries': 0, 'oom': 0, 'swap': 0}, 'job': 'Load DiT'}
Progress 13.84s/it ██████████████ 100% 3/3 00:41 00:00 Loading checkpoint shards
17:42:37-309562 DEBUG GC: current={'gpu': 1.64, 'ram': 19.2, 'oom': 0} prev={'gpu': 1.64, 'ram': 19.2} load={'gpu': 7, 'ram': 20} gc={'gpu': 0.0, 'py': 264} fn=postprocess_model:_process_model_after_weight_loading why=sdnq time=0.55
17:42:37-323119 DEBUG Quantization: module="TE" type=sdnq mode=pre dtype=int8 matmul=True group_size=0 svd_rank=32 svd_steps=8 use_svd=False quant_conv=False matmul_conv=False dequantize_fp32=False quantize_with_gpu=True quantization_device=cuda return_device=cpu
device_map=default offload_mode=balanced non_blocking=False modules_to_not_convert=['wo', 'wo', 'wo', 'wo'] modules_dtype_dict={}
17:42:37-329221 DEBUG Load model: text_encoder="Disty0/t5-xxl" cls=T5EncoderModel quant="SDNQConfig" shared=True
Progress 6432.98it/s █████████████████████ 100% 2/2 00:00 00:00 Fetching 2 files
Progress ?it/s 0% 0/2 00:00 ? Loading checkpoint shards17:42:41-310165 TRACE Monitor: {'ram': {'total': 94.3, 'rss': 22.17, 'used': 22.64, 'free': 52.92, 'avail': 70.58, 'buffers': 0.01, 'cached': 18.74}, 'gpu': {'used': 2.66, 'total': 24.0, 'active': 0.11, 'peak': 12.21, 'retries': 0, 'oom': 0, 'swap': 0}, 'job': 'Load TE'}
Progress 3.32s/it ██████████████ 100% 2/2 00:06 00:00 Loading checkpoint shards
17:42:45-333943 DEBUG GC: current={'gpu': 1.64, 'ram': 26.56, 'oom': 0} prev={'gpu': 2.66, 'ram': 26.57} load={'gpu': 7, 'ram': 28} gc={'gpu': 1.02, 'py': 127} fn=postprocess_model:_process_model_after_weight_loading why=sdnq time=0.57
Progress 14.51it/s ██████████████ 100% 1/1 00:00 00:00 Loading checkpoint shards
Progress 16.25it/s █████████ 100% 7/7 00:00 00:00 Loading pipeline components...
17:42:46-751303 DEBUG GC: current={'gpu': 1.64, 'ram': 26.61, 'oom': 0} prev={'gpu': 1.64, 'ram': 26.61} load={'gpu': 7, 'ram': 28} gc={'gpu': 0.0, 'py': 131} fn=load_diffuser_force:load_flux why=load time=0.56
17:42:46-755231 DEBUG Setting model: component=vae {'slicing': True, 'tiling': True, 'tile': 1024, 'overlap': 0.25}
17:42:46-757130 INFO Offload: type=balanced op=init watermark=0.2-0.6 gpu=4.80-14.40:24.00 cpu=94.000 limit=0.00 always=['T5EncoderModel', 'UMT5EncoderModel'] never=['CLIPTextModel', 'CLIPTextModelWithProjection', 'AutoencoderKL'] pre=True streams=True
17:42:46-779945 DEBUG Module: name=transformer cls=FluxTransformer2DModel size=11.179 params=11904434240 quant=QuantizationMethod.SDNQ
17:42:46-788316 DEBUG Module: name=text_encoder_2 cls=T5EncoderModel size=7.372 params=4763195392 quant=QuantizationMethod.SDNQ
17:42:46-791569 DEBUG Module: name=text_encoder cls=CLIPTextModel size=0.229 params=123060480 quant=None
17:42:46-795271 DEBUG Module: name=vae cls=AutoencoderKL size=0.156 params=83819683 quant=None
17:42:46-796443 INFO Model class=FluxPipeline modules=4 size=18.936
17:42:46-800105 INFO Load model: family=f1 time={'total': 52.59, 'load': 52.54} native=1024 memory={'ram': {'total': 94.3, 'rss': 21.91, 'used': 26.61, 'free': 44.27, 'avail': 66.61, 'buffers': 0.01, 'cached': 23.4}, 'gpu': {'used': 1.64, 'total': 24.0, 'active': 0.03, 'peak':
12.21, 'retries': 0, 'oom': 0, 'swap': 0}, 'job': ''}
17:42:46-807537 DEBUG Save: file="/home/ohiom/sdnext/config.json" json=55 bytes=2787 time=0.005
17:42:59-760340 TRACE Server: alive=True requests=614 memory=26.63/94.3 status='idle' task='' timestamp=None current='task(4tdh9hm30s3sk29)' id='' job=0 jobs=0 total=385 step=0 steps=0 queued=0 uptime=3977 elapsed=12.96 eta=None progress=0
17:44:59-866058 TRACE Server: alive=True requests=638 memory=26.64/94.3 status='idle' task='' timestamp=None current='task(4tdh9hm30s3sk29)' id='' job=0 jobs=0 total=385 step=0 steps=0 queued=0 uptime=4097 elapsed=133.06 eta=None progress=0
Backend
Diffusers
Compute
nVidia CUDA
Interface
ModernUI
Branch
Dev
Model
Other
Acknowledgements
- [x] I have read the above and searched for existing issues
- [x] I confirm that this is classified correctly and its not an extension issue
- what does ui status show? modernui hover over very top-left icon
- post full content of browser console log
- The UI status shows as normal
- Nothing unusual in the browser console log
Navigated to http://127.0.0.1:7855/
logger.js?1755984769.2189236:13 18:00:12.540 createSplash {theme: 'dark'}
logger.js?1755984769.2189236:13 18:00:38.942 gradio time=26553
logger.js?1755984769.2189236:13 18:00:38.942 initStartup
logger.js?1755984769.2189236:13 18:00:38.984 toggleCompact false undefined
logger.js?1755984769.2189236:13 18:00:39.005 setFontSize 14
logger.js?1755984769.2189236:13 18:00:39.009 updateOpts settings=574 callbacks=25 apply=1
logger.js?1755984769.2189236:13 18:00:39.042 initPrompts
logger.js?1755984769.2189236:13 18:00:39.083 updateOpt sd_model_checkpoint Diffusers/Qwen/Qwen-Image-Edit-2509 [d3968ef930] Diffusers/Tiwaz/CenKreChro [b7a5f36b42]
logger.js?1755984769.2189236:13 18:00:39.086 updateOpts settings=590 callbacks=3 apply=0
logger.js?1755984769.2189236:13 18:00:41.378 initModels 37
logger.js?1755984769.2189236:13 18:00:41.384 initPromptChecker
logger.js?1755984769.2189236:13 18:00:41.400 initContextMenu
logger.js?1755984769.2189236:13 18:00:41.406 initDragDrop
logger.js?1755984769.2189236:13 18:00:41.494 initSettings
logger.js?1755984769.2189236:13 18:00:41.760 initImageViewer
logger.js?1755984769.2189236:13 18:00:41.769 initGallery
logger.js?1755984769.2189236:13 18:00:41.787 initGenerationParams
logger.js?1755984769.2189236:13 18:00:41.798 initControlUI
logger.js?1755984769.2189236:13 18:00:41.834 reconnectUI
logger.js?1755984769.2189236:13 18:00:41.841 mountURL
logger.js?1755984769.2189236:13 18:00:41.844 refreshInterval hidden 500
logger.js?1755984769.2189236:13 18:00:41.847 uiReady
logger.js?1755984769.2189236:13 18:00:41.851 userAgent Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36 Edg/141.0.0.0
logger.js?1755984769.2189236:13 18:00:41.855 settings {uiux_grid_image_size: 150, uiux_panel_min_width: 35, uiux_hide_legacy: true, uiux_persist_layout: true, uiux_no_slider_layout: false, …}
logger.js?1755984769.2189236:13 18:00:42.101 removeElements elements=24669/24834 stylesheets=33 time=243
logger.js?1755984769.2189236:13 18:00:42.106 mainUiUx time=255
logger.js?1755984769.2189236:13 18:00:42.112 initLogMonitor
logger.js?1755984769.2189236:13 18:00:42.159 initNetworks 140
logger.js?1755984769.2189236:13 18:00:42.183 loadTemplate template-app-root
logger.js?1755984769.2189236:13 18:00:43.045 loadTemplate main/template-txt2img-params
logger.js?1755984769.2189236:13 18:00:43.423 monitorConnection: online {app: 'sd.next', updated: '2025-10-19', hash: 'f05c2917', branch: 'dev', url: 'https://github.com/vladmandic/sdnext.git/tree/dev', …}
logger.js?1755984769.2189236:13 18:00:43.816 monitorConnection: message {"msg":"send_hash"}
logger.js?1755984769.2189236:13 18:00:43.822 loadTemplate main/template-txt2img-results
logger.js?1755984769.2189236:13 18:00:43.850 monitorConnection: online {app: 'sd.next', updated: '2025-10-19', hash: 'f05c2917', branch: 'dev', url: 'https://github.com/vladmandic/sdnext.git/tree/dev', …}
logger.js?1755984769.2189236:13 18:00:43.856 monitorConnection: message {"msg":"send_hash"}
logger.js?1755984769.2189236:13 18:00:43.861 loadTemplate main/template-img2img-params
logger.js?1755984769.2189236:13 18:00:43.884 loadTemplate main/template-img2img-results
logger.js?1755984769.2189236:13 18:00:43.896 loadTemplate main/template-control-params
logger.js?1755984769.2189236:13 18:00:43.911 loadTemplate main/template-control-results
logger.js?1755984769.2189236:13 18:00:43.924 loadTemplate main/template-video-params
logger.js?1755984769.2189236:13 18:00:43.936 loadTemplate main/template-video-results
logger.js?1755984769.2189236:13 18:00:43.948 loadTemplate main/template-extras-params
logger.js?1755984769.2189236:13 18:00:43.959 loadTemplate main/template-extras-results
logger.js?1755984769.2189236:13 18:00:43.971 loadTemplate main/template-caption-params
logger.js?1755984769.2189236:13 18:00:43.982 loadTemplate main/template-caption-results
logger.js?1755984769.2189236:13 18:00:43.994 loadTemplate main/template-gallery-params
logger.js?1755984769.2189236:13 18:00:44.004 loadTemplate main/template-gallery-results
logger.js?1755984769.2189236:13 18:00:44.016 loadTemplate template-aside-panel
logger.js?1755984769.2189236:13 18:00:44.028 loadTemplate aside/template-quicksettings
logger.js?1755984769.2189236:13 18:00:44.040 loadTemplate aside/template-extensions
logger.js?1755984769.2189236:13 18:00:44.051 loadTemplate aside/template-system
logger.js?1755984769.2189236:13 18:00:44.063 loadTemplate aside/template-info
logger.js?1755984769.2189236:13 18:00:44.075 loadTemplate aside/template-settings
logger.js?1755984769.2189236:13 18:00:44.095 loadTemplate aside/template-models
logger.js?1755984769.2189236:13 18:00:44.111 loadTemplate aside/template-history
logger.js?1755984769.2189236:13 18:00:44.126 loadTemplate aside/template-extra-networks
logger.js?1755984769.2189236:13 18:00:44.141 loadTemplate aside/template-console
logger.js?1755984769.2189236:13 18:00:44.198 loadAllTemplates load=1976 replace=39
logger.js?1755984769.2189236:13 18:00:46.927 loadAllPortals time=2656
logger.js?1755984769.2189236:13 18:00:49.881 setupToolButtons 56
logger.js?1755984769.2189236:13 18:00:50.777 waitForUiPortal time=14
logger.js?1755984769.2189236:13 18:00:50.847 hideLegacy true
logger.js?1755984769.2189236:13 18:00:51.343 setUserColors null null null
logger.js?1755984769.2189236:13 18:00:51.699 extraTweaks time=87
logger.js?1755984769.2189236:13 18:00:52.274 removeSplash
logger.js?1755984769.2189236:13 18:00:52.320 startupTime 39931
logger.js?1755984769.2189236:13 18:00:52.368 showNetworks
logger.js?1755984769.2189236:13 18:00:52.429 getLocale Auto en
logger.js?1755984769.2189236:13 18:00:52.476 markSelectedCards [''] style
logger.js?1755984769.2189236:13 18:00:52.540 initStartup 14
logger.js?1755984769.2189236:13 18:00:52.898 initIndexDB 100
logger.js?1755984769.2189236:13 18:00:55.643 touchDevice false
logger.js?1755984769.2189236:13 18:00:56.211 setHints {type: 2, locale: 'en', elements: 5350, localized: 0, hints: 3526, …}
logger.js?1755984769.2189236:13 18:01:20.775 refreshInterval visible 500
logger.js?1755984769.2189236:13
18:43:00.484 submitControl
logger.js?1755984769.2189236:13 18:43:03.665 refreshInterval hidden 2500
logger.js?1755984769.2189236:13 18:43:49.293 refreshInterval visible 500
logger.js?1755984769.2189236:13 18:43:49.705 refreshInterval hidden 2500
logger.js?1755984769.2189236:13 18:44:06.545 refreshInterval visible 500
logger.js?1755984769.2189236:13 18:44:06.856 refreshInterval hidden 2500
logger.js?1755984769.2189236:13 18:44:25.029 refreshInterval visible 500
logger.js?1755984769.2189236:13 18:44:30.233 refreshInterval hidden 2500
logger.js?1755984769.2189236:13 18:44:47.962 refreshInterval visible 500
logger.js?1755984769.2189236:13 18:44:52.033 refreshInterval hidden 2500
logger.js?1755984769.2189236:13 18:45:17.824 refreshInterval visible 500
logger.js?1755984769.2189236:13 18:45:30.263 refreshInterval hidden 2500
logger.js?1755984769.2189236:13 18:55:46.890 refreshInterval visible 500
logger.js?1755984769.2189236:13 18:55:48.312 refreshInterval hidden 2500
logger.js?1755984769.2189236:13 18:56:29.366 refreshInterval visible 500
logger.js?1755984769.2189236:13 18:56:29.907 refreshInterval hidden 2500
logger.js?1755984769.2189236:13 18:56:30.408 refreshInterval visible 500
logger.js?1755984769.2189236:13 18:56:56.282 refreshInterval hidden 2500
logger.js?1755984769.2189236:13 18:59:18.022 refreshInterval visible 500
logger.js?1755984769.2189236:13 18:59:24.443 refreshInterval hidden 2500
hmm, main ws link is still active. any browser extensions? I mean any. also, give me a time - what is approx min time for a long running task to reproduce this?
The log was provided from Edge, running 1password, uBlock Origin, VisBug, Tampermonkey, Vencord Web.
For the rest of the testing, none, the Chrome I use for reproducing and testing is completely free of any extensions.
It seems to be happening for tasks over 10 minutes, I'm currently trying to establish if that's a fully reproducible goalpost. In the log I provided the task has been running for over 15.
But I wonder if it's somehow related to efficiency mode or something similar. Doesn't make much sense as the grid appears as normal in the tab without reloading, but I'm just noting my thoughts.
Edit: Just reproduced on task of 724 seconds, the generation buttons became inoperable, it's on Chrome with 0 extensions:
23:48:43-242765 INFO XYZ grid complete: images=20 results=21 size=(4864, 5382) time=742.44 save=0.33
23:48:43-245595 DEBUG XYZ grid: grid=1 subgrids=0 images=20 total=21
23:48:43-402097 INFO Save: grid="outputs/grids/CenKreChro/grid-00210-2025-10-19-CenKreChro.jpg" type=JPEG width=4864 height=5382 size=2188513
23:48:44-923439 TRACE Server: alive=True requests=462 memory=27.6/94.3 status='running' task='UI' timestamp='20251019234843' current='task(ng2cgzda6szg0s4)' id='3b11f131115b4d8' job=0 jobs=1 total=469 step=0 steps=0 queued=0 uptime=3013 elapsed=1.36 eta=None progress=0
23:50:44-979347 TRACE Server: alive=True requests=473 memory=27.62/94.3 status='running' task='UI' timestamp='20251019234843' current='task(ng2cgzda6szg0s4)' id='3b11f131115b4d8' job=0 jobs=1 total=469 step=0 steps=0 queued=0 uptime=3133 elapsed=121.42 eta=None progress=0
The previous task of 596 seconds completed without the issue.
23:35:06-580592 INFO XYZ grid complete: images=16 results=17 size=(4864, 4358) time=596.52 save=0.30
23:35:06-583088 DEBUG XYZ grid: grid=1 subgrids=0 images=16 total=17
23:35:06-717419 INFO Save: grid="outputs/grids/CenKreChro/grid-00189-2025-10-19-CenKreChro.jpg" type=JPEG width=4864 height=4358 size=1821542
just ran a quen-image-edit-2509 img2img grid of 12 images, that took a while:
Processed: timers={'total': 827.65 ...
tried twice, no issues, cant reproduce
Sorry to hear that. I will try to reproduce on another network and a new computer to see if this could somehow be related to either.
I'm going off on a limb here, could you try with preview disabled? I noticed that disabling preview also causes the animated generate button with the progress bar to not show up.
disabling live preview should not disable progress updates, just fixed that. lmk if it has impact on the rest of the issue.
So far it seems the fix to the progress updates being disabled while disabling preview has also fixed this issue, I am going to run a proper test of it today an will report if the issue is still present. I "tested" it by-the-by while investigating other things yesterday.
Unfortunately, I ran some grids recently and the problem persists. I will let you know if I find anything else.