Nan norm

Open YacratesWyh opened this issue 8 months ago • 1 comments

/home/admin188/AI/kohya_ss/sd-scripts/library/sdxl_lpw_stable_diffusion.py:975: RuntimeWarning: invalid value encountered in cast
images = (images * 255).round().astype("uint8")
steps: 32%|██████ | 640/2000 [34:52<1:14:06, 3.27s/it, Average key norm=nan, Keys Scaled=980, avr_loss=nan]

How could I figure the bug? I've no idea how to trace it.

May 28 '25 10:05 YacratesWyh

Show your full config and describe dataset characteristics. At what step do nan start? Usually, NaN indicates latent issues, for example, if you are using full_fp16, then some optimizers and settings can cause instability in fp16.

May 29 '25 08:05 deGENERATIVE-SQUAD