wyyfff

icon indicating a website link https://www.sustech.edu.cn/ icon indicating an email [email protected]

icon company Southern University of Science and Technology icon location Shenzhen, China icon location Southern University of Science and Technology, CS, MS

Results 4 comments of wyyfff

NaN Loss after Forward Pass on Step 4 when Full Fine-Tuning Wan 2.2 5B

> [@wyyfffff](https://github.com/wyyfffff) Please reduce the learning rate. 1e-6 is safe. @Artiprocher Thank you for your reply ! I have already reduced the learning rate to 1e-8, with learning rate warmup...

NaN Loss after Forward Pass on Step 4 when Full Fine-Tuning Wan 2.2 5B

> [@wyyfffff](https://github.com/wyyfffff) I encountered a similar issue. I reduced the learning rate to 1e-8 and applied gradient clipping with accelerator.clip_grad_norm_(model.trainable_modules(), 1.0). My training uses DeepSpeed ZeRO-2 on four GPUs, and...

NaN Loss after Forward Pass on Step 4 when Full Fine-Tuning Wan 2.2 5B

For vae and text_encoder, you can still use `fetch_model`. But if you have modified the layers in dit, I think it’s better to try my method. > [@wyyfffff](https://github.com/wyyfffff) I encountered...

NaN Loss after Forward Pass on Step 4 when Full Fine-Tuning Wan 2.2 5B

@zzhang2816 Below is my modified code, maybe you can try it: In my case, I removed the WAN safetensor path from `model_configs` to create a new `wan_paths: list[str]` ``` @staticmethod...