Bug: lora trained with train_dreambooth_lora_sdxl_advanced.py, the weight cannot be loaded

[garychan22]

Describe the bug

I have trained a sdxl lora with the script https://github.com/huggingface/diffusers/blob/7bc0a07b1947bcbe0e84bbe9ecf8ae2d234382c4/examples/advanced_diffusion_training/train_dreambooth_lora_sdxl_advanced.py , and a loading error was encountered during the validation in the end of the training whenc calling load_lora_weights https://github.com/huggingface/diffusers/blob/7bc0a07b1947bcbe0e84bbe9ecf8ae2d234382c4/examples/advanced_diffusion_training/train_dreambooth_lora_sdxl_advanced.py#L2426

Loading adapter weights from state_dict led to missing keys in the model: down_blocks.1.attentions.0.transformer_blocks.0.attn1.to_q.lora_A.default_0.weigh
t, down_blocks.1.attentions.0.transformer_blocks.0.attn1.to_q.lora_B.default_0.weight, down_blocks.1.attentions.0.transformer_blocks.0.attn1.to_k.lora_A.de
fault_0.weight, down_blocks.1.attentions.0.transformer_blocks.0.attn1.to_k.lora_B.default_0.weight, down_blocks.1.attentions.0.transformer_blocks.0.attn1.t
o_v.lora_A.default_0.weight, down_blocks.1.attentions.0.transformer_blocks.0.attn1.to_v.lora_B.default_0.weight, down_blocks.1.attentions.0.transformer_blo
cks.0.attn1.to_out.0.lora_A.default_0.weight, down_blocks.1.attentions.0.transformer_blocks.0.attn1.to_out.0.lora_B.default_0.weight, down_blocks.1.attenti
ons.0.transformer_blocks.0.attn2.to_q.lora_A.default_0.weight, down_blocks.1.attentions.0.transformer_blocks.0.attn2.to_q.lora_B.default_0.weight, 
....

Reproduction

running the https://github.com/huggingface/diffusers/blob/7bc0a07b1947bcbe0e84bbe9ecf8ae2d234382c4/examples/advanced_diffusion_training/train_dreambooth_lora_sdxl_advanced.py

by changing the target_modules = ["to_k", "to_q", "to_v", "to_out.0","proj_in", "proj_out","conv1", "conv2"] for the unet

Logs

System Info

linux python 3.9.21 diffusers==0.34.0.dev0 peft==0.15.0

Who can help?

No response