huggingface
SDXL LoRA key not loaded "lora_te2_text_projection.*" (LoRA from OneTrainer)
Describe the bug
When trying to use a LoRA in Diffusers, created in OneTrainer:
These keys are not loaded:
Python\Python311\site-packages\diffusers\loaders\lora_conversion_utils.py", line 273, in _convert_kohya_lora_to_diffusers
raise ValueError(f"The following keys have not been correctly be renamed: \n\n {', '.join(state_dict.keys())}")
ValueError: The following keys have not been correctly be renamed:
lora_te2_text_projection.lora_down.weight, lora_te2_text_projection.lora_up.weight
Reading up on the topic here: https://github.com/Nerogar/OneTrainer/issues/47 reveals that it is not considered a OneTrainer bug, and something which should be fixed externally.
This is how the bug is described:
So the question is: can the attempt to load the extra text encoder, also be ignored, if not found, so we can use OneTrainer LoRAs in Diffusers?
Reproduction
Import a SDXL LoRA from OneTrainer into Diffusers. (The one I've got is far too big to share)
Logs
No response
System Info
Windows 11, Diffusers latest alpha version.
Who can help?
@sayakpaul
You have to provide a fully reproducible code snippet for us to look into this further.
from diffusers import DiffusionPipeline
import torch
pipe_id = "stabilityai/stable-diffusion-xl-base-1.0"
pipe = DiffusionPipeline.from_pretrained(pipe_id, torch_dtype=torch.float16).to("cuda")
pipe.load_lora_weights("C:/Users/user_name/Downloads/LORA/", weight_name="lora.safetensors", adapter_name="engraving")
lora_scale= 0.9
prompt = "Engravings Henri Meyer"
image = pipe(
prompt, num_inference_steps=30, cross_attention_kwargs={"scale": lora_scale}, generator=torch.manual_seed(0)
).images[0]
image
The LoRA file: https://filetransfer.io/data-package/SKG6lXd3#link
If at all possible, please also host the LoRA file on the Hub so that is easily retrievable.
Thank you to all involved for prioritizing and resolving this issue superfast!