Full SDXL DreamBooth finetuning, LoRA extraction

[neuron-party]

In the dreambooth community, it has been empirically shown that extracting a LoRA results in better performance than directly training a LoRA. Enabling a full dreambooth finetune of SDXL would not only enable this functionality, but also further enable users to extract and test out LoRAs of multiple different network ranks without having to rerun the training script multiple times.

[asomoza]

Hi, what would be the issue here?

You can do a dreambooth finetune with https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/train_dreambooth.py and then to extract it you can use any tool available for that.

I'm guessing that what you probably want is a diffusers script to extract the lora? I don't think we have the bandwidth to do that soon, so probably that will depend if someone in the community wants to do it.

Also it seems that Invoke-AI has working script for that, so maybe try that one?

https://github.com/invoke-ai/invoke-training https://github.com/invoke-ai/invoke-training/blob/62de7a6618a7084cbaa1e96956c37265ed2131f5/src/invoke_training/scripts/_experimental/lora_extraction/extract_lora_from_checkpoint.py

[neuron-party]

you can't train sdxl with that script i thought

[asomoza]

oh yeah, totally missed that sorry.

So what you're really asking is why we don't have a dreambooth SDXL training script and if we can add it? I don't really know the reason why we don't have it, maybe @sayakpaul can comment on this.

What I can tell is that we probably don't have the bandwidth for this and not that many people will benefit from it too, because of the high VRAM restriction that comes with it.

It shouldn't be that hard to adapt to SDXL, so maybe someone in the community is open to do it.

[neuron-party]

yeah that makes sense. i still think it would be a good opportunity for those that do have the available resources for a full model finetune which is why i asked

[sayakpaul]

Happy to accept PRs from the community on full DreamBooth fine-tuning on SDXL but this might quite limiting because of the compute needs as pointed out by @asomoza. I remember @linoytsaban was interested in exploring extracting LoRAs. So, pinging her here.

[linoytsaban]

Hey @neuron-party! As @sayakpaul mentioned I am exploring adding support for full dreambooth training and lora extraction to our advanced sdxl training, will try to send a PR soon

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.