[Community] Is there a code for pre-training stable diffusion?

[980202006]

This project works great. Is there a code for pre-training stable diffusion?

[patrickvonplaten]

Hey @980202006,

Thanks for the nice words, we currently don't have a pretraining script for stable diffusion. Currently I don't think we have the time to support such a new script, but we would welcome any contribution from the community.

Also cc'ing @patil-suraj here FYI.

[980202006]

Thank you!

[patil-suraj]

The train_text_to_image script can be adapted for pre-training stable diffusion. With SD, pre-training and fine-tuning objectives are similar. https://github.com/huggingface/diffusers/tree/main/examples/text_to_image, so to train from scratch we'll have to load random models, initialize the correct scheduler, handle data pre-processing etc.

[980202006]

@patil-suraj Thank you! Do you know the approximate loss of vae convergence? The vae loss I trained is very large.

[patil-suraj]

No, I haven't trained VAE. Also, stable diffusion does not train VAE, it uses pre-trained VAE checkpoints trained with taming-transformer. In stable diffusion only the unet is trained, the rest is frozen.