Training diffusers examples using torch FSDP

[shrinath-suresh]

Describe the bug

I am able to run the current diffuser examples as specified in the readme steps

https://github.com/huggingface/diffusers/tree/main/examples/text_to_image

https://github.com/huggingface/diffusers/tree/main/examples/unconditional_image_generation

Would like to know if the examples can be trained with torch fsdp.

Reproduction

accelerate config

provides an option to enable fsdp based training.

But when the FSDP option is set to be Yes, the training is getting failed during the initialization.

Logs

To be attached.

System Info

diffusers - master pytorch - 1.12.1

[patrickvonplaten]

I don't think we will be able to support FSDP any time soon (cc @sgugger just FYI)

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.