Support for hpcaitech OpenSora's STDiT for text2video and text2image generation

[kabachuha]

Model/Pipeline/Scheduler description

STDiT builds on Latte and DiT and yields a trade-off between generation quality and speed

https://github-production-user-asset-6210df.s3.amazonaws.com/99191637/313485495-983a1965-a374-41a7-a76b-c07941a6c1e9.mp4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAVCODYLSA53PQK4ZA%2F20240318%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240318T093601Z&X-Amz-Expires=300&X-Amz-Signature=89fc6e69755160d4b0c00efc5a166a04405f29a99f464410dfa53b73e251a0fd&X-Amz-SignedHeaders=host&actor_id=14872007&key_id=0&repo_id=760231710

Open source status

• [X] The model implementation is available.
• [X] The model weights are available (Only relevant if addition is not a scheduler).

Provide useful links for the implementation

https://github.com/hpcaitech/Open-Sora

https://github.com/hpcaitech/Open-Sora#model-weights

[kabachuha]

If nobody mind, I'd like to proceed with its implementation this week

[sayakpaul]

Cc: @DN6 @patil-suraj

@kabachuha sure, feel free to proceed and we will definitely help you out :)

[patil-suraj]

awesome, feel free to start @kabachuha we'll def help! Maybe for now we could keep it as researc_project since it's still experimental and the arch might change in future.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[ElliotQi]

If nobody mind, I'd like to proceed with its implementation this week

@kabachuha Hi, what's current status of diffusers version?

[narrowsnap]

@kabachuha @patil-suraj Is there any update?

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.