[new model] DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors

[kabachuha]

Model/Pipeline/Scheduler description

A very great model for image2video generation (also frame interpolation, storyboarding and looping videos). The architecture is based on the current text2video Unet paradigm, so it shouldn't be hard to adjust it.

The most important feature of the diffusers implementation would be the fine-tuning code, similar to the lora/dreambooth training scripts, as the DynamiCrafter's authors don't want to release the training code :(

Open source status

• [X] The model implementation is available.
• [X] The model weights are available (Only relevant if addition is not a scheduler).

Provide useful links for the implementation

https://github.com/Doubiiu/DynamiCrafter (official repo) https://github.com/ExponentialML/ComfyUI_Native_DynamiCrafter (the ComfyUI code will be soon) https://arxiv.org/abs/2310.12190 (paper) https://github.com/Doubiiu/DynamiCrafter/tree/main#-models (weights)

[a-r-r-o-w]

Apologies for hijacking the issue for pushing this.

Related: #2984 (VideoCrafter), #5576 (LongerCrafter), #5442 (ScaleCrafter), StyleCrafter

Adding on to this, we also have MotionCtrl #6688, which has checkpoints trained for VideoCrafter and is one of the best ways to achieve controllability among open source models (also featured on Two Minute Papers). I'm currently working on VideoCrafter integration so that I can work on MotionCtrl support (currently only SVD version PR is live).

I'd love to see the Crafter family of models supported and usable from diffusers. The only simple way at the moment is ComfyUI plugins. Let's work together on shipping these and make them more accessible, WDYT? :rocket:

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[sayakpaul]

Closing because of inactivities.