Tuning-Free Noise Rectification for High Fidelity Image-to-Video Generation

[clarencechen]

Model/Pipeline/Scheduler description

Applying pretrained Text-to-Video (T2V) Diffusion models to Image-to-video (I2V) generation tasks using SDEdit often results in low source image fidelity in open domains. This method achieves high source image fidelity in output videos through supplementing more precise source image information using noise interpolation during early denoising steps, resulting in a simple-to-implement, tuning-free, and plug-and-play implementation. The experimental results demonstrate the effectiveness in improving the source image fidelity of generated videos when applied to I2V generation using SDEdit with several different T2V models.

Open source status

• [ ] The model implementation is available.
• [ ] The model weights are available (Only relevant if addition is not a scheduler).

Provide useful links for the implementation

Website: https://noise-rectification.github.io/ Paper: https://arxiv.org/pdf/2403.02827.pdf

[UmerHA]

@sayakpaul I'd take this on, if okay with you!

[sayakpaul]

Aure. Let's start with a community pipeline.

[UmerHA]

Update: The model implementation is currently not openly available. I've asked the authors if they would consider open-sourcing the model.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[UmerHA]

Update: Model implementation kindly been open sourced. ~~I plan to implement this soon.~~ Edit: I won't be able to do it, as another much more important (to me) project came up. Hopy another contributor takes a shot!

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[abhiramvad]

@sayakpaul I'd like to take this up. if okay with you?

[a-r-r-o-w]

@abhiramvad Thanks, feel free to take it up! You can add it to the community folder with your name & contribution to the community README.

[abhiramvad]

Thank you @a-r-r-o-w ! Taking this up.

[abhiramvad]

I have added changes to my fork, but I see a few GitHub workflows failing. May I know which ones are essential before I raise a PR?

[UmerHA]

Form my experience: Some workflows fail because they require the use of HuggingFace resources, which external accounts don't have. If you open a PR, they'll likely work on the HF repo.

So as long as all tests pass & you've run linting, you should open a PR

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.