Use an image as the initial status to generate videos

[sunnyHelen]

Hi,

Thank you very much for sharing your great work and the code. Is it possible to take an initial image to animate it rather than randomly generate videos from the text prompt? Could you provide some suggestions on how to do this?

[ruizhaocv]

Hi. I think there are two ways to achieve your goal. The first one is that train spatial LoRAs on the initial image, combining with the temporal LoRAs learned from reference videos. The second one is that you can try to use the inversion noise from the initial image to denoise and generate videos.

[xiueran]

Hi,

Thank you very much for sharing your great work and the code. I encountered some problems when using MotionDirector for Image Animation. I trained temporal LoRAs and spatial LoRAs according to the instructions. However, the noise_prior parameter in the inference is 0.1, and the result is still similar to my reference video. Not the motion of the initial image.