AnimateDiff SparseCtrl RGB does not work as expected

[a-r-r-o-w]

Relevant comments are this and this.

AnimateDiff SparseCtrl RGB does not work similar to other implementations and cannot replicate their outputs. This makes me believe that there is something incorrect with our SparseControlNet or MotionAdapter implementation.

When comparing the results of the original/Comfy implementation to Diffusers implementation, one can notice that if an image is used with an unrelated prompt, the Diffusers implementation ignores the image and just follows the prompt whereas the other implementations try to incorporate both.

Since the original and Comfy implementations produce this behaviour consistently, this seems more like a problem with Diffusers implementation. However, I've not been able to spot differences in implementation just by comparing the code visually. I also tried matching outputs layerwise and it seemed to be alright (although I didn't investigate this as deeply as I should have due to other priorities).

If someone from the community actively following/using the AnimateDiff implementations can help determine the cause of this bug, it would be really awesome and helpful.

[ighoshsubho]

Hi @a-r-r-o-w I would like to pick this up and resolve.

[a-r-r-o-w]

Awesome, thanks @ighoshsubho! Looking forward to the fix and your investigation of this behaviour :eyes:

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[a-r-r-o-w]

Gentle ping @ighoshsubho in case you're still interested

[ighoshsubho]

@a-r-r-o-w Thank you for the follow-up. I need to apologize - while I initially expressed interest in working on this AnimateDiff SparseCtrl issue, I've been unable to dedicate the necessary time due to unexpected commitments in my primary work. Since this is an important bug that deserves thorough investigation and proper attention, I believe it would be best to let another contributor take this up who can give it the focus it needs. I apologize for leaving this issue stagnant.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[RevanthGundala]

I'm happy to pick this up @a-r-r-o-w

[a-r-r-o-w]

@RevanthGundala Awesome, please feel free to :)

[RevanthGundala]

I've been looking at this for the past week and I am also struggling to find the difference. I'll be dropping working on this, but if someone wants to pick this up I added a reference test to verify that the conditioning image isn't properly being used.