diffusers CLIP Guided Image2Image

The CLIP guided text2img is pretty good. Would be nice to be able to use it for img2img

Jan 05 '23 14:01 dblunk88

cc @patil-suraj,

Would be great to make this a community pipeline !

Jan 10 '23 15:01 patrickvonplaten

Hi @patrickvonplaten , since it will be my first issue ever, i will be happy to take this up if someone can guide me. Can this be assigned to me?

Jan 11 '23 16:01 akshay-babbar

Hey @akshay-babbar,

sure would you like to open a PR for it?

Jan 13 '23 13:01 patrickvonplaten

Yes @patrickvonplaten i would like to, but from the above , it is not clear to me what needs to be done exactly? Can you explain? Is this a reference point?

Jan 13 '23 18:01 akshay-babbar

Hey @akshay-babbar,

The goal here is to make a community pipeline (see: https://github.com/huggingface/diffusers/issues/841) that does img2img and is guided by CLIP, so quite similar to: https://github.com/huggingface/diffusers/blob/main/examples/community/clip_guided_stable_diffusion.py

Jan 16 '23 11:01 patrickvonplaten

Sure @patrickvonplaten , will be a good learning for me, can you assign this to me?

Jan 16 '23 11:01 akshay-babbar

Hey @akshay-babbar thanks, just assigned it to you, let me know if you need any help :)

Jan 16 '23 12:01 patil-suraj

Thanks @patil-suraj

Jan 16 '23 12:01 akshay-babbar

Hi may I ask if any update here?

Feb 14 '23 21:02 ryx19th

Hello @patrickvonplaten,

I'd like to take it up and raise a PR. Before moving ahead can you clarify my understanding, I'd have to add another pipeline in the community (as suggested in #841) with additional CLIP guidance? For the Img2Img pipeline, do I inherit from https://github.com/huggingface/diffusers/blob/v0.14.0/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_img2img.py#L93 or should I duplicate the code and mark duplicate as I read on other threads.

Would love to get clarification to enable faster convergence.

Mar 14 '23 09:03 nipunjindal

Hey @patil-suraj, Sorry but I was/am keeping busy, so could not get back to this. So for now please feel free to assign this issue to someone else. Thanks.

Mar 14 '23 14:03 akshay-babbar

Hello @patil-suraj @patrickvonplaten,

Here is the PR(https://github.com/huggingface/diffusers/pull/2723) for CLIP guidance for Img2Img pipeline. Let me know your thoughts.

Mar 17 '23 14:03 nipunjindal

Hello @patil-suraj @patrickvonplaten,

Sorry for the ping, were you able to take a look at the PR and would you have some initial comments?

Mar 21 '23 03:03 nipunjindal

Merged! Well done, looks great!

Mar 21 '23 14:03 patrickvonplaten

wohoo! Awesome work! Thank you for the awesome feature

Mar 21 '23 22:03 dblunk88