Pipeline proposal: Depth2ImgVariation

[maxfrei750]

Model/Pipeline/Scheduler description

It would be nice to have a pipeline that combines the StableDiffusionImageVariationPipeline with the StableDiffusionDepth2ImgPipeline, i.e. a pipeline that creates an image from a depth map, but guided by image embeddings rather than a prompt.

Open source status

• [ ] The model implementation is available
• [ ] The model weights are available (Only relevant if addition is not a scheduler).

Provide useful links for the implementation

The use-case would be to provide a style image and a depth image to produce images with the content of the depth image, but the style of the style image.

While the code of both pipelines seems to be pretty easy to merge, I'm wondering:

1. Would the concept work?
2. Is there a clever way around retraining?

Feedback is very appreciated and if the approach is promising, I would be happy to try myself on an implementation.

Thanks!

[patrickvonplaten]

Sounds like a cool project for the community! Anybody interested in taking this one?

[maxfrei750]

@patrickvonplaten Thanks for the kind words. So you think that it should work as imagined?

Concerning the implementation: I'd be happy to write the code, which should be just a combination of the StableDiffusionImageVariationPipeline with the StableDiffusionDepth2ImgPipeline, right?

My main concern is the training, since I lack access to the required hardware and don't know if/which strategies can be applied to avoid a full re-training. Do you have any guidance, e.g., if only parts of the weights need to be refined?

[patrickvonplaten]

Hey @maxfrei750,

Sure, I think there is no reason why this shouldn't work. All you will have to fine-tune is the UNet (no text encoder etc...)

[patrickvonplaten]

Also see: https://github.com/justinpinkney/stable-diffusion#image-variations

[maxfrei750]

Hey @patrickvonplaten, thanks for the information. I was also looking around for a training script for Stable Diffusion with the depth conditioning, to adapt it or at least see exactly how they did it. Unfortunately, I didn't find anything. Are you aware of such a resource?

[patrickvonplaten]

Hmm not really sadly. Could you maybe try to ask on our discord? https://discord.gg/G7tWnz98XR

[maxfrei750]

I'll try. Thanks for the hint!

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.