Depth 2 Image

[JackLangerman]

Any plans to support depth2image via the API?

[pharmapsychotic]

100%

[JackLangerman]

Hey @pharmapsychotic -- I took a lot at your PRs #166, #168, #171 and tried to use your branch anima_stable. It is correct that the grpc.stability.ai:443 doesn't support stable-diffusion-depth-v2-0 yet?

I tried this:

import os 
from stability_sdk import client

channel = client.open_channel(host="grpc.stability.ai:443", api_key=os.environ['STABILITY_KEY'])
stability_api = client.Api(
    channel=channel
)

but got this error:

_MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
	status = StatusCode.NOT_FOUND
	details = "The specified engine (ID stable-diffusion-depth-v2-0) was not found."
	debug_error_string = "UNKNOWN:Error received from peer ipv4:104.18.21.212:443 {grpc_message:"The specified engine (ID stable-diffusion-depth-v2-0) was not found.", grpc_status:5, created_time:"2023-01-26T14:25:56.850072-05:00"}"
>

I also tried your animation notebook but got this error with grpc.stability.ai:443:

_MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
	status = StatusCode.UNIMPLEMENTED
	details = "chains are not implemented"
	debug_error_string = "UNKNOWN:Error received from peer ipv4:104.18.20.212:443 {grpc_message:"chains are not implemented", grpc_status:12, created_time:"2023-01-26T15:27:19.747949-05:00"}"
>

I then tried grpc-staging.stability.ai:443, but unsurprisingly it didn't work with my API key.

I'm very excited use depth-to-image for my application (nice results w/ self hosting, but not enough scale to justify keeping a GPU running 24/7 at this time). Do you know if there is a way for me to help with testing / development (and if there is a way to locally host the API or get a staging API key)?

If I can't self host or develop on staging would you support the idea of splitting the depth2image work from the animation work (which seems like it has many more moving parts)? Could this help accelerate getting depth2image into main?

Really love your work and would love to help out if I can!

[pharmapsychotic]

Hi Jack! You're correct these endpoints are not available in production yet. For your use case would you be providing depth maps your self or like them to be computed for you?

[JackLangerman]

I'd be more than happy either way. Probably it is marginally more convenient at this stage to get them computed for me, but I am also comfortable running AdaBins/MiDaS myself and in the future it would be helpful to be able to manipulate the depth maps on my end before sending them to Stability.

[ck19910321]

Any updates on this?

[pharmapsychotic]

It is used as part of the animation pipeline but has not been documented or called out as a separate feature yet. There are newer techniques which give better results and more control which are coming so I think that's what we'll be promoting for people to use instead.

You can see the animation generating depth from image https://github.com/Stability-AI/stability-sdk/blob/main/src/stability_sdk/animation.py#L382 and then using the stable-diffusion-depth-v2-0 engine https://github.com/Stability-AI/stability-sdk/blob/main/src/stability_sdk/animation.py#L65 where depth is fed as the init_depth parameter to generate https://github.com/Stability-AI/stability-sdk/blob/main/src/stability_sdk/api.py#L117