[`Cont'd`] Add the SDE variant of DPM-Solver and DPM-Solver++ to DPM Single Step

[tolgacangoz]

Thanks for the opportunity to continue #4251 @yiyixuxu.

There was uncertainty about whether sigma_s0 or sigma_s1 should be used in the second-order update in sde-dpmsolver++ when @cmdr2 was implementing. I tried both and chose sigma_s1:

sigma_s0	sigma_s1

Reproduction notebook.

@LuChengTHU I want to congratulate you on this invention and thank you for your previous DPM-Solver contributions in the repo. It would be very nice to review this version. This PR is the exact copy of your "Add the SDE variant of DPM-Solver and DPM-Solver++ [for multi-step]" #3344 PR except sigma_s1 and alpha_s1 are used in the second-order update rather than sigma_s0 and alpha_s0.

[LuChengTHU]

Thanks for this nice PR! For single-step method we indeed should use sigma_s1 because we always take a "large step" for single-step method and re-use some intermediate variables (i.e., at time s0). So I believe this PR is correct!

[tolgacangoz]

Thank you very much for your review!

[tolgacangoz]

sde-dpmsolver++ seems reasonable; but sde-dpmsolver creates almost complete noise. I am into this.

!git clone -b sde-dpmsolver-single-step https://github.com/tolgacangoz/diffusers.git
%cd diffusers
!pip install -e . accelerate -q

from diffusers import AutoPipelineForText2Image, DPMSolverSinglestepScheduler
import torch

pipeline_text2image = AutoPipelineForText2Image.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16, variant="fp16",
    use_safetensors=True).to("cuda")
prompt = "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k"
generator = torch.Generator('cpu').manual_seed(0)
pipeline_text2image.scheduler = DPMSolverSinglestepScheduler.from_config(
    pipeline_text2image.scheduler.config, algorithm_type='sde-dpmsolver',
    final_sigmas_type='sigma_min')
image = pipeline_text2image(prompt=prompt, generator=generator).images[0]
image

sde-dpmsolver for DPMSolverMultistepScheduler also creates considerably noisy outputs; but not like complete noise.

[tolgacangoz]

For sde-dpmsolver:

Multi-step num_inference_steps=50	Single-step num_inference_steps=50

Multi-step num_inference_steps=100	Single-step num_inference_steps=100

Even multi-step with num_inference_steps=100 seems noisy :thinking:. Tbh, my math understanding level is insufficient to understand this study well. @LuChengTHU How should we conclude about sde-dpmsolver implementation?

[LuChengTHU]

Hi @tolgacangoz , I would suggest only use sde-dpmsolver++ because this formulation is more suitable for SDEs and can include stochastic DDIM / DDPM as the first order case. For sde-dpmsolver it may even need 1000 steps to converge so it is hard to use in practice.

[tolgacangoz]

Hi @tolgacangoz , I would suggest only use sde-dpmsolver++ because this formulation is more suitable for SDEs and can include stochastic DDIM / DDPM as the first order case. For sde-dpmsolver it may even need 1000 steps to converge so it is hard to use in practice.

OK, thank you for your feedback!

[tolgacangoz]

@yiyixuxu, this is ready for your review now. Will one continue with this?

[yiyixuxu]

@LuChengTHU is this PR good to merge now?

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[yiyixuxu]

@tolgacangoz thank you very much! this is a very nice PR :)

[tolgacangoz]

Thanks for merging and labeling as a good PR!