What does this PR do?

This PR support inpaint pipeline in stable diffusion 3. It follows the idea of StableDiffusionInpaintPipelineLegacy as it do not need specific weight for inpainting.

We hope this PR can be an initial version for StableDiffusion3InpaintPipeline. It can be replaced by finetuned inpaint weight in the later version.

We put the demo here: DEMO.

How to use it?

import torch
from torchvision import transforms

from diffusers import StableDiffusion3InpaintPipeline
from diffusers.utils import load_image

def preprocess_image(image):
    image = image.convert("RGB")
    image = transforms.CenterCrop((image.size[1] // 64 * 64, image.size[0] // 64 * 64))(image)
    image = transforms.ToTensor()(image)
    image = image.unsqueeze(0).to("cuda")
    return image

def preprocess_mask(mask):
    mask = mask.convert("L")
    mask = transforms.CenterCrop((mask.size[1] // 64 * 64, mask.size[0] // 64 * 64))(mask)
    mask = transforms.ToTensor()(mask)
    mask = mask.to("cuda")
    return mask

pipe = StableDiffusion3InpaintPipeline.from_pretrained(
    "stabilityai/stable-diffusion-3-medium-diffusers",
    torch_dtype=torch.float16,
).to("cuda")

prompt = "Face of a yellow cat, high resolution, sitting on a park bench"
source_image = load_image(
    "./overture-creations-5sI6fQgYIuo.png"
)
source = preprocess_image(source_image)
mask = preprocess_mask(
    load_image(
        "./overture-creations-5sI6fQgYIuo_mask.png"
    )
)

image = pipe(
    prompt=prompt,
    image=source,
    mask_image=mask,
    height=1024,
    width=1024,
    num_inference_steps=28,
    guidance_scale=7.0,
    strength=0.6,
).images[0]

image.save("output.png")

Image Input:

Mask Input:

Prompt: Face of a yellow cat, high resolution, sitting on a park bench

SD3 output:

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag members/contributors who may be interested in your PR.

@yiyixuxu

Jun 26 '24 08:06 IrohXu

Our teammate has implemented Inpaint pipeline for SD3. Could you review this PR? @yiyixuxu @sayakpaul Thanks!

Jun 27 '24 03:06 IrohXu

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Jun 27 '24 03:06 HuggingFaceDocBuilderDev

Hi @yiyixuxu Can you review the new version? Thanks a lot!

Jun 27 '24 18:06 IrohXu

Thanks for your PR for SD3 inpainting! I am currently trying to finetune the inpainting version by 33 channels. Your version raised an Error on inpainting-specific checkpoints.

the masked latent and mask should apply classifier-free guidance.
Mask latent needs to apply VAE "normalize" as well. Hence, it would be better to check the conv_in dimension and raise ERROR when the channel of conv_in != 16.

Jun 30 '24 15:06 George0726

Thanks for your PR for SD3 inpainting! I am currently trying to finetune the inpainting version by 33 channels. Your version raised an Error on inpainting-specific checkpoints.

the masked latent and mask should apply classifier-free guidance.

Mask latent needs to apply VAE "normalize" as well. Hence, it would be better to check the conv_in dimension and raise ERROR when the channel of conv_in != 16.

Thanks for the comment. I will update it based on your idea tomorrow. It is great that you will provide checkpoint for inpainting version by 33 channels. Would like me to add your ideas into this PR or creat a new PR for the 33 channels version later?

Jul 01 '24 06:07 IrohXu

@yiyixuxu @George0726 Could you review the new update? Thanks. We follow your suggestion on using SDXL and SD inpaint pipeline and also consider @George0726 's idea on keeping the if-else check for 33 channels and implement VAE "normalize" for mask input.

Jul 01 '24 08:07 IrohXu

@George0726 Could you try your 33 channel checkpoint in this PR? Thanks! Let me known if you meet any new error.

Jul 01 '24 08:07 IrohXu

@IrohXu Sure. I have tested and modified some parts to make it work.

SD3 VAE added additional self.vae.config.shift_factor to normalize, you can refer to https://github.com/huggingface/diffusers/blob/49979753e14e7fed61bebd5700d4dbd1b14a6ffc/src/diffusers/pipelines/stable_diffusion_3/pipeline_stable_diffusion_3_img2img.py#L629
masked_latent should also apply the VAE scaling, referred as https://github.com/huggingface/diffusers/blob/49979753e14e7fed61bebd5700d4dbd1b14a6ffc/src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl_inpaint.py#L876 Here is my git diff:

diff --git a/src/diffusers/pipelines/stable_diffusion_3/pipeline_stable_diffusion_3_inpaint.py b/src/diffusers/pipelines/stable_diffusion_3/pipeline_stable_diffusion_3_inpaint.py
index c8f05fe25..a5434d90c 100644
--- a/src/diffusers/pipelines/stable_diffusion_3/pipeline_stable_diffusion_3_inpaint.py
+++ b/src/diffusers/pipelines/stable_diffusion_3/pipeline_stable_diffusion_3_inpaint.py
@@ -677,8 +677,8 @@ class StableDiffusion3InpaintPipeline(DiffusionPipeline):
         else:
             image_latents = retrieve_latents(self.vae.encode(image), generator=generator)
 
-        image_latents = self.vae.config.scaling_factor * image_latents
 
+        image_latents = (image_latents - self.vae.config.shift_factor) * self.vae.config.scaling_factor
         return image_latents
 
     def prepare_mask_latents(
@@ -711,6 +711,7 @@ class StableDiffusion3InpaintPipeline(DiffusionPipeline):
         else:
             masked_image_latents = retrieve_latents(self.vae.encode(masked_image), generator=generator)
 
+        masked_image_latents = (masked_image_latents - self.vae.config.shift_factor) * self.vae.config.scaling_factor
         # duplicate mask and masked_image_latents for each generation per prompt, using mps friendly method
         if mask.shape[0] < batch_size:
             if not batch_size % mask.shape[0] == 0:

Here are some results of my current models: input and mask: case5_2_ori case5_2_mask

prompt: red hair

inpainting by SD3 2000_2_0_SD3

inpainting by SD3-inpainting 14000_2_3

inpainting by SD3-inpainting with modified codes 14000_2_1

Jul 01 '24 14:07 George0726

@George0726 Thanks a lot! I have added your improved codes into this PR.

Jul 01 '24 17:07 IrohXu

@a-r-r-o-w can you give this a review too if you have time?

Jul 01 '24 22:07 yiyixuxu

@a-r-r-o-w Thanks for the comments. I have updated the code based on your suggestions. For this first issue, I think we should wait for @yiyixuxu reply to us. It might be solved by another PR I think as all SD3 pipelines have it.

Jul 03 '24 04:07 IrohXu

@yiyixuxu @a-r-r-o-w I have updated it based on your comments. Thanks a lot!

Jul 03 '24 22:07 IrohXu

@IrohXu can you run make style and make fix-copies?

Jul 03 '24 22:07 yiyixuxu

@IrohXu can you run make style and make fix-copies?

@yiyixuxu I just run them and push the code again.

Jul 03 '24 23:07 IrohXu

@George0726 let us know when your checkpoints are ready :)

Jul 03 '24 23:07 yiyixuxu

@IrohXu can you make sure the tests pass? the new inpainting tests are failing here

Jul 03 '24 23:07 yiyixuxu

@George0726 let us know when your checkpoints are ready :)

I have uploaded the alpha version of SD3 inpainting model. I don't have enough data and GPUs for large-scale pre-training. https://huggingface.co/George0667/SD3_inpaint_alpha/tree/main

Jul 04 '24 02:07 George0726

@IrohXu can you make sure the tests pass? the new inpainting tests are failing here

@yiyixuxu I tested it locally in my machine today, it seems it can pass all failed cases. Do you know how this error appear differently in different test environment?

RuntimeError: The size of tensor a (4) must match the size of tensor b (2) at non-singleton dimension 0

Here is my log:

============================= test session starts ==============================
platform linux -- Python 3.10.0, pytest-8.2.2, pluggy-1.5.0 -- /home/xucao2/miniconda3/envs/sd_train/bin/python
cachedir: .pytest_cache
rootdir: /home/xucao2/diffusers
configfile: pyproject.toml
plugins: requests-mock-1.10.0, xdist-3.6.1, timeout-2.3.1
created: 32/32 workers
32 workers [30 items]

scheduling tests via LoadFileScheduling

tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_StableDiffusionMixin_component 
[gw0] PASSED tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_StableDiffusionMixin_component 
tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_attention_slicing_forward_pass 
[gw0] PASSED tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_attention_slicing_forward_pass 
tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_callback_cfg 
[gw0] PASSED tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_callback_cfg 
tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_callback_inputs 
[gw0] PASSED tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_callback_inputs 
tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_cfg 
[gw0] PASSED tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_cfg 
tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_components_function 
[gw0] PASSED tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_components_function 
tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_cpu_offload_forward_pass_twice 
[gw0] PASSED tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_cpu_offload_forward_pass_twice 
tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_dict_tuple_outputs_equivalent 
[gw0] PASSED tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_dict_tuple_outputs_equivalent 
tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_float16_inference 
[gw0] PASSED tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_float16_inference 
tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_inference_batch_consistent Token indices sequence length is longer than the specified maximum sequence length for this model (402 > 77). Running this sequence through the model will result in indexing errors
The following part of your input was truncated because CLIP can only handle sequences up to 77 tokens: ['<|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|>', 'ongvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery long']
Token indices sequence length is longer than the specified maximum sequence length for this model (402 > 77). Running this sequence through the model will result in indexing errors
The following part of your input was truncated because CLIP can only handle sequences up to 77 tokens: ['<|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|>', 'ongvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery long']
The following part of your input was truncated because `max_sequence_length` is set to  256 tokens: ['<pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad>', 'longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery longvery long']

[gw0] PASSED tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_inference_batch_consistent 
tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_inference_batch_single_identical Token indices sequence length is longer than the specified maximum sequence length for this model (402 > 77). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (402 > 77). Running this sequence through the model will result in indexing errors

[gw0] PASSED tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_inference_batch_single_identical 
tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_latents_input 
[gw0] PASSED tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_latents_input 
tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_model_cpu_offload_forward_pass 
[gw0] PASSED tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_model_cpu_offload_forward_pass 
tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_multi_vae 
[gw0] PASSED tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_multi_vae 
tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_num_images_per_prompt 
[gw0] PASSED tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_num_images_per_prompt 
tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_pipeline_call_signature 
[gw0] PASSED tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_pipeline_call_signature 
tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_progress_bar 
[gw0] PASSED tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_progress_bar 
tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_pt_np_pil_inputs_equivalent 
[gw0] PASSED tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_pt_np_pil_inputs_equivalent 
tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_pt_np_pil_outputs_equivalent 
[gw0] PASSED tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_pt_np_pil_outputs_equivalent 
tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_save_load_float16 
Loading pipeline components...:   0%|          | 0/9 [00:00<?, ?it/s]
Loading pipeline components...:  11%|█         | 1/9 [00:00<00:01,  7.78it/s]
Loading pipeline components...:  56%|█████▌    | 5/9 [00:00<00:00, 21.23it/s]
Loading pipeline components...:  89%|████████▉ | 8/9 [00:00<00:00, 22.20it/s]
Loading pipeline components...: 100%|██████████| 9/9 [00:00<00:00, 23.09it/s]

[gw0] PASSED tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_save_load_float16 
tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_save_load_local 
Loading pipeline components...:   0%|          | 0/9 [00:00<?, ?it/s]Loaded text_encoder_2 as CLIPTextModelWithProjection from `text_encoder_2` subfolder of /tmp/tmpdyb_z201.

Loading pipeline components...:  11%|█         | 1/9 [00:00<00:01,  7.61it/s]Loaded scheduler as FlowMatchEulerDiscreteScheduler from `scheduler` subfolder of /tmp/tmpdyb_z201.
Loaded tokenizer_3 as T5TokenizerFast from `tokenizer_3` subfolder of /tmp/tmpdyb_z201.
An error occurred while trying to fetch /tmp/tmpdyb_z201/vae: Error no file named diffusion_pytorch_model.safetensors found in directory /tmp/tmpdyb_z201/vae.
Defaulting to unsafe serialization. Pass `allow_pickle=False` to raise an error instead.
Loaded vae as AutoencoderKL from `vae` subfolder of /tmp/tmpdyb_z201.
Loaded text_encoder as CLIPTextModelWithProjection from `text_encoder` subfolder of /tmp/tmpdyb_z201.

Loading pipeline components...:  56%|█████▌    | 5/9 [00:00<00:00, 20.03it/s]Loaded tokenizer as CLIPTokenizer from `tokenizer` subfolder of /tmp/tmpdyb_z201.
An error occurred while trying to fetch /tmp/tmpdyb_z201/transformer: Error no file named diffusion_pytorch_model.safetensors found in directory /tmp/tmpdyb_z201/transformer.
Defaulting to unsafe serialization. Pass `allow_pickle=False` to raise an error instead.
Loaded transformer as SD3Transformer2DModel from `transformer` subfolder of /tmp/tmpdyb_z201.
Loaded text_encoder_3 as T5EncoderModel from `text_encoder_3` subfolder of /tmp/tmpdyb_z201.

Loading pipeline components...:  89%|████████▉ | 8/9 [00:00<00:00, 21.17it/s]Loaded tokenizer_2 as CLIPTokenizer from `tokenizer_2` subfolder of /tmp/tmpdyb_z201.

Loading pipeline components...: 100%|██████████| 9/9 [00:00<00:00, 22.05it/s]

[gw0] PASSED tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_save_load_local 
tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_save_load_optional_components 
Loading pipeline components...:   0%|          | 0/9 [00:00<?, ?it/s]Loaded text_encoder_2 as CLIPTextModelWithProjection from `text_encoder_2` subfolder of /tmp/tmptksec1h_.

Loading pipeline components...:  11%|█         | 1/9 [00:00<00:01,  7.64it/s]Loaded scheduler as FlowMatchEulerDiscreteScheduler from `scheduler` subfolder of /tmp/tmptksec1h_.
Loaded tokenizer_3 as T5TokenizerFast from `tokenizer_3` subfolder of /tmp/tmptksec1h_.
An error occurred while trying to fetch /tmp/tmptksec1h_/vae: Error no file named diffusion_pytorch_model.safetensors found in directory /tmp/tmptksec1h_/vae.
Defaulting to unsafe serialization. Pass `allow_pickle=False` to raise an error instead.
Loaded vae as AutoencoderKL from `vae` subfolder of /tmp/tmptksec1h_.
Loaded text_encoder as CLIPTextModelWithProjection from `text_encoder` subfolder of /tmp/tmptksec1h_.

Loading pipeline components...:  56%|█████▌    | 5/9 [00:00<00:00, 20.29it/s]Loaded tokenizer as CLIPTokenizer from `tokenizer` subfolder of /tmp/tmptksec1h_.
An error occurred while trying to fetch /tmp/tmptksec1h_/transformer: Error no file named diffusion_pytorch_model.safetensors found in directory /tmp/tmptksec1h_/transformer.
Defaulting to unsafe serialization. Pass `allow_pickle=False` to raise an error instead.
Loaded transformer as SD3Transformer2DModel from `transformer` subfolder of /tmp/tmptksec1h_.
Loaded text_encoder_3 as T5EncoderModel from `text_encoder_3` subfolder of /tmp/tmptksec1h_.

Loading pipeline components...:  89%|████████▉ | 8/9 [00:00<00:00, 21.24it/s]Loaded tokenizer_2 as CLIPTokenizer from `tokenizer_2` subfolder of /tmp/tmptksec1h_.

Loading pipeline components...: 100%|██████████| 9/9 [00:00<00:00, 22.15it/s]

[gw0] PASSED tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_save_load_optional_components 
tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_sequential_cpu_offload_forward_pass 
[gw0] PASSED tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_sequential_cpu_offload_forward_pass 
tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_sequential_offload_forward_pass_twice 
[gw0] PASSED tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_sequential_offload_forward_pass_twice 
tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_stable_diffusion_3_inpaint_different_negative_prompts 
  0%|          | 0/2 [00:00<?, ?it/s]
100%|██████████| 2/2 [00:00<00:00, 260.00it/s]

  0%|          | 0/2 [00:00<?, ?it/s]
100%|██████████| 2/2 [00:00<00:00, 268.14it/s]

[gw0] PASSED tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_stable_diffusion_3_inpaint_different_negative_prompts 
tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_stable_diffusion_3_inpaint_different_prompts 
  0%|          | 0/2 [00:00<?, ?it/s]
100%|██████████| 2/2 [00:00<00:00, 268.14it/s]

  0%|          | 0/2 [00:00<?, ?it/s]
100%|██████████| 2/2 [00:00<00:00, 269.70it/s]

[gw0] PASSED tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_stable_diffusion_3_inpaint_different_prompts 
tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_stable_diffusion_3_inpaint_prompt_embeds 
  0%|          | 0/2 [00:00<?, ?it/s]
100%|██████████| 2/2 [00:00<00:00, 269.07it/s]

  0%|          | 0/2 [00:00<?, ?it/s]
100%|██████████| 2/2 [00:00<00:00, 262.92it/s]

[gw0] PASSED tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_stable_diffusion_3_inpaint_prompt_embeds 
tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_to_device 
[gw0] PASSED tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_to_device 
tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_to_dtype 
[gw0] PASSED tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_to_dtype 
tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_xformers_attention_forwardGenerator_pass 
[gw0] SKIPPED tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_xformers_attention_forwardGenerator_pass 

=============================== warnings summary ===============================
src/diffusers/models/transformers/transformer_2d.py:34: 32 warnings
  /home/xucao2/diffusers/src/diffusers/models/transformers/transformer_2d.py:34: FutureWarning: `Transformer2DModelOutput` is deprecated and will be removed in version 1.0.0. Importing `Transformer2DModelOutput` from `diffusers.models.transformer_2d` is deprecated and this will be removed in a future version. Please use `from diffusers.models.modeling_outputs import Transformer2DModelOutput`, instead.
    deprecate("Transformer2DModelOutput", "1.0.0", deprecation_message)

tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py: 22 warnings
  /home/xucao2/diffusers/src/diffusers/configuration_utils.py:140: FutureWarning: Accessing config attribute `vae_latent_channels` directly via 'VaeImageProcessor' object attribute is deprecated. Please access 'vae_latent_channels' over 'VaeImageProcessor's config object instead, e.g. 'scheduler.config.vae_latent_channels'.
    deprecate("direct config name access", "1.0.0", deprecation_message, standard_warn=False)

tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_inference_batch_consistent
tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_inference_batch_single_identical
tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py::StableDiffusion3InpaintPipelineFastTests::test_num_images_per_prompt
  /home/xucao2/diffusers/src/diffusers/image_processor.py:528: FutureWarning: Passing `image` as a list of 4d torch.Tensor is deprecated.Please concatenate the list along the batch dimension and pass it as a single 4d torch.Tensor
    warnings.warn(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================= 29 passed, 1 skipped, 57 warnings in 21.43s ==================

Jul 08 '24 05:07 IrohXu

Hi, does this inpainting pipeline support loras?

Aug 14 '24 15:08 baldesco

Hi, sadly no, it's missing the SD3LoraLoaderMixin.

Aug 14 '24 16:08 asomoza

Hi, I see that now SD3LoraLoaderMixin is included in the sd3 inpaint pipeline script. Does that mean that it supports LORAs now?

And is it possible to use the lora sd3 dreambooth script with the inpainting checkpoint?

Oct 16 '24 08:10 alvaro-budria

Add pipeline_stable_diffusion_3_inpaint.py for SD3 Inference

What does this PR do?

How to use it?

Who can review?