Unsuccessful cross-attention weight loading in Custom Diffusion with StableDiffusionAttendAndExcitePipeline

[daeunni]

Describe the bug

I load the trained pytorch_custom_diffusion_weights.bin using StableDiffusionAttendAndExcitePipeline, however, loading is failed. I am wondering if is it the same cause with this issue (https://github.com/huggingface/diffusers/issues/7261#issuecomment-1990962268) .

Reproduction

pipeline = StableDiffusionAttendAndExcitePipeline.from_pretrained( "CompVis/stable-diffusion-v1-4", torch_dtype=torch.float16, ).to("cuda")

cur_ckpt = '/d1/daeun/diffusers/examples/custom_diffusion/exp/cyw_step300' pipeline.unet.load_attn_procs(cur_ckpt, #+ '/checkpoint-300', # weight_name = 'model.safetensors' weight_name="pytorch_custom_diffusion_weights.bin" ) pipeline.load_textual_inversion(cur_ckpt + "/.bin", weight_name=".bin") pipeline.load_textual_inversion(cur_ckpt + "/.bin", weight_name=".bin") pipeline.load_textual_inversion(cur_ckpt + "/.bin", weight_name=".bin") pipeline.load_textual_inversion(cur_ckpt + "/.bin", weight_name=".bin") pipeline.load_textual_inversion(cur_ckpt + "/.bin", weight_name=".bin")

pipeline.scheduler = DDPMScheduler.from_config(pipeline.scheduler.config) generator = torch.Generator(device="cuda") prompt = "a cat playing with a duck toy in the grassland" image = pipeline( prompt, token_indices=[3, 8], #token_indices, num_inference_steps=100, guidance_scale=6.0, generator = generator,
eta=1.0 ).images[0]

Logs

No response

System Info

• diffusers version: 0.26.0.dev0
• Platform: Linux-4.15.0-99-generic-x86_64-with-glibc2.17
• Python version: 3.8.18
• PyTorch version (GPU?): 2.2.0+cu118 (True)
• Huggingface_hub version: 0.20.3
• Transformers version: 4.37.2
• Accelerate version: 0.26.1
• xFormers version: 0.0.24+cu118
• Using GPU in script?:
• Using distributed or parallel set-up in script?:

Who can help?

@sayakpaul

[sayakpaul]

Can we try to reduce the reproduction code?

Also, I don't see the error trace.

[daeunni]

Sure, here is it.

pipeline = StableDiffusionAttendAndExcitePipeline.from_pretrained(
    "CompVis/stable-diffusion-v1-4", torch_dtype=torch.float16,
).to("cuda")
cur_ckpt = '/d1/daeun/diffusers/examples/custom_diffusion/exp/cyw_step300'
pipeline.unet.load_attn_procs(cur_ckpt, 
                                weight_name="pytorch_custom_diffusion_weights.bin"
                                )
pipeline.load_textual_inversion(cur_ckpt + "/<new1>.bin", weight_name="<new1>.bin")
pipeline.load_textual_inversion(cur_ckpt + "/<new2>.bin", weight_name="<new2>.bin")
pipeline.scheduler = DDPMScheduler.from_config(pipeline.scheduler.config)
generator = torch.Generator(device="cuda")

prompt = "a <new1> cat playing with a <new2> duck toy in the grassland"
token_dict = pipeline.get_indices(prompt)
image = pipeline(
    prompt,
    token_indices=[3, 8], #token_indices,
    num_inference_steps=100,
    guidance_scale=6.0,
    eta=1.0
).images[0]

My point was I think loading the trained pytorch_custom_diffusion_weights.bin failed. I can generate multi-concept images from stablediffusion with the above code, however, I cannot generate personalized concept images because I cannot load CustomDiffusion unet weights.

How can I utilize attend-and-excite with CustomDiffusion and DreamBooth + LoRA?

[sayakpaul]

We don't have access to any of the stuff you're using here:

pipeline.unet.load_attn_procs(cur_ckpt, 
                                weight_name="pytorch_custom_diffusion_weights.bin"
                                )
pipeline.load_textual_inversion(cur_ckpt + "/<new1>.bin", weight_name="<new1>.bin")
pipeline.load_textual_inversion(cur_ckpt + "/<new2>.bin", weight_name="<new2>.bin")

Please make it accessible for us to fully test it.

You're yet to provide the error trace as well.

[Rbrq03]

Combing them seems a good idea. Could you share some images generated by this piece of code? I am not sure what it looks like.

[Rbrq03]

hey @daeunni, maybe #7294 helps

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[sayakpaul]

Closing because of inactivity.