[FluxMultiControlNetModel] object has no attribute 'config

[darhsu]

Describe the bug

This commit on main https://github.com/huggingface/diffusers/commit/14a1b86fc7de53ff1dbf803f616cbb16ad530e45 seems to have broke FluxMultiControlNetModel. Reverting this commit fixes the issue on line pipeline_flux_controlnet.py:844

Referencing this issue: https://huggingface.co/Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro/discussions/20

Reproduction

base_model = "/archive/me_flux_shallow/FLUX.1-dev"
controlnet_model_union = 'Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro'

controlnet_union = FluxControlNetModel.from_pretrained(controlnet_model_union, torch_dtype=torch.bfloat16)
controlnet = FluxMultiControlNetModel([controlnet_union])

pipe = FluxControlNetPipeline.from_pretrained(base_model, controlnet=controlnet, torch_dtype=torch.bfloat16)
pipe.to("cuda:0")

prompt = 'A bohemian-style female travel blogger with sun-kissed skin and messy beach waves.'
control_image_depth = load_image("/home/me/repos/flux_control_net/assets_depth.jpg")
control_mode_depth = 2

control_image_canny = load_image("/home/me/repos/flux_control_net/assets_canny.jpg")
control_mode_canny = 0

width, height = control_image_depth.size


image = pipe(
    prompt, 
    control_image=[control_image_depth, control_image_canny],
    control_mode=[control_mode_depth, control_mode_canny],
    width=width,
    height=height,
    controlnet_conditioning_scale=0.6,
    num_inference_steps=24, 
    guidance_scale=3.5,
    generator=torch.manual_seed(42),
).images[0]

Logs

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[8], line 1
----> 1 image = pipe(
      2     prompt, 
      3     control_image=[control_image_depth, control_image_canny],
      5     control_mode=[control_mode_depth, control_mode_canny],
      6     width=width,
      7     height=height,
      8     controlnet_conditioning_scale=0.6,
      9     num_inference_steps=24, 
     10     guidance_scale=3.5,
     11     generator=torch.manual_seed(42),
     12 ).images[0]

File ~/.local/lib/python3.10/site-packages/torch/utils/_contextlib.py:116, in context_decorator.<locals>.decorate_context(*args, **kwargs)
    113 @functools.wraps(func)
    114 def decorate_context(*args, **kwargs):
    115     with ctx_factory():
--> 116         return func(*args, **kwargs)

File ~/.local/lib/python3.10/site-packages/diffusers/pipelines/flux/pipeline_flux_controlnet.py:844, in FluxControlNetPipeline.__call__(self, prompt, prompt_2, height, width, num_inference_steps, timesteps, guidance_scale, control_image, control_mode, controlnet_conditioning_scale, num_images_per_prompt, generator, latents, prompt_embeds, pooled_prompt_embeds, output_type, return_dict, joint_attention_kwargs, callback_on_step_end, callback_on_step_end_tensor_inputs, max_sequence_length)
    840 # broadcast to batch dimension in a way that's compatible with ONNX/Core ML
    841 timestep = t.expand(latents.shape[0]).to(latents.dtype)
    843 guidance = (
--> 844     torch.tensor([guidance_scale], device=device) if self.controlnet.config.guidance_embeds else None
    845 )
    846 guidance = guidance.expand(latents.shape[0]) if guidance is not None else None
    848 # controlnet

File ~/.local/lib/python3.10/site-packages/diffusers/models/modeling_utils.py:151, in ModelMixin.__getattr__(self, name)
    148     return self._internal_dict[name]
    150 # call PyTorch's https://pytorch.org/docs/stable/_modules/torch/nn/modules/module.html#Module
--> 151 return super().__getattr__(name)

File ~/.local/lib/python3.10/site-packages/torch/nn/modules/module.py:1729, in Module.__getattr__(self, name)
   1727     if name in modules:
   1728         return modules[name]
-> 1729 raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")

AttributeError: 'FluxMultiControlNetModel' object has no attribute 'config'

System Info

• 🤗 Diffusers version: 0.31.0.dev0
• Platform: Linux-5.10.225-213.878.amzn2.x86_64-x86_64-with-glibc2.31
• Running on Google Colab?: No
• Python version: 3.11.9
• PyTorch version (GPU?): 2.4.1+cu121 (True)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Huggingface_hub version: 0.25.0
• Transformers version: 4.44.2
• Accelerate version: 0.34.2
• PEFT version: 0.12.0
• Bitsandbytes version: not installed
• Safetensors version: 0.4.5
• xFormers version: 0.0.28.post1
• Accelerator: NVIDIA L40S, 46068 MiB
• Using GPU in script?:
• Using distributed or parallel set-up in script?:

Who can help?

@sayakpaul @yiyixuxu @DN6

also @liamdonnellymv for the original issue on huggingface

[yiyixuxu]

hey sorry we will merge this one soon https://github.com/huggingface/diffusers/pull/9507 that will fix this

[burgalon]

@yiyixuxu any I seem to be encountering this with FluxControlNetInpaintPipeline with latest main diffusers

[yiyixuxu]

yes! would you be interested in helping us fix it for inpaint? would need similar change to this I think https://github.com/huggingface/diffusers/pull/9586/files

[nemoooooooooo]

@darhsu try this: base_model = 'black-forest-labs/FLUX.1-dev' controlnet_model_union = 'InstantX/FLUX.1-dev-Controlnet-Union'

controlnet_union = FluxControlNetModel.from_pretrained(controlnet_model_union, torch_dtype=torch.bfloat16) controlnet = FluxMultiControlNetModel([controlnet_union]) # we always recommend loading via FluxMultiControlNetModel

controlnet.config = controlnet_union.config

pipe = FluxControlNetPipeline.from_pretrained(base_model, controlnet=controlnet, torch_dtype=torch.bfloat16)

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[sayakpaul]

Gentle ping @yiyixuxu. Is this resolved?

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[yiyixuxu]

seems to be fixed already! let us know if not and we will reopen :)