vladmandic
[Feature]: ControlNet Omni pipeline
Feature description
This is opened as a feature just to keep track of things.
The recently-released ControlNet Omni model may require a specific Diffusers pipeline (or similar approaches) for inference (https://github.com/xinsir6/ControlNetPlus/tree/main?tab=readme-ov-file#controlmode), but it's not clear (clarification asked: https://huggingface.co/xinsir/controlnet-union-sdxl-1.0/discussions/8).
If it is required, it would be nice to have it in SD.Next as one single ControlNet model would make its use considerably easier. If it is not required, this issue can be simply closed.
Version Platform Description
Latest dev as of 2024-07-09.
There's a bunch of code at https://github.com/xinsir6/ControlNetPlus/blob/main/pipeline/pipeline_controlnet_union_sd_xl.py but I have no idea how much this is copied from diffusers and where the "secret sauce" lies.
i've already added support for new xinsir controlnet union model in dev branch and it works fine with standard controlnet pipeline. the new "enhanced" pipeline is not required, but it is recommended to allow for more granular control - i do plan to add it as well, but its not a blocker for adoption of the new model.
