comfyanonymous
ControlNet - doesn't work with difference models
ControlNet comes in two variations: The full models (5.2 GB) which contain the full ControlNet network together with weights from the model it was trained with, and difference models which are made by subtracting the model it was trained on from the ControlNet model (e.g: ControlNet trained on SD1.5 -> Difference = ControlNet - SD.15) as mentioned here and described in the linked discussions there. In ComfyUI, only the full models seem to work. This is most likely because of the implementation, which I quite frankly have no idea of. Therefor I unfortunately cannot provide a solution suggestion. It if can't be fixed that's fine as well, the normal models are good enough, just unnecessarily big. In that case it would suffice to mention it somewhere, because some people will only have the difference models and will probably be wondering why it doesn't work.
Yes loading the diff models needs another node that also takes a model as input that I have yet to add.
These small regular ones ones should work fine though: https://huggingface.co/webui/ControlNet-modules-safetensors/tree/main
Should be fixed with: https://github.com/comfyanonymous/ComfyUI/commit/62df8dd62a90d2842f3ce16ddbaa8c4c6232af8f