ControlNet speed improvement PR breaks multi-controlNet support

[SaladDays831]

I have an iPadOS project that uses Canny + Inpaint ControlNets

Pipeline:

StableDiffusionPipeline(resourcesAt: resourceURL,
                                         controlNet: ["LllyasvielControlV11PSd15Inpaint", "LllyasvielSdControlnetCanny"],
                                         configuration: configuration,
                                         disableSafety: true,
                                         reduceMemory: true)

After updating the package in my project to the latest main branch to include the change in this PR, the second ControlNet seems to be ignored.

Below are examples of the generated images. The only change in my codebase is the commit dependency rule of this package

Canny ControlNet input image:

Before PR	After PR

When switching places of the ControlNets in the configuration like so:

...
controlNet: ["LllyasvielSdControlnetCanny", "LllyasvielControlV11PSd15Inpaint"],
...

the first one (canny) works, but the second (inpaint) gets ignored again

@JustinMeans tagging you as the author of that PR. Does it work as expected for you?

[SaladDays831]

This commit needs to be reverted/fixed. Using multiple controlNets is impossible at the moment