feat: Add transformer_function and sampler_cfg_rescaler hook

[JettHu]

The authors of the T-GATE introduced a speed up tech, like DeepCache

We find that cross-attention outputs converge to a fixed point during the initial denoising steps.

Ignoring text conditions in the fidelity-improving stage not only reduces computation complexity, but also slightly decreases FID score. This yields a simple and training-free method called TGATE for efficient generation, which caches the cross-attention output once it converges and keeps it fixed during the remaining inference steps.

Major Features

• Training-Free.
• Easily Integrate into Existing Frameworks.
• Only a few lines of code are required.
• Friendly support CNN-based U-Net, Transformer, and Consistency Model
• 10%-50% speed up for different diffusion models.

Implementation details

I have implement a T-GATE comfyui node, but introduced some hooks. More detail of T-GATE nodes see T-GATE comfyui node.

Resolves https://github.com/comfyanonymous/ComfyUI/issues/3285

[JettHu]

hi, I wanted to kindly ask if you might have a moment to review the pull request I've submitted.

latest news: TGATE [v0.1.1] is officially added to diffusers.

[comfyanonymous]

I think it would be better if the set_model_transformer_function() was like the def set_model_attn1_replace(self, patch, block_name, number, transformer_index=None):

model_options["sampler_cfg_rescaler"]({"cond_scale": cond_scale, "timestep": timestep})

should be: model_options["sampler_cfg_rescaler"]({"cond_scale": cond_scale, "sigma": timestep})

[JettHu]

Is it a good design that to include attn1_patch and attn1_output_patch in the new replace hook function

[JettHu]

I adjusted the implementation.

• Change the model_options["sampler_cfg_rescaler"] calling parameters to {"cond_scale": cond_scale, "sigma": timestep}
• Removed transformer_function, and added attn_function_wrapper in ModelPatcher which like the def set_model_attn1_replace(self, patch, block_name, number, transformer_index=None).

Does this meet your expectations?

[JettHu]

Hey I adjusted my PR, do you have time to take a look?

[JettHu]

Prepare to release another implementation using monkey patch instead