[Tencent Hunyuan Team] Add LoRA Inference Support for Hunyuan-DiT

[gnobitab]

Added LoRA support to HunyuanDiT pipeline. Currently can only support lora_scale=1.

You may test the PR with test_hunyuandit_lora.py. A pre-trained LoRA model is uploaded here: https://huggingface.co/XCLiu/hunyuandit-lora-test

Please change YOUR_LORA_PATH to the dir you store the downloaded lora file.

The generated image should be

@yiyixuxu @sayakpaul Please have a look! thank you so much!

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[gnobitab]

Thanks for the comments! I'll update the commit.

My question is: if we don't add cross_attention_kwargs, how do we control lora scale?

I will provide a doc update after all the changes

[sayakpaul]

I think the comment was about not propagating cross_attention_kwargs throughout the codebase. Just the Transfomer block is fine similar to: https://github.com/huggingface/diffusers/blob/d457beed92e768af6090238962a93c4cf4792e8f/src/diffusers/models/unets/unet_2d_condition.py#L1181

[sayakpaul]

@gnobitab please take note of https://github.com/huggingface/diffusers/pull/8670. It is going to help with this support too. IMO, it might be better to wait for a little while till that PR gets merged. Once done, we will surely revisit this.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.