[Alpha-VLLM Team] Add Lumina-T2X to diffusers

[PommesPeter]

What does this PR do?

Add Lumina-T2X to diffusers

Fixes https://github.com/huggingface/diffusers/pull/8652

Before submitting

• [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• [x] Did you read the contributor guideline?
• [x] Did you read our philosophy doc (important for complex PRs)?
• [x] Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
• [x] Did you make sure to update the documentation with your changes? Here are the documentation guidelines, and here are tips on formatting docstrings.
• [x] Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag members/contributors who may be interested in your PR.

[zhuole1025]

I reviewed Attenton and LuminaAttnProcessor2_0. looking very nice! I Ieft some questions:) most importantly I want to understand the kv_heads variable we added to Attention - Is this based on your research or some other paper? why do we give k and v smaller dimensions and then duplicate them for the attention calculation?

Yes, this is called Grouped Query Attention proposed in this paper, which can optimize training and inference efficiency.

[PommesPeter]

We have fixed all the problems above. should we go next?

[yiyixuxu]

@PommesPeter thanks! can we fix the failing CI? looking at it, I think you need to:

1. add the new doc pages to https://github.com/huggingface/diffusers/blob/main/docs/source/en/_toctree.yml
2. run make style and make fix-copies

we will wait for @DN6 to do a review also in the meantime!

[PommesPeter]

@PommesPeter thanks! can we fix the failing CI? looking at it, I think you need to:

1. add the new doc pages to https://github.com/huggingface/diffusers/blob/main/docs/source/en/_toctree.yml
2. run make style and make fix-copies

we will wait for @DN6 to do a review also in the meantime!

okay, we have added all our docs and run make style and make fix-copies for current branch.

[yiyixuxu]

can you run make fix-copies again?

[PommesPeter]

can you run make fix-copies again?

run it~

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[yiyixuxu]

@PommesPeter lumina tests still fail I think we need to update the lumina tests now because we made updates to the model

[PommesPeter]

@PommesPeter lumina tests still fail I think we need to update the lumina tests now because we made updates to the model

yep, we have fixed the problem from test class.

[yiyixuxu]

@PommesPeter we need the make style again, sorry!

[vladmandic]

not sure what's the status of the pr since simple load is still failing?

pip install git+https://github.com/PommesPeter/diffusers@lumina

import torch
from diffusers import LuminaText2ImgPipeline
pipe = LuminaText2ImgPipeline.from_pretrained("Alpha-VLLM/Lumina-Next-SFT-diffusers", torch_dtype=torch.bfloat16).cuda()

ValueError: Cannot load <class 'diffusers.models.transformers.lumina_nextdit2d.LuminaNextDiT2DModel'> from /mnt/models/Diffusers/models--Alpha-VLLM--Lumina-Next-SFT-diffusers/snapshots/f82702c1b6a9bac3db9155edad1fd8dbf088cdf6/transformer because the following keys are missing:
...

[PommesPeter]

not sure what's the status of the pr since simple load is still failing?

pip install git+https://github.com/PommesPeter/diffusers@lumina

import torch
from diffusers import LuminaText2ImgPipeline
pipe = LuminaText2ImgPipeline.from_pretrained("Alpha-VLLM/Lumina-Next-SFT-diffusers", torch_dtype=torch.bfloat16).cuda()

ValueError: Cannot load <class 'diffusers.models.transformers.lumina_nextdit2d.LuminaNextDiT2DModel'> from /mnt/models/Diffusers/models--Alpha-VLLM--Lumina-Next-SFT-diffusers/snapshots/f82702c1b6a9bac3db9155edad1fd8dbf088cdf6/transformer because the following keys are missing:
...

sorry, we have network problem in pushing our newest model to huggingface. I'm re-pushing the newest model for lumina.

[PommesPeter]

not sure what's the status of the pr since simple load is still failing?

pip install git+https://github.com/PommesPeter/diffusers@lumina

import torch
from diffusers import LuminaText2ImgPipeline
pipe = LuminaText2ImgPipeline.from_pretrained("Alpha-VLLM/Lumina-Next-SFT-diffusers", torch_dtype=torch.bfloat16).cuda()

ValueError: Cannot load <class 'diffusers.models.transformers.lumina_nextdit2d.LuminaNextDiT2DModel'> from /mnt/models/Diffusers/models--Alpha-VLLM--Lumina-Next-SFT-diffusers/snapshots/f82702c1b6a9bac3db9155edad1fd8dbf088cdf6/transformer because the following keys are missing:
...

sorry, we have network problem in pushing our newest model to huggingface. I'm re-pushing the newest model for lumina.

Hi @vladmandic we have pushed our model to huggingface repo. could you re-pull the huggingface model repo for testing you want?

[yiyixuxu]

@PommesPeter can you check if you need to update the slow tests? since the checkpoints have been updated a couple of times I will merge it tomorrow once the slow tests are updated

[PommesPeter]

@PommesPeter can you check if you need to update the slow tests? since the checkpoints have been updated a couple of times I will merge it tomorrow once the slow tests are updated

okay, i will fix the problem.

[vladmandic]

@PommesPeter confirmed as working with updated model on hf.

[yiyixuxu]

merged! thank you!

[PommesPeter]

wow! thank you for your reviewing to our pr