comfyanonymous
Feature/qwen eligen support
Pull Request: Add Entity-Level Image Generation (EliGen) for Qwen Image
Summary
This update implements Entity-Level Image Generation (EliGen) for the Qwen Image model, allowing region-specific prompts through spatial masks. The feature provides fine-grained control over image generation by applying separate attention masks for each entity.
Key Features • Spatial attention masking with isolated entity prompts • Automatic mask resizing to match latent dimensions • RoPE embedding implementation aligned with DiffSynth Studio • Support for batch_size > 1 • Backward compatible with standard Qwen Image workflows
Hey @nolan4 thank you so much for this amazing work! I am trying to test this out but not able to run it, I am using this PR and using the Eligen Lora provided by Diffsynth Studio and using this workflow, please check.
Hi @krigeta — here’s a screenshot of my workflow, which is based on the Qwen text-to-image template. I’m also using the same EliGen LoRA from Diffsynth Studio that you linked. Hope this helps you get it running!
Hey @nolan4 thank you so much for this, what if this branch is not merged then is it possible to create a custom node of this?
and yeah i will test this and share the results for sure.
Looks like you have multiple loras, try just one for testing purposes. I tried your prompt with a few minor modifications:
test locally and works, comfy will do a code review to see if anything else needs to be changed!
Looks like you have multiple loras, try just one for testing purposes. I tried your prompt with a few minor modifications:
Will gonna test it asap and this time I am gonna test the official example as shown in the official Eligen lora implementation and one more thing I want to ask:
In the Diffsynth studio repo, they said it is important to make those colored masks with text overlays. Is that true?
As in their official example, the masks are overlaid to achieve smooth results.
How can I get this node?
you have to install this PR and not the actual comfyUI setup to use this node, as of now it is not the part of the main repo.
Hey @nolan4, I guess this implementation is missing the colour-coded masks that help the lora to differentiate between the regions when they overlap. Please look into it.
btw something similar was implemented in the Inspire Pack called "regional conditioning by color masks" in case you need inspiration or code
https://github.com/ltdrdata/ComfyUI-extension-tutorials/blob/Main/ComfyUI-Inspire-Pack/workflow/RegionalCFG-RegionalConditioning.png
https://github.com/ltdrdata/ComfyUI-Inspire-Pack
How can I get this node?
comfy-cli --workspace ./ComfyUI_eligen install --pr "#10473"
comfy-cli --workspace ./ComfyUI_eligen launch
Looks like you have multiple loras, try just one for testing purposes. I tried your prompt with a few minor modifications
I can confirm it works: 8step lora -> eligen lora -> ksampler
I guess this implementation is missing the colour-coded masks that help the lora to differentiate between the regions when they overlap. Please look into it.
I can confirm it works
note i use euler+beta, cfg=1, 1328x1328 but that shouldn't make much difference
Hey @geroldmeisinger , thank you so much for the share, is there any other social media where we can chat?
here are the original masks https://www.modelscope.cn/datasets/DiffSynth-Studio/examples_in_diffsynth/files data/examples/eligen/entity_control
of the eligen demo page https://www.modelscope.cn/models/DiffSynth-Studio/Qwen-Image-EliGen
here are the original masks https://www.modelscope.cn/datasets/DiffSynth-Studio/examples_in_diffsynth/files
data/examples/eligen/entity_controlof the eligen demo page https://www.modelscope.cn/models/DiffSynth-Studio/Qwen-Image-EliGen
These are the ones I got to know about Eligen. In my case, the overlapped masking is not working properly, as I want to create two characters in front of each other, and the view is from the back. Or I would say I use one entity for a character and another for another character, and with ControlNet? Does it work?
And if possible, may you share your Discord or other social app?
Haha indeed, they are needed btw, I want to ask, are you able to get the result shown here?
@geroldmeisinger @krigeta
Latest version supports 8 entities! Below is a screenshot for the referenced DiffSynth example.
awesome! see https://docs.comfy.org/custom-nodes/backend/more_on_inputs#dynamically-created-inputs in case you want to make it dynamic
@krigeta https://github.com/krigeta/eligen_test/issues/1
There will be a native growing input type soon - any custom javascript to do this will not be allowed in core outside of a 'general' implementation! That will allow an indefinite amount of inputs into the node.