MONAI icon indicating copy to clipboard operation
MONAI copied to clipboard

Enable Volume Generation from Concrete Labelmaps in MAISI ControlNet

Open nvahmadi opened this issue 1 year ago • 0 comments

Is your feature request related to a problem? Please describe. Image generation with MAISI

Describe the solution you'd like Afaiu, currently, MAISI can "only" generate volumes conditionally on controlnet_cond. These are relative size changes of certain anatomical labels. But it cannot generate a volume from a concrete labelmap that is given.

Describe alternatives you've considered In ControlNet's forward() method, I did not see an alternative that would allow embedding a concrete labelmap. The Diffusion Model requires embeddings from ControlNet, and cannot accept concrete labelmaps either.

Additional context Figure 2 in the MAISI paper also seems to indicate that ControlNet can only accept "task-specific conditions", and "Diffusion Model" depends on ControlNet embeddings.

nvahmadi avatar Jan 27 '25 16:01 nvahmadi