Request to Update Code to Reflect Predicted Density Usage in FeatureNeRF.

[minjung-s]

Thank you for sharing your excellent work and making the code available! While studying the paper and the corresponding implementation, I noticed a potential inconsistency that I would like to clarify.

In the paper, it is mentioned:

"Further, in a particular encoder or decoder block, we use the density predicted by previous FeatureNeRF blocks to importance sample the points along the ray for the next FeatureNeRF block 90% of the times."

However, upon inspecting the code (specifically [this section](https://github.com/customdiffusion360/custom-diffusion360/blob/main/sgm/modules/diffusionmodules/openaimodel.py#L1054-L1077)), it seems that the predicted density from the previous FeatureNeRF blocks is not used for importance sampling.

If this is an oversight, could you please update the implementation to reflect the described behavior in the paper?

Thank you for your time and for the great work!

[minjung-s]

I have submitted a pull request (PR #3) that addresses this issue by implementing the predicted density usage for importance sampling as described in the paper. I would greatly appreciate it if you could review the changes and share your feedback :)

[nupurkmr9]

Hi @minjung-s , Thanks for your interest in our paper and pointing this out.

I only used the importance sampling from the previous FeatureNeRF output in a specific transformer block (which in the case of SDXL, consists of ~10 attention layers with 3 of them as pose-conditioned attention layers) and not across different blocks in the U-Net. Will clarify this implementation detail in the paper as well.

Since, different layers in the diffusion has been shown to focus on different aspects of image generation, e.g., high-frequency features vs overall shape, thus the current design choice. But, propagating it across all layers in the U-Net might be good to try and compare against the current setup. Will post here when I can do this comparison. Will also take a look at the PR and enable that as a flag in the method.

[minjung-s]

Thank you for your detailed explanation. I understand that importance sampling is applied only to specific FeatureNeRF blocks, as in [this part of the code](https://github.com/customdiffusion360/custom-diffusion360/blob/main/sgm/modules/attention.py#L781).

However, I noticed that in the raymarcher, the imp_sample_next_step option is not being provided as an input ([reference](https://github.com/customdiffusion360/custom-diffusion360/blob/main/sgm/modules/nerfsd_pytorch3d.py#L442)). As a result, weights for the next step are not being output.

I believe incorporating the changes in my PR could enable importance sampling for the next step by considering the previous weights. I would greatly appreciate it if you could review the PR and share your thoughts.

Thank you for your time and support!