nerfacto: Weird circular crops and ripples (when using no scene contraction and no render crop)

[f-dy]

Describe the bug When using no scene contraction, I get weird empty discs (no rgb, no accumulation, no depth) in renders, unless I enable "crop viewport" or enable scene contraction. This happens even if the camera is inside the crop box. Those are always perfect discs in 3D (ellipses in 2D), which are aligned and centered on the 3 axes. crop may happen inside or outside the disc (see screenshots)

Also ripples appear in normals when not cropping (see images in comment below)

No issue with tensorf.

To Reproduce Steps to reproduce the behavior:

dataset="blender/lego"
model=nerfacto
model_opts=(--pipeline.model.predict-normals True) # for mesh extraction
model_opts+=(--experiment-name "${dataset}")
viewer_opts=(--vis viewer --viewer.websocket-port=7007)
data_opts=(blender-data --data data/"${dataset}")

# Disable camera optimization
model_opts+=(--pipeline.datamanager.camera-optimizer.mode off)
# For white-background images
model_opts+=(--pipeline.model.background-color white)
model_opts+=(--pipeline.model.proposal-initial-sampler uniform)
#model_opts+=(--pipeline.model.near-plane 2. --pipeline.model.far-plane 6.)
# All images in a 360 have the same exposure etc.:
model_opts+=(--pipeline.model.use-average-appearance-embedding False)
# Distortion loss from mipnerf360, prevents floaters:
model_opts+=(--pipeline.model.distortion-loss-mult 0)
# Disable scene contraction,since the scene is bounded:
model_opts+=(--pipeline.model.disable-scene-contraction True)

# Train with blender lego dataset:
timestamp=$(date "+%Y-%m-%d_%H%M%S")
ns-train "${model}" "${model_opts[@]}" --timestamp "${timestamp}" "${viewer_opts[@]}" "${data_opts[@]}"

• launch training
• move around until the discs appear. They often appear when moving close.
• all the maps have the same holes (depth, accumulation, rgb)
• all holes disappear when cropping.

Expected behavior no holes

Screenshots without "crop viewport": with crop viewport:

without crop: with crop:

you can get two discs if you look along a diagonal:

or even three: with crop:

 nvidia-smi
Wed Jun 21 23:56:30 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.85.12    Driver Version: 525.85.12    CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  On   | 00000000:00:1E.0 Off |                    0 |
| N/A   42C    P0    26W / 300W |      0MiB / 16384MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

[f-dy]

I think there's more. Here's a render of the pred_normals without cropping. Notice the ripples. Same with crop viewport, I get much less ripples: Looks like the no-crop render path has an issue

[f-dy]

I would tend to think that _intersect_with_sphere is the culprid, but the math looks ok

[gafniguy]

Hi @f-dy , have you had any success with Nerfacto on the object-centric datasets, without contraction, since posting?

[f-dy]

nope

[engrmusawarali]

@f-dy I am having similar problem. For example Splatfacto renders images perfectly. However, when I train nerfacto it fails to render any guidance? @tancik , @jb-ye @jkulhanek

Left-> Splatfacto Right-> Nerfacto

This is my ns-train run_command(f"ns-train nerfacto " f"--data {SCENE_DIR} " f"--vis viewer+tensorboard " f"--max-num-iterations 30000 " f"--pipeline.model.background_color 'random' " f"--pipeline.model.disable-scene-contraction 'True' " f"--pipeline.model.camera-optimizer.mode off " f"--steps-per-eval-image 100000000000000 " f"--steps-per-eval-batch 100000000000000 " f"--steps-per-eval-all-images 60000 " f"--output-dir {OUTPUT_DIR} "

[engrmusawarali]

@tancik , @jb-ye @jkulhanek

[ncan33]

I am also having major issues with object-centric dataset (images without any background features), even when ground truth COLMAP data is available. I've been working hard to isolate the cause of the issue, but no success. Does someone have any starting points or tips? Looks like this has been a long-standing issue

[engrmusawarali]

@ncan33 this works for some objects but for some it does not work. There is something wrong here.

@tancik @jb-ye @jkulhanek kindly help on this.

[engrmusawarali]

@ncan33 for me it works for alot of objects, for few it does not work.

You could add --pipeline.model.disable-scene-contraction True, if you are not adding already

[ncan33]

Likewise here, works for all of my objects if the pose estimation is correct. For me, this made all the difference: --pipeline.model.background_color "random"

[engrmusawarali]

I think think this pipeline.model.background_color "random" is suggested practice for alpha transparency carving. Yet, after this some issues happen.