Unable to train SDF with overhead images

[alberthli]

I'm trying to train an SDF using a series of photos taken from a bird's eye view of a table using a robotic arm. I'm able to get reasonable performance using nerfacto after only about a thousand iterations (you can at least recognize what the object is with some artifacts). However, after a day of messing with various models and hyperparameter settings, I have been unable to get any model from sdfstudio to train.

On nerfstudio, here's a typical output. I used the command

ns-train nerfacto --data . --max-num-iterations 1000 --output-dir . --pipeline.model.disable-scene-contraction True --pipeline.model.background-color white nerfstudio-data --auto-scale-poses False --scale-factor 1. --scene-scale 0.5 --center-method none --orientation-method none

On sdfstudio, here's a typical output. I trained it using a script with the following config (1000 iterations):

config = Config(
    method_name="neus-facto",
    trainer=TrainerConfig(
        relative_model_dir="",
        steps_per_eval_image=max_num_iterations + 1,  # no eval
        steps_per_eval_batch=max_num_iterations + 1,
        steps_per_save=max_num_iterations - 1,  # save right before finishing
        steps_per_eval_all_images=1000000,  # set to large value, don't eval
        max_num_iterations=max_num_iterations,
        mixed_precision=False,
    ),
    pipeline=VanillaPipelineConfig(
        datamanager=VanillaDataManagerConfig(
            dataparser=SdfStudioDataParserConfig(
                auto_orient=False,
            ),
            train_num_rays_per_batch=2048,
            eval_num_rays_per_batch=1024,
        ),
        model=NeuSFactoModelConfig(
            sdf_field=SDFFieldConfig(
                use_grid_feature=True,
                num_layers=2,
                num_layers_color=2,
                hidden_dim=256,
                bias=0.1,
                beta_init=0.3,
                use_appearance_embedding=False,
                inside_outside=False,
            ),
            background_model="none",
            eval_num_rays_per_chunk=1024,
            background_color="white",
        ),
    ),
    optimizers={
        "proposal_networks": {
            "optimizer": AdamOptimizerConfig(lr=1e-2, eps=1e-15),
            "scheduler": MultiStepSchedulerConfig(max_steps=max_num_iterations),
        },
        "fields": {
            "optimizer": AdamOptimizerConfig(lr=5e-4, eps=1e-15),
            "scheduler": NeuSSchedulerConfig(
                warm_up_end=int(0.025 * max_num_iterations),
                # warm_up_end=500,
                learning_rate_alpha=0.05,
                max_steps=max_num_iterations,
            ),
        },
        "field_background": {
            "optimizer": AdamOptimizerConfig(lr=5e-4, eps=1e-15),
            "scheduler": NeuSSchedulerConfig(
                warm_up_end=int(0.025 * max_num_iterations),
                # warm_up_end=500,
                learning_rate_alpha=0.05,
                max_steps=max_num_iterations,
            ),
        },
    },
    viewer=ViewerConfig(num_rays_per_chunk=1 << 15),
    vis="viewer",
)

Some details:

• my setup only has RGB image supervision. I can take depth images, but there are many times where the camera goes too close to the object to get a reliable reading.
• models I've tried in some capacity: neus-facto, volsdf, neus.
• assorted parameters I've tried tuning: the scene_box sizes in the process_nerfstudio_to_sdfstudio.py script, the pose scaling (tried both with and without the scaling), use_grid_feature, bias, beta_init, use_appearance_embedding, inside_outside, all types of background_model, background_color (also tried adding a white tablecloth to the tabletop in some runs).

Attached are data and the meta_data.json file associated with a typical run. Please let me know if there's something obvious I'm doing wrong. Thank you! debug.zip

Original transforms.json (zipped because github doesn't allow jsons): transforms.zip