Could the nerfstudio provide comparison on public benchmarks?

[LZleejean]

Is your feature request related to a problem? Yes. Excellent works. We train the model on seveal data with nerfstudio and get good resutls. But I'm confused at the quantitative permance of nerfstudio. For other repos of NeRF, e.g. ngp_pl, instant-nsr-pl, torch-ngp, they give the quantitative comparion. Therefore, we can clearly know their actual performance. However, the actual performance of nerfstudio is not clear although we could get good results of custom data. Moreover, we also do not know which combination of several components achieves best performance. I think the qantitative comparison is helpful for researchers. Like #1560, we also found the performance of instant-ngp-bounded is bad on blender-data, e.g., only 14-17 PSNR for lego (36 papers). We follow the command of nerfacto in 'launch_train_blender.sh' and achieves 32.5 PSNR for lego. Moreover, we also test TensoRF for lego and achieves 32.5 PSNR (35-36 paper's).

Describe the solution you'd like Could the nerfstudio provide the comparison against SOTA reported in their papers, and detailed commands for reproducing the results.

Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

Additional context Add any other context or screenshots about the feature request here.

[tancik]

There are some result numbers and ablations provided in the paper - https://arxiv.org/abs/2302.04264

We have intentionally optimized nerfacto for fast optimization of real-world casually captured data (and provide a new dataset). While we can optimize the model to work well with datasets like the blender synthetic dataset, we find that these special case modifications (ie, assume white background, no pose optimization, bounded scene, no noise, ect) are not general and hurt when using real world data.

We are open to others optimizing the methods to other datasets and will happily include the metrics and associated commands in the repo (feel free to submit PRs). However this is not something that we will be focusing on in the short term. I realize this is probably not the answer you were looking for, but we simply don't have the capacity at the moment.

[emedinac]

I found this link @LZleejean maybe, it helps a bit. https://jkulhanek.com/nerfbaselines/blender

I found the same as you mentioned.

did you get a configuration to work with the Blender dataset? I was checking "lego" with instant-ngp. I got these results under 2 different configurations:

ns-train instant-ngp --data /NerfStudio/blender/lego --vis viewer+tensorboard --pipeline.datamanager.camera-optimizer.mode off blender-data

  "results": {
    "psnr": 17.47579002380371,
    "psnr_std": 4.081431865692139,
    "ssim": 0.845123827457428,
    "ssim_std": 0.06784264743328094,
    "lpips": 0.24575063586235046,
    "lpips_std": 0.10936141014099121,
    "num_rays_per_sec": 1374682.0,
    "num_rays_per_sec_std": 160299.671875,
    "fps": 2.1479403972625732,
    "fps_std": 0.25046825408935547
  }

ns-train instant-ngp --data /NerfStudio/blender/lego --vis viewer+tensorboard --pipeline.model.near-plane 2. --pipeline.model.far-plane 6. --pipeline.model.camera-optimizer.mode off --pipeline.model.use-average-appearance-embedding False blender-data

"results": {
    "psnr": 15.03935718536377,
    "psnr_std": 8.094335556030273,
    "ssim": 0.8235980272293091,
    "ssim_std": 0.08926142752170563,
    "lpips": 0.27945783734321594,
    "lpips_std": 0.16361458599567413,
    "num_rays_per_sec": 383626.25,
    "num_rays_per_sec_std": 119828.6640625,
    "fps": 0.5994160175323486,
    "fps_std": 0.18723228573799133
  }