nerfstudio-project
Allow gaussian splatting to initialize start points from other data pipelines than COLMAP
Currently, gaussian splatting intakes a pointcloud for initialization either through: 1) using a COLMAP dataset, or 2) adding a pointcloud to transforms.json and using the nerfstudio dataparser to read it. Currently, data intake pipelines like polycam, metashape, etc do not output a pointcloud and thus gaussian splatting will use a random initialization of points, which produces much worse results.
To fix this, we should edit the data intake scripts to add point clouds to the output transforms.json files and use the nerfstudio dataparser to read them. #2557 added the capability to load .ply files in the nerfstudio dataparser
- [x] Add to the COLMAP data processing pipeline to save pointclouds of the SFM points as output, and add it to the
transforms.json - [x] Switch to nerfstudio dataparser as the default for splatting
- [ ] implement pointcloud saving in other data processing pipelines
FYI, metashape has a script for exporting colmap dataset (undistorted images): https://github.com/agisoft-llc/metashape-scripts/blob/master/src/export_for_gaussian_splatting.py
@kerrj is it your experience that the guassian-splat pipeline runs, just with reduced quality, when using other data sources? What I'm seeing ATM is that when using polycam as input an error is thrown.
assert colmap_path.exists(), f"Colmap path {colmap_path} does not exist."
AssertionError: Colmap path Garage/colmap/sparse/0 does not exist.
I am also getting this when using RealityCapture. It would be super to get this up and running. If we could export the point cloud that Reality Capture generates would that help?
assert colmap_path.exists(), f"Colmap path {colmap_path} does not exist."
AssertionError: Colmap path C:\Users\1user\nerfstudio\data\nerfstudio\farmhouse\colmap\sparse\0 does not exist.
(nerfstudio) C:\Users\Tekuser\nerfstudio>ns-train nerfacto --data C:\Users\1user\nerfstudio\data\nerfstudio\farmhouse [15:24:27] Using --data alias for --data.pipeline.datamanager.data
I think these errors are because the default dataparser for gaussian-splatting is set to colmap, while all other models set it to nerfstudio-data. There are plans to transition splatting over to the latter.
Specifying the nerfstudio-data subcommand should help fix assert colmap_path.exists() errors: ns-train gaussian-splatting --data /path/to/a/processed/dataset nerfstudio-data
That worked and I was able to train a Gaussian splat! Next issue I have is outputting the splat to PLY I get this error below. Would anyone know how to solve this?
(nerfstudio) C:\Users\1user\nerfstudio>ns-export gaussian-splat --load-config C:\Users\1user\nerfstudio\outputs\farmhouse\gaussian-splatting\2024-01-19_003623\config.yml --output-dir C:\Users\1user\nerfstudio\outputs\farmhouse\gaussian-splatting\2024-01-19_003623\PLY
Traceback (most recent call last):
File "C:\Users\1user\miniconda3\envs\nerfstudio\lib\runpy.py", line 194, in run_module_as_main
return run_code(code, main_globals, None,
File "C:\Users\1user\miniconda3\envs\nerfstudio\lib\runpy.py", line 87, in run_code
exec(code, run_globals)
File "C:\Users\1user\miniconda3\envs\nerfstudio\Scripts\ns-export.exe_main.py", line 4, in
Update on this.... items 1 and 2 are done thanks to @CardiacMangoes, it should be pretty straightforward (but a bit tedious) to update other pipelines to output a pointcloud .ply file and add it to the transforms.json
Hi - short question:
So I use the https://github.com/agisoft-llc/metashape-scripts/blob/master/src/export_for_gaussian_splatting.py to export the metashape project and then wanted to preprocess it with https://github.com/nerfstudio-project/nerfstudio/pull/3122#issue-2274206385 but i need additionally to export the ply point cloud, or? I probably will export the dense model as ply
Perfect! But sorry for asking - what is right now the suggested pipeline out of Metashape and/or Reality Capture? After the finished processing with the linked script creates already .bin files, but I cannot relate that to any documented pre/processing. Thank you very much for any up-to-date guidance or a documention.
@SchmidL Use ns-process-data to convert metashape / RC exports into nerfstudio format (i.e. https://github.com/nerfstudio-project/nerfstudio/pull/3122#issue-2274206385 and https://github.com/nerfstudio-project/nerfstudio/pull/3131#issue-2286033201), and train with the nerfstudio formattedd data.
@SchmidL Use ns-process-data to convert metashape / RC exports into nerfstudio format (i.e. #3122 (comment) and #3131 (comment)), and train with the nerfstudio formattedd data.
@jb-ye Hi, I am wondering how other open-sourced Gaussian Splatting methods can be used with nerfstudio formatted data. Quite a lot of them use input in COLMAP data format. Are there any methods that can convert nerfstudio format into COLMAP? I did do some searching but I cannot find one. Or just write a dataloader for those methods? Thank you for your time.
