Current Reality Capture Nerfstudio Import process producing very bad quality for Splatfacto.

[Maxiviper117]

Describe the bug The current nerfstudio instructions on importing from Reality Capture at: https://docs.nerf.studio/quickstart/custom_dataset.html#realitycapture don't produce expected results.

I have a suspicion that the issue may be in using the exported CSV from Internal/External camera parameters export option.

~Does the CSV export take into account the distortion the images undergo as Reality Capture optimizes the alignment? Or is that do in Nerfstudio?~

This problem does not happen if using the colmap .txt and colmap.db format generated from the bundler .out and imagelist .lst exports with undistorted images, using the process shown here https://gist.github.com/jo-chemla/258e6e40d3d6c2220b29518ff3c17c40

Then taking that colmap format and using the instant-ngp colmap2nerf.py script to convert to a transforms.json file with this section of that script disabled in the colmap2nerf.py script:

if __name__ == "__main__":
	args = parse_args()
	#if args.video_in != "":  <---
	#	run_ffmpeg(args)  <---
	# if args.run_colmap:    <---
	#	run_colmap(args)  <---
	AABB_SCALE=int(args.aabb_scale)
	SKIP_EARLY=int(args.skip_early)
	IMAGE_FOLDER=args.images

Using this newely generate transforms.json file with the undistorted images, results is far greater quality (as shown in the screenshot) below.

To Reproduce Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior Expecting quality to not be this bad.

Screenshots If applicable, add screenshots to help explain your problem.

Additional context Add any other context about the problem here.

[jb-ye]

The converter code was created before splatfacto. I believe a possible issue could be related to porting point cloud from reality capture to ply data with proper coordinate transform.

Just want to confirm if nerfacto works for your reality capture data? If the quality is also bad, then maybe something has been broken.

[Maxiviper117]

I havent tried with Nerf but my only interest is with Gaussian, I doubt the Nerf would fare much better if the conversion is incorrect to begin with. After looking at the function that handles the conversion, I can see it's definitely overlooking a few things.

Reality Capture has documentation regarding the Principal Point and Distortion Models.

[jb-ye]

@Maxiviper117 I am not a user of reality capture, if you know how to make it work with splatfacto, please send a PR fixing that.

[Maxiviper117]

I'm not qualified to do that, closing issue hopefully someone else can help in future :(

[Maxiviper117]

Re-opening to leave it to another good samaritan.

[jb-ye]

I was able to fix this issue using #3131 , the screenshot is done at about 3k iterations