Strange 3d reconstruction result

[zlukich]

Hi, I am trying to use your project to create verification station for 3D printed parts, to measure how precise object printed I compare point clouds of reconstructed object and ground truth point cloud. After training on several objects with different amount of training images and different parameters I get some cursed refined mesh output even though colmap scene is perfect. Also same setup using instant-ngp gives perfect result, so I am trying to understand where is the problem here.

As I understand standard output of SuGar .obj file from refined mesh represents Gaussian splats and not mesh in usual understanding, isn't it? Can problem lay in way how I open this file?

Thanks

[Anttwo]

Hello @zlukich,

Indeed, the resulting mesh looks quite bad. The output .OBJ file is a mesh, so you should get a good-looking result if you open it with Blender, for example. SuGaR should be able to produce better meshes/reconstructions than methods like Instant-NGP or NeuS.

So, let's investigate! May I ask a few questions:

1. Could you show me a few images of your dataset?
2. Did you segment the object in your input images? Currently, there's not mask compatibility implemented in SuGaR (but I'm planning to add this functionality in near future). Therefore, if you use masked images (with a monochrome, masked background), you will indeed get some chaotic geometry in the background. That could explain this cursed result. If so, you can try using the unmasked images.
3. You can try to change the lighting in the scene, as it looks very dark right now and we can't really see the geometry. You should try an "ambient lighting" with intensity 1.

[zlukich]

I think I need also explain a bit, that we don't move camera, my object rotates at rotating bed, while camera stays static. We tested it on several objects and scene configurations, and ended with ambient lighting from above and static background(black mate surface). With other reconstruction methods, it was easy to work when we removed background, so I tried it in your method.

1. In this specific case I used images with removed background (73 images), because my target is to get object without any scene background information, but after you said that, I think problem can lay here (in INGP it works though, mb it is somehow possible to run without background in Sugar?)

2. This is result with background

3. Ambient light is already in use, intensity can be modified

So to my intuition, scene can be Improved with lightning and capture better image quality, but also scene setup plays important role, because flat surface is corrupted, and when we try to remove background it optimizes on dark pixels somehow wrong. I would be glad, if you share your opinion on that, thanks :)

[Anttwo]

You're right, the setup is important; Here, your images are close-ups on the main object, placed on a monochrome floor. Unfortunately, a flat, monochrome surface like this generally results in chaotic reconstructions when using photogrammetry/Neural surface methods/SuGaR because there is too much ambiguity in the projective inverse problem (from a maths point of view, a chaotic geometry on the floor is a valid solution to solve this projective inverse problem).

I agree that in this case, using masks is the best way to solve the problem; Unfortunately, there's no mask functionality for SuGaR currently, I'm sorry for that. I really should implement a mask functionality as soon as possible. It's currently the top priority on my to-do list for this repo.

In the meantime, you could try to capture your object on a textured floor (you can add a piece of cloth or paper with texture/drawings on it), and you should be able to reconstruct the object without having such chaotic geometry.

Concerning Instant-NGP: when you say that the reconstruction looks good, do you mean the images rendered by the NeRF (since Instant-NGP does not provide a mesh)? Actually, achieving good-looking rendered images is easier than reconstructing a nice surface. For instance, if you visualize the hybrid representation of the SuGaR reconstruction rather than just the mesh, the result would probably be nice too. You can use our dedicated viewer to visualize the hybrid representation.

[zlukich]

You're right, the setup is important; Here, your images are close-ups on the main object, placed on a monochrome floor. Unfortunately, a flat, monochrome surface like this generally results in chaotic reconstructions when using photogrammetry/Neural surface methods/SuGaR because there is too much ambiguity in the projective inverse problem (from a maths point of view, a chaotic geometry on the floor is a valid solution to solve this projective inverse problem).

I agree that in this case, using masks is the best way to solve the problem; Unfortunately, there's no mask functionality for SuGaR currently, I'm sorry for that. I really should implement a mask functionality as soon as possible. It's currently the top priority on my to-do list for this repo.

In the meantime, you could try to capture your object on a textured floor (you can add a piece of cloth or paper with texture/drawings on it), and you should be able to reconstruct the object without having such chaotic geometry.

Concerning Instant-NGP: when you say that the reconstruction looks good, do you mean the images rendered by the NeRF (since Instant-NGP does not provide a mesh)? Actually, achieving good-looking rendered images is easier than reconstructing a nice surface. For instance, if you visualize the hybrid representation of the SuGaR reconstruction rather than just the mesh, the result would probably be nice too. You can use our dedicated viewer to visualize the hybrid representation.

No when I compare results with INGP I compare them taking in accountant mesh reconstruction(previous comparisons were built on sugar on masked dataasets, so it was incorrect assumption). I retrained model with flat surface but with different colorful ground, and here you can see results from sugar and ingp.

From the images(first is INGP, second Sugar) we also see that number of vertices and faces from INGP mush higher because we ran marching cubes algorithm with high resolution, but is there available for Sugar some kind of parameters to improve mesh quality, or adjust mesh reconstruction? Because as we can see here, that INGP algorithm sees eyes of frog nor just plain, but with this pit. Also is this possible to extract ply from gaussians?