Descriptor Synthesis by NeRF for D2S

Project Page | Paper

Demo

https://github.com/user-attachments/assets/98071e95-8ace-417e-a44c-58f5c62f6af8

Installation

The program is tested with Python 3.8+ and torch 1.13.1 and dependencies in requirements.txt.

You will also need to install the following from sources:

• nerfstudio
• lightglue
• Hloc
• Feat2map

Supported datasets

• Microsoft 7scenes
• 12scenes

How to use

The code is still under refactoring so it still contains hardcode and bugs.

• Hloc need to run first in order to obtain the sfm model (in this case we use triangulation from superpoint+superglue).
• Extract a subset of the dataset to train nerf on by preprocessing.py.
• Generate novel poses using create_novel_pose.py
• Generate synthetic images by using view_synthesis.py
• Generate descriptors by matching using generate_synthetic.py
• Finally training both synthetic and original data using Feat2map

Citation

Consider citing if you find this usefull

@article{bui2024leveraging,
  title={Leveraging Neural Radiance Field in Descriptor Synthesis for Keypoints Scene Coordinate Regression},
  author={Bui, Huy-Hoang and Bui, Bach-Thuan and Tran, Dinh-Tuan and Lee, Joo-Ho},
  booktitle={2024 IEEE/RSJ International Conference on Intelligent Robotics and Systems},
  year={2024}
}

Acknowledgement

We thank the author of Hloc, and nerfstudio for providing their opensource contribution.