MIT-SPARK
Splitting your current work from the ROS wrapper
Hi,
Thanks for your contribution and work. I am very interested in testing your code, but without using ROS wrapper. You mentioned splitting this work from ROS wrapper. I am wondering if this is possible.
Also, is it possible to use different NeRF archetictures? Like Zip-NeRF or Nerfacto. They are both implemented in pytorch.
Many thanks in advance.
Best regards, Isso
There are not many connection points between Loc-NeRF and ROS so it should be pretty easy to separate them. This should involve making changes in this script https://github.com/MIT-SPARK/Loc-NeRF/blob/main/src/nav_node.py. If you just want to run single image localization (iNeRF style experiments) then you just need to remove using rospy.get_param to get parameters, publishing particles to a ros msg (I used this to visualize the particles in rviz), and any other mention of ros such as initializing the ros node. Then you can just call nav_node.py.
Let me know if you have any issues with it and I'll be happy to assist!
You can use different NeRF architectures since Loc-NeRF just assumes access to some way to render rgb values for certain pixels in an image. It will take some work though to load in the new NeRF model interface and make sure coordinate conventions are right but other than that there shouldn't be much to do it. I think some of the forks have implemented this with NeRFacto such as https://github.com/naamiinepal/Loc-NeRF
@Dominic101 Many thanks for your response. I just checked the NERFacto fork of your work but it does not look complete. I opened an issue there and I hope I get a response soon.
I would love to integrate nerfacto with your work and test it. So, right now I have created a model and I am able to render images from the model, given a pose. My model is metrically scaled, since I used known poses from a simulator, I know the scale as well as the rotation matrix of my scene (from nerfstudio).
Could you guide me on how should I make sure the coordinate conventions are right? Also how do you choose which pixels to render in an image, is it done randomely?
I appreciate your help!
Pixels are chosen at random. An alternative is to do interest point/interest region sampling like iNeRF did, but when i tried this with Loc-NeRF I didn't see much if any improvement so I just kept it simple with random sampling (see here https://github.com/MIT-SPARK/Loc-NeRF/blob/main/cfg/llff.yaml#L13).
For coordinate systems, one check you can do to verify the coordinate system is correct is to render an image from Loc-NeRF and make sure it matches your sensor image. This can be done from the yaml files by setting this value to 1 (which will render an image after 1 iteration https://github.com/MIT-SPARK/Loc-NeRF/blob/main/cfg/llff.yaml#L96 and setting all noise values to 0 https://github.com/MIT-SPARK/Loc-NeRF/blob/main/cfg/llff.yaml#L84 and by setting no error in the initial guess by setting the translation and rotation to 0 here https://github.com/MIT-SPARK/Loc-NeRF/blob/main/src/nav_node.py#L78 and https://github.com/MIT-SPARK/Loc-NeRF/blob/main/src/nav_node.py#L82. This renders an image at the ground truth pose.
Many thanks for your response, Dominic! I'll give it a try !
My last question: it was mentioned in the paper, that the Loc-NeRF works without initial guess. Does the single image localization work without an initial guess as well?
Yes, it does. This yaml file will run the LLFF single shot localization experiment using global localization https://github.com/MIT-SPARK/Loc-NeRF/blob/main/cfg/llff_global.yaml. This is the config file that was used to generate the global localization results in Figure 3 of the paper which was doing single image localization.