Hierarchical-Localization icon indicating copy to clipboard operation
Hierarchical-Localization copied to clipboard

Add semi-dense matching support (LoFTR)

Open Phil26AT opened this issue 3 years ago • 2 comments

  • LoFTR module (hloc/matchers/loftr.py)
  • semi-dense matching on image pairs (hloc/match_dense.py)
  • quantization to store features and matches in hloc format
  • supports quantization to pre-extracted features (e.g. superpoint)

Phil26AT avatar Mar 24 '22 12:03 Phil26AT

I've had some good results using this. I was wondering though, if there is a reason the matching is hard-coded to a batch size of 1? It occurs to me that more fully utilising the GPU & maybe allowing data-parallel across multiple GPUs might make working with this a lot faster / more practical.

I'm also slightly confused about the use of conf.top_k. It appears to be referenced, but not assigned?

reynoldscem avatar May 27 '22 10:05 reynoldscem

Is this still in the pipeline to be merged?

reynoldscem avatar Sep 21 '22 18:09 reynoldscem

Yes this is still in the pipeline to be merged, and should be ready now. Tested on Aachen v1.1 with setup loftr_aachen (no superpoint keypoints), which should run on machines with 32GB RAM:

Reconstruction:
        num_reg_images = 6697
	num_cameras = 4331
	num_points3D = 6106347
	num_observations = 29844267
	mean_track_length = 4.88742
	mean_observations_per_image = 4456.36
	mean_reprojection_error = 1.31786

Localization results:

Day: 88.7 / 96.4 / 99.0
Night: 75.9 / 90.6 / 99.0

Re batch size: I tested it and for me it only works with a batch size of 1 on Aachen (RTX 3080, 6 pairs / second). Nevertheless, I put the match extraction using LoFTR in a separate for-loop in case someone is willing to make this multi-GPU compatible.

Phil26AT avatar Sep 26 '22 17:09 Phil26AT

cc @JiamingSuen

sarlinpe avatar Oct 04 '22 09:10 sarlinpe