cvg
Cannot reproduce the same result as benchmarks
Hi, @Skydes
I try to do some further research based on hloc. However, I find it hard to get the same result as reported on leaderboard. I follow run script 'pipeline_Aachen.ipynb' with the same configurations, while I only change netVLAD retrieve number.
The results on aachen dataset are as followed: netvlad50: day: 89.7 / 95.9 / 98.7 night: 83.7 / 90.8 / 100.0 netvlad30: day: 88.8 / 95.4 / 98.3 night: 85.7 / 92.9 / 100.0 netvlad20: day: 88.0 / 94.9 / 97.8 night: 84.7 / 91.8 / 99.0
While the result on leaderboard is: day: 89.6 / 95.4 / 98.8 night: 86.7 / 93.9 / 100.0
I would like to know if there is some special setting to get the same result as the leaderboard? Thanks a lot for your help.
Hi @rann1018, The pipeline has some randomness, mostly due to the use of RANSAC i) in the two-view verification during SfM, and ii) in the absolute pose estimation for each query. Aachen (v1.0) has only 100 night queries so the results can vary quite a lot between different runs. Did you try to run the pipeline for NetVLAD-50 several times? This should be fast since the matching can be cached as it is deterministic.
You could also try to run the dev branch, which tries to decrease the randomness of the two-view verification. As far as I remember, the results in the leaderboard were the best out of 2 runs.
Hi, I also can't reproduce the wining result, I run twice，and get the same result. and someone also get the result below the leaderboard，such as https://github.com/cvg/Hierarchical-Localization/issues/5
-- | duc1 | duc2 my | 46.5 / 65.7 / 78.3 | 52.7 / 75.6 / 79.4 leaderboard | 49.0 / 69.2 / 79.8 | 53.4 / 77.1 / 80.9 do you know how to get the leaderboard result? thank you
hi, i only see the visualization of the results in pipeline_Aachen.ipynb. How can i get the results like " netvlad50: day: 89.7 / 95.9 / 98.7 night: 83.7 / 90.8 / 100.0 netvlad30: day: 88.8 / 95.4 / 98.3 night: 85.7 / 92.9 / 100.0 netvlad20: day: 88.0 / 94.9 / 97.8 night: 84.7 / 91.8 / 99.0".
