Why mAP drops on MapTR trained with CAMA?

[soytony]

Hi there!

Really appreciate the work :)

I noticed that MapTR model trained with CAMA exhibited better performance in terms of SRE and PR scores. However, mAP on the three classes of map elements showed a decline compared with that on the "nosier" nuScenes dataset. And I am curious why is this happening.

Looking forward to your reply! Thanks!

[manymuch]

Thanks for the interest, since the validation data has different labeled ground truth(cama vs nuscenes), thus the mAP metrics can not be compared fairly. Besides, the nuScenes train/val splits have many geo-overlaps, thus the model can easily achieve high validation mAP by "memorizing the map" (overfitting to the map). We are working on the version2 of CAMA labeled data by reconstructing multiple scenes in the same map (sec V in paper) and refine the MapTR model training.