UniAD icon indicating copy to clipboard operation
UniAD copied to clipboard

Add option to use legacy planning metric definition

Open wljungbergh opened this issue 1 year ago • 0 comments

As described in #166, there is a discrepancy between how planning metrics (L2 and collision rate) are computed in this repo vs previous works, such as VAD and ST-P3.

However, in the paper, these values are compared in the same table, leading to confusion. This PR enables computation of the UniAD metrics using the legacy definition used by VAD/ST-P3, making comparison easier. However, it still defaults to the "original" UniAD way of evaluating. This results in a ranking shift when comparing the methods, see the table below

Method L2 1s (m) L2 2s (m) L2 3s (m) CR 1s (%) CR 2s (%) CR 3s (%)
ST-P3 1.33 2.11 2.90 0.23 0.62 1.27
UniAD (your current metric) 0.51 0.98 1.65 0.10 0.15 0.61
UniAD (legacy ST-P3 metric) 0.42 0.64 0.91 0.07 0.10 0.22
VAD-Tiny 0.46 0.76 1.12 0.21 0.35 0.58
VAD-Base 0.41 0.70 1.05 0.07 0.17 0.41

wljungbergh avatar Mar 19 '24 10:03 wljungbergh