evaluation problem of pv-rcnn++ on waymo dataset

[sjtuljw520]

The evaluation results are weird as follows:

2022-09-10 13:24:55,922 INFO *************** Performance of EPOCH 30 ***************** 2022-09-10 13:24:55,923 INFO Generate label finished(sec_per_example: 0.0646 second). 2022-09-10 13:24:55,924 INFO recall_roi_0.3: 0.286138 2022-09-10 13:24:55,924 INFO recall_rcnn_0.3: 0.286463 2022-09-10 13:24:55,924 INFO recall_roi_0.5: 0.245043 2022-09-10 13:24:55,924 INFO recall_rcnn_0.5: 0.248979 2022-09-10 13:24:55,924 INFO recall_roi_0.7: 0.123475 2022-09-10 13:24:55,924 INFO recall_rcnn_0.7: 0.135775 2022-09-10 13:24:55,964 INFO Average predicted number of objects(40077 samples): 15.277 2022-09-10 13:26:17,472 INFO
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_1/AP: 0.0000 OBJECT_TYPE_TYPE_VEHICLE_LEVEL_1/APH: 0.0000 OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2/AP: 0.0000 OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2/APH: 0.0000 OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_1/AP: 0.0000 OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_1/APH: 0.0000 OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2/AP: 0.0000 OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2/APH: 0.0000 OBJECT_TYPE_TYPE_SIGN_LEVEL_1/AP: 0.0000 OBJECT_TYPE_TYPE_SIGN_LEVEL_1/APH: 0.0000 OBJECT_TYPE_TYPE_SIGN_LEVEL_2/AP: 0.0000 OBJECT_TYPE_TYPE_SIGN_LEVEL_2/APH: 0.0000 OBJECT_TYPE_TYPE_CYCLIST_LEVEL_1/AP: 0.0000 OBJECT_TYPE_TYPE_CYCLIST_LEVEL_1/APH: 0.0000 OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2/AP: 0.0000 OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2/APH: 0.0000

[sshaoshuai]

Can you post the full log file? Since the provided information is limited to localize the problem.

[sjtuljw520]

Thank you for replying. I find that the problem is due to the waymo dataset version (my dataset version is v1.0). Is there other method to obtain the evaluation results when I have generated the detection results which are saved in file "../val/result.pkl"? @sshaoshuai

[jihanyang]

https://github.com/open-mmlab/OpenPCDet/blob/bd96d39af2389478820b34db7ad0272e9bb205db/pcdet/datasets/waymo/waymo_dataset.py#L436-L439

[sjtuljw520]

我现在已经定位到问题的原因，是我下载的waymo数据版本问题。我现有的数据版本，frame.laser_labels[i].num_lidar_points_in_box都是0，导致评估时所有的gt_box都过滤掉了。 想问下我已经得到检测结果文件了，即results.pkl，可否通过其他软件库做评估，比如mmdetection3D里带的waymo评估工具（../evaluation/waymo_utils/compute_detection_metrics_main）？ 如果可以，如何操作？

[sshaoshuai]

Then you need to transform the result.pkl to the right format in the official Waymo evaluation tool compute_detection_metrics_main . But I still suggest you to use the latest Waymo Dataset (>= version 1.2), since the validation performance of version 1.0 is different with version 1.2. (The number of validation samples are also different.)

[sjtuljw520]

Hi, can you provide some suggestion or code script that can transform the result.pkl to the right format for compute_detection_metrics_main ? @sshaoshuai

[sshaoshuai]

Sorry we do not have such script since we directly evaluate result.pkl with our python evaluation interface codes as here https://github.com/open-mmlab/OpenPCDet/blob/master/pcdet/datasets/waymo/waymo_eval.py.

One alternative way is to calculate the attribute of here by using the codes here. But note that this can only get approximate results as the number of points may not be exactly the same with the provided number of Waymo dataset.

Nevertheless, I still recommend to use the latest Waymo data and re-generate the info for evaluation.

[sjtuljw520]

OK. Thanks for your kind help. :) @sshaoshuai

[sjtuljw520]

Do you have other way to download the waymo dataset? I found download from the offical website (google cloud) is very slow. @sshaoshuai

[sshaoshuai]

Sorry, I do not have another way.