Evaluation metrics - Object detection

[csudre]

Is your feature request related to a problem? Please describe. Need for the following metrics when dealing with object detection problems

Intersection over Union (IoU) measurements can either be made at the level of the bounding boxes of the objects or of the segmentation themselves.

At the element level

• Dice per element
• Centroid distance

At the level of the image considering independent elements:

• F1 score with IoU thresholds ranging from 0.5 to 0.95
• Average precision with IoU thresholds ranging from 0.5 to 0.95 https://cocodataset.org/#detection-eval
• Average recall with IoU thresholds ranging from 0.5 to 0.95 https://cocodataset.org/#detection-eval
• Correlation of detected volumes
• Number difference
• F1 with minimum overlap of 1 voxel
• Outline error https://link.springer.com/article/10.1186/1471-2342-12-17
• Detection error https://link.springer.com/article/10.1186/1471-2342-12-17

Additional context This list of metrics is the output of the initial brainstorming of the metrics task force

[mibaumgartner]

Amazing list of detection metrics, thanks for all the effort of the metrics task force to collect them!

I am curious about the F1 with minimum overlap of 1 voxel metric, could someone add some papers which use that metric? (I would like to read them ^^)

Also a small addition: The Free-Response Receiver Operating Characteristic (FROC) might also be a metric which should be added (used by LUNA , DeepLesion and RibFrac)

[iliathesmirnov]

Hello everyone,

I am new here and would like to contribute to implementing some (or all) of the evaluation metrics on @csudre's list. Did anyone start working on them? If not, should I create a pull request (labelled [WIP]) and get going?

Thank you, Ilia

[wyli]

Hello everyone,

I am new here and would like to contribute to implementing some (or all) of the evaluation metrics on @csudre's list. Did anyone start working on them? If not, should I create a pull request (labelled [WIP]) and get going?

Thank you, Ilia

welcome and please go ahead! (I don't think anyone is currently working on them)

[wyli]

today we discussed the requirement of FROC curves for lesion based evaluation

[Can-Zhao]

Please check https://github.com/Project-MONAI/MONAI/blob/dev/monai/apps/detection/metrics/coco.py