roboflow
Instance Segmentation Confusion Matrix
Search before asking
- [X] I have searched the Supervision issues and found no similar feature requests.
Description
The current evaluation api supports confusion matrix only for object detection. Giving a view on how the model is performing for instance segmentation per class is desirable.
Slight change in API expected:
@dataclass
class ConfusionMatrix:
matrix: np.ndarray
segmentationMatrix: np.ndarray
classes: List[str]
conf_threshold: float
iou_threshold: float
@classmethod
def from_detections(
cls,
predictions: List[sv.Detections],
target: List[sv.Detections],
classes: List[str],
conf_threshold: float = 0.3,
iou_threshold: float = 0.5
) -> ConfusionMatrix:
pass
@classmethod
def benchmark(
cls,
dataset: sv.DetectionDataset,
callback: Callable[[np.ndarray], sv.Detections],
conf_threshold: float = 0.3,
iou_threshold: float = 0.5
) -> ConfusionMatrix:
pass
def plot(self, target_path: str) -> None:
pass
Discussion points:
- Note that I'm not changing the previous name
matrixtodetectionMatrixfor backward compatibility. If we believe it's not necessary and brings readability for the long term, can make that change. - Should this be an opt-in feature. Not sure what the user behaviour is. If the 90 percentile user wants both let's not. If it's the other way around, we can instead make a separate confusion matrix class called
SegmentationConfusionMatrix
Use case
No response
Additional
No response
Are you willing to submit a PR?
- [X] Yes I'd like to help by submitting a PR!
@SkalskiP @hardikdava @kirilllzaitsev Would like to have your input on the API change. I have implemented mask_iou_batch for an iou score on masks. Everything else should be fairly straight forward
@mayankagarwals , hi, thanks for bringing up this use case. I like the idea of having different ConfusionMatrix subclasses, each implementing in their own way these methods:
- detections_to_tensor
- _validate_input_tensors
- evaluate_detection_batch
And these look generic enough to be left as is, except for documentation:
- from_detections
- from_tensors
- _drop_extra_matches
- benchmark
- plot
@SkalskiP , what do you think?
Maybe from_tensors is better implemented in a subclass, else our prediction and target batch needs to be an np array. This is fine for now as both polygons and bounding boxes are np arrays.
@classmethod
def from_tensors(
cls,
predictions: List[np.ndarray],
targets: List[np.ndarray],
classes: List[str],
conf_threshold: float = 0.3,
iou_threshold: float = 0.5,
) -> ConfusionMatrix:
We can keep the following generic:
from_detections
_drop_extra_matches
benchmark
plot
@mayankagarwals It would be great to have a single API sv.ConfusionMatrix which can support for object detection and segmentation confusion matrix. Something like check if masks exists then also find confusion matrix for masks, otherwise compute confusion matrix only for bounding box.
@hardikdava This is a design decision that we need to take based on what supervision ideology + user base is a. If we believe a more significant fraction of our users would want both object detection and segmentation confusion matrices computed every time they compute any confusion matrix, we should go ahead with the solution you have proposed b. If the use cases for these two matrices differ, it would make sense to have a separate function/class to compute it independently
Personally, I am with @hardikdava, the use case generally around these is to evaluate models, the more matrices available the better. The compute cost isn't the primary problem as these matrices are hardly used in production.
For now, I'll go ahead with this. @SkalskiP @kirilllzaitsev please let me know if you feel this is contradictory in anyway.
@SkalskiP Following up on this, let me know what you think. Would like to start dev soon