myria3d icon indicating copy to clipboard operation
myria3d copied to clipboard

Incoherent logged val/iou value despite coherent confusion matrix and per-class IoUs when training model in DDP

Open CharlesGaydon opened this issue 2 years ago • 1 comments

Observed in DDP mode with 2x3 GPUs (not tested on single node training).

This is surprising, because the per-class IoUs are computed with a bespoke method from the confusion matrix of the MultiClassJaccardIndex object that computes mean IoU

https://github.com/IGNF/myria3d/blob/d52810eb6024c6b4dbb7816ff3be83fb1dd732b0/myria3d/models/model.py#L190

Mean IoU: image

Per-class: image

Might be linked to https://github.com/IGNF/myria3d/issues/108

CharlesGaydon avatar Apr 23 '24 07:04 CharlesGaydon

Note: maybe https://github.com/IGNF/myria3d/pull/124 has corrected this since the computation of the IoU is now performed using a different function. Feel free to close if you do not observe this behavior again.

CharlesGaydon avatar May 07 '24 08:05 CharlesGaydon