myria3d
myria3d copied to clipboard
Incoherent logged val/iou value despite coherent confusion matrix and per-class IoUs when training model in DDP
Observed in DDP mode with 2x3 GPUs (not tested on single node training).
This is surprising, because the per-class IoUs are computed with a bespoke method from the confusion matrix of the MultiClassJaccardIndex object that computes mean IoU
https://github.com/IGNF/myria3d/blob/d52810eb6024c6b4dbb7816ff3be83fb1dd732b0/myria3d/models/model.py#L190
Mean IoU:
Per-class:
Might be linked to https://github.com/IGNF/myria3d/issues/108
Note: maybe https://github.com/IGNF/myria3d/pull/124 has corrected this since the computation of the IoU is now performed using a different function. Feel free to close if you do not observe this behavior again.