HuguesTHOMAS
Share Training Log
Hi HuguesTHOMAS! Thank you for your awesome work! I followed readme.md and successfully test your pretrained model, but when I train your model, I found that the validation loss varied a lot during training process. For example, in epoch 400 I got mean IOU 68.6% on S3DIS validation data, but just after one epoch it changed to 60.8%. Is this Ok? Could anyone share his/her training log? Looking forward to your reply.
It could be normal, as the validation during training is performed on a small subset of the whole validation cloud. Depending on which part get picked, the score can be quite different. YOu should not refer to the printed message for evaluating your training performances.
Instead, you can use the plot_convergence.py script which plots both training metrics (time, loss), and then validation metrics. The validation metrics include the averaged score on the subsets picked at each epoch (the numbers you see printed), but also an overall score computed on the whole point cloud once in a while (only for epochs where a checkpoint is saved). This last score is the one that you should refer to for real performances.
Eventually, when your training is finished, you can use the test_models.py to have a final evaluation of the score with voting etc.
