training_extensions icon indicating copy to clipboard operation
training_extensions copied to clipboard
verified publisher icon openvinotoolkit

Abnormal training progress

Open yunchu opened this issue 2 years ago • 3 comments

Describe the bug

While running a regression test for instance-segmentation, I've met the training progress exceed 100%. image

Steps to Reproduce

  1. prepare dataset for running regression test
  2. run a test CI_DATA_ROOT=<regression-dataset-root-path> tests-iseg-py310-pt1 -- tests/regression/instance_segmentation/test_instance_segmentation.py::TestRegressionInstanceSegmentation::test_otx_train[Custom_Counting_Instance_Segmentation_MaskRCNN_SwinT_FP16]
  3. monitor displayed log

Environment:

  • OS: Ubuntu 22.04 WSL
  • Framework version: torch 1.13
  • Python version: 3.10
  • OpenVINO version: 2022.3
  • CUDA/cuDNN version: 11.7.1
  • GPU model and memory: 3090 / 24G

yunchu avatar Sep 21 '23 05:09 yunchu

Can we close this issue? @sungmanc might have fixed this issue but need to check if it's applied to all release branches.

goodsong81 avatar Nov 06 '23 04:11 goodsong81

Can we close this issue? @sungmanc might have fixed this issue but need to check if it's applied to all release branches.

Is this issue solved ????

Fxisxl avatar Jan 06 '24 14:01 Fxisxl

Can we close this issue? @sungmanc might have fixed this issue but need to check if it's applied to all release branches.

Is this issue solved ????

In our tests , it seems that this was resolved. Found anything?

sungmanc avatar Jan 08 '24 01:01 sungmanc