training_extensions icon indicating copy to clipboard operation
training_extensions copied to clipboard

error is raise when training tilling ins-seg models with coco subset.

Open eunwoosh opened this issue 2 years ago • 0 comments

Describe the bug When training tilling ins-seg model with "coco decremental dataset" which is used for regression test dataset, an error is raised due to index out of range error in dataset. Because ins-seg models now use coco pretrained model weight, when using sub dataset of coco, task adaptive algorithm is executed. Then, ClsIncrSampler replaces existing sampler and ClsIncrSampler decides dataset size using new_indices and old_indices from OTXDetDataset. This can cause two abnormal cases:

  1. If tilling dataset size is bigger than original dataset size, only original dataset size of tilling dataset is used.
  2. If tilling dataset size is smaller than original dataset , Index out of range error is raised. (this is our case)

I think current tilling algorithm doesn't consider task adapt case. So, code about that needs to be updated, I think. ps. Truning off adaptive batch size with --learning_parameters.auto_adapt_batch_size None can makes debugging easier.

Steps to Reproduce

  1. training a tilling ins-seg models with subset of coco.

Environment:

  • OS: Ubuntu 20.04
  • Framework version: Torch 1.13
  • Python version: 3.10
  • OpenVINO version: 2023.0
  • CUDA/cuDNN version: 11.7
  • GPU model and memory: RTX3090, 24GB

eunwoosh avatar Jul 20 '23 05:07 eunwoosh