error is raise when training tilling ins-seg models with coco subset.
Describe the bug
When training tilling ins-seg model with "coco decremental dataset" which is used for regression test dataset, an error is raised due to index out of range error in dataset.
Because ins-seg models now use coco pretrained model weight, when using sub dataset of coco, task adaptive algorithm is executed.
Then, ClsIncrSampler replaces existing sampler and ClsIncrSampler decides dataset size using new_indices and old_indices from OTXDetDataset.
This can cause two abnormal cases:
- If tilling dataset size is bigger than original dataset size, only original dataset size of tilling dataset is used.
- If tilling dataset size is smaller than original dataset , Index out of range error is raised. (this is our case)
I think current tilling algorithm doesn't consider task adapt case. So, code about that needs to be updated, I think.
ps. Truning off adaptive batch size with --learning_parameters.auto_adapt_batch_size None can makes debugging easier.
Steps to Reproduce
- training a tilling ins-seg models with subset of coco.
Environment:
- OS: Ubuntu 20.04
- Framework version: Torch 1.13
- Python version: 3.10
- OpenVINO version: 2023.0
- CUDA/cuDNN version: 11.7
- GPU model and memory: RTX3090, 24GB