microsoft
No evaluation seems to be happening on validation set with custom dataset
Hi, I don't see any results on the validation set. I've created a custom dataset with 1 class. On the stdout, I'm seeing this every 4000 iterations:
2022-03-24 18:24:46,351 - mmdet.ssod - INFO - Iter [12000/720000] lr: 1.000e-02, eta: 2 days, 21:06:52, time: 0.365, data_time: 0.014, memory: 2890, ema_momentum: 0.9990, sup_loss_rpn_cls: 0.0000, sup_loss_rpn_bbox: 0.0000, sup_loss_cls: 0.0000, sup_acc: 100.0000, sup_loss_bbox: 0.0000, unsup_loss_rpn_cls: 0.0000, unsup_loss_rpn_bbox: 0.0000, unsup_loss_cls: 0.0000, unsup_acc: 100.0000, unsup_loss_bbox: 0.0000, loss: 0.0000
[ ] 0/200, elapsed: 0s, ETA:/vc_data/users/lashar/fhl_spring2022/reinstalled/SoftTeacher/thirdparty/mmdetection/mmdet/datasets/pipelines/formating.py:7: UserWarning: DeprecationWarning: mmdet.datasets.pipelines.formating will be deprecated, please replace it with mmdet.datasets.pipelines.formatting.
warnings.warn('DeprecationWarning: mmdet.datasets.pipelines.formating will be '
/vc_data/users/lashar/fhl_spring2022/reinstalled/SoftTeacher/thirdparty/mmdetection/mmdet/datasets/pipelines/formating.py:7: UserWarning: DeprecationWarning: mmdet.datasets.pipelines.formating will be deprecated, please replace it with mmdet.datasets.pipelines.formatting.
warnings.warn('DeprecationWarning: mmdet.datasets.pipelines.formating will be '
/vc_data/users/lashar/fhl_spring2022/reinstalled/SoftTeacher/thirdparty/mmdetection/mmdet/datasets/pipelines/formating.py:7: UserWarning: DeprecationWarning: mmdet.datasets.pipelines.formating will be deprecated, please replace it with mmdet.datasets.pipelines.formatting.
warnings.warn('DeprecationWarning: mmdet.datasets.pipelines.formating will be '
[ ] 2/200, 0.2 task/s, elapsed: 10s, ETA: 1013s/vc_data/users/lashar/fhl_spring2022/reinstalled/SoftTeacher/thirdparty/mmdetection/mmdet/datasets/pipelines/formating.py:7: UserWarning: DeprecationWarning: mmdet.datasets.pipelines.formating will be deprecated, please replace it with mmdet.datasets.pipelines.formatting.
warnings.warn('DeprecationWarning: mmdet.datasets.pipelines.formating will be '
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 200/200, 15.3 task/s, elapsed: 13s, ETA: 0s2022-03-24 18:25:00,201 - mmdet.ssod - INFO - Evaluating bbox...
Loading and preparing results...
2022-03-24 18:25:00,202 - mmdet.ssod - ERROR - The testing results of the whole dataset is empty.
[ ] 0/200, elapsed: 0s, ETA:/vc_data/users/lashar/fhl_spring2022/reinstalled/SoftTeacher/thirdparty/mmdetection/mmdet/datasets/pipelines/formating.py:7: UserWarning: DeprecationWarning: mmdet.datasets.pipelines.formating will be deprecated, please replace it with mmdet.datasets.pipelines.formatting.
warnings.warn('DeprecationWarning: mmdet.datasets.pipelines.formating will be '
/vc_data/users/lashar/fhl_spring2022/reinstalled/SoftTeacher/thirdparty/mmdetection/mmdet/datasets/pipelines/formating.py:7: UserWarning: DeprecationWarning: mmdet.datasets.pipelines.formating will be deprecated, please replace it with mmdet.datasets.pipelines.formatting.
warnings.warn('DeprecationWarning: mmdet.datasets.pipelines.formating will be '
/vc_data/users/lashar/fhl_spring2022/reinstalled/SoftTeacher/thirdparty/mmdetection/mmdet/datasets/pipelines/formating.py:7: UserWarning: DeprecationWarning: mmdet.datasets.pipelines.formating will be deprecated, please replace it with mmdet.datasets.pipelines.formatting.
warnings.warn('DeprecationWarning: mmdet.datasets.pipelines.formating will be '
[ ] 2/200, 0.2 task/s, elapsed: 10s, ETA: 1013s/vc_data/users/lashar/fhl_spring2022/reinstalled/SoftTeacher/thirdparty/mmdetection/mmdet/datasets/pipelines/formating.py:7: UserWarning: DeprecationWarning: mmdet.datasets.pipelines.formating will be deprecated, please replace it with mmdet.datasets.pipelines.formatting.
warnings.warn('DeprecationWarning: mmdet.datasets.pipelines.formating will be '
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 200/200, 15.2 task/s, elapsed: 13s, ETA: 0s2022-03-24 18:25:14,215 - mmdet.ssod - INFO - Evaluating bbox...
2022-03-24 18:25:14,216 - mmdet.ssod - ERROR - The testing results of the whole dataset is empty.
Loading and preparing results...
2022-03-24 18:25:14,218 - mmdet.ssod - INFO - Exp name: soft_teacher_faster_rcnn_r50_caffe_fpn_coco_full_720k.py
2022-03-24 18:25:14,218 - mmdet.ssod - INFO - Iter(val) [12000]
2022-03-24 18:25:26,778 - mmdet.ssod - INFO - Iter [12050/720000] lr: 1.000e-02, eta: 2 days, 22:13:09, time: 0.809, data_time: 0.571, memory: 2890, ema_momentum: 0.9990, sup_loss_rpn_cls: 0.0000, sup_loss_rpn_bbox: 0.0000, sup_loss_cls: 0.0000, sup_acc: 100.0000, sup_loss_bbox: 0.0000, unsup_loss_rpn_cls: 0.0000, unsup_loss_rpn_bbox: 0.0000, unsup_loss_cls: 0.0000, unsup_acc: 100.0000, unsup_loss_bbox: 0.0000, loss: 0.0000
In the logs, I'm seeing this:
{"mode": "train", "epoch": 1, "iter": 12000, "lr": 0.01, "memory": 2890, "ema_momentum": 0.999, "data_time": 0.01367, "sup_loss_rpn_cls": 0.0, "sup_loss_rpn_bbox": 0.0, "sup_loss_cls": 0.0, "sup_acc": 100.0, "sup_loss_bbox": 0.0, "unsup_loss_rpn_cls": 0.0, "unsup_loss_rpn_bbox": 0.0, "unsup_loss_cls": 0.0, "unsup_acc": 100.0, "unsup_loss_bbox": 0.0, "loss": 0.0, "time": 0.36499}
{"mode": "val", "epoch": 1, "iter": 12000, "lr": 0.01}
{"mode": "train", "epoch": 1, "iter": 12050, "lr": 0.01, "memory": 2890, "ema_momentum": 0.999, "data_time": 0.57121, "sup_loss_rpn_cls": 0.0, "sup_loss_rpn_bbox": 0.0, "sup_loss_cls": 0.0, "sup_acc": 100.0, "sup_loss_bbox": 0.0, "unsup_loss_rpn_cls": 0.0, "unsup_loss_rpn_bbox": 0.0, "unsup_loss_cls": 0.0, "unsup_acc": 100.0, "unsup_loss_bbox": 0.0, "loss": 0.0, "time": 0.80853}
I'm not sure why it doesn't seem to evaluate. I've changed the config in /vc_data/users/lashar/fhl_spring2022/reinstalled/SoftTeacher/configs/soft_teacher/soft_teacher_faster_rcnn_r50_caffe_fpn_coco_full_720k.py to have val as a dict:
_base_="base.py"
data = dict(
samples_per_gpu=2,
workers_per_gpu=2,
train=dict(
sup=dict(
ann_file="data/coco/annotations/instances_train2017.json",
img_prefix="data/coco/train2017/",
),
unsup=dict(
ann_file="data/coco/annotations/instances_unlabeled2017.json",
img_prefix="data/coco/unlabeled2017/",
),
),
val=dict(
ann_file="data/coco/annotations/instances_val2017.json",
img_prefix="data/coco/val2017/",
),
sampler=dict(
train=dict(
sample_ratio=[1, 1],
)
),
)
semi_wrapper = dict(
train_cfg=dict(
unsup_weight=2.0,
)
)
lr_config = dict(step=[120000 * 4, 160000 * 4])
runner = dict(_delete_=True, type="IterBasedRunner", max_iters=180000 * 4)
I choose not to use wandb to visualize results, not sure if this causes any issues.
wandb: (1) Create a W&B account
wandb: (2) Use an existing W&B account
wandb: (3) Don't visualize my results
wandb: Enter your choice: 3
Appreicate any pointers, thanks!
One other detail: some of the images in my train and val set don't have the object of interest. For such images, there is no corresponding entry in "annotations" : but there is an entry in "images" ::
@sharmalakshay93 ensure that you have specified the workflow variable in your config and have included (val,1) in that
@mjehanzaib999 thanks for your response! I'm not sure I understand what you're referring to. Could you pls point me to the line in the config I should change? Did this solve it for you?
@sharmalakshay93 workflow = [('train', 1), ('val',1)] add this to your config if its not already there
@sharmalakshay93 btw what bash command are you running for soft teacher training?
Thanks, I've been running bash tools/dist_train.sh configs/soft_teacher/soft_teacher_faster_rcnn_r50_caffe_fpn_coco_full_720k.py 2