ValueError: zero-size array to reduction operation maximum which has no identity

[murphypei]

Hi all ,

I got a new issue during my training.

2018-08-23 18:12:20,224 - root - INFO - Epoch[0] Batch [1400]   Speed: 7.74 samples/sec RPNAcc=0.944368 RPNLogLoss=0.181685     RPNL1Loss=1.067499      RCNNAcc=0.800806        RCNNLogLoss=0.756302RCNNL1Loss=2.519708
Traceback (most recent call last):
  File "train.py", line 303, in <module>
    main()
  File "train.py", line 299, in main
    train_net(sym, roidb, args)
  File "train.py", line 120, in train_net
    arg_params=arg_params, aux_params=aux_params, begin_epoch=args.start_epoch, num_epoch=args.epochs)
  File "/home/peichao/.virtualenvs/py2/local/lib/python2.7/site-packages/mxnet/module/base_module.py", line 515, in fit
    next_data_batch = next(data_iter)
  File "/home/peichao/code/faster-rcnn/symdata/loader.py", line 183, in next
    data_batch = mx.io.DataBatch(data=self.getdata(), label=self.getlabel(),
  File "/home/peichao/code/faster-rcnn/symdata/loader.py", line 221, in getlabel
    b_label, b_bbox_target, b_bbox_weight = self._as.assign(anchors, b_gt_boxes, b_im_height, b_im_width)
  File "/home/peichao/code/faster-rcnn/symdata/anchor.py", line 138, in assign
    gt_max_overlaps = overlaps.max(axis=0)
  File "/home/peichao/.virtualenvs/py2/local/lib/python2.7/site-packages/numpy/core/_methods.py", line 26, in _amax
    return umr_maximum(a, axis, None, out, keepdims)
ValueError: zero-size array to reduction operation maximum which has no identity

It looks like that no overlap gt box found.

@ijkguo

[ijkguo]

I think it does no harm to filter out a training image that does not have any label (ground truth boxes).

[murphypei]

@ijkguo you mean I should filter those no label image before training? I don't know whether the PASCAL VOC 2007 contains such images?

[ijkguo]

Pascal voc does not have images with no labels. COCO has and they are filtered.

[murphypei]

@ijkguo But I meet this issue when I used pascal voc dataset

[ijkguo]

OK. When I did experiment with v6 before making a pull request to MXNet, there was no problem. But it is an easy fix anyway, https://github.com/ijkguo/mx-rcnn/commit/e556bb4e0209c2ce55386be8f701115538c3f010.

[murphypei]

@ijkguo Thank you, it's my fault, I reduced images size to 300x500 but didn't change the size of anchors, so there is no anchor after filter out anchors outside the region.