MhLiao
Training is very slow
Hi, my training speed is very slow. In most cases, 100 iterations will take nearly two hours.I used a single GPU（NVIDIA 1080）, CUDA8.0, cudn 6.0。Tell me, what can I do to improve my training speed。 The image size is 1280x720（1600 images）
You can check the utilization of your GPU. There is a bottleneck in the data loader when the original images are large. But 1280x720 should be OK.
The original SSD repo also suffers such problems. It seems that the training speed is sensitive to GPU and drivers. See https://github.com/weiliu89/caffe/issues/691 I guess the low gpu-util is caused by the data layer. You can open the debug mode and see the time consuming of each layer.