Deep-Feature-Flow icon indicating copy to clipboard operation
Deep-Feature-Flow copied to clipboard

too large launch parameter: AddTakeGrad[142000,1], [64,1,1]

Open xuwentang opened this issue 6 years ago • 0 comments

Who can help me?thanks a lot

Traceback (most recent call last): File "/home/xu/Project/Deep-Feature-Flow/dff_rfcn/train_end2end.py", line 186, in main() File "/home/xu/Project/Deep-Feature-Flow/dff_rfcn/train_end2end.py", line 183, in main config.TRAIN.begin_epoch, config.TRAIN.end_epoch, config.TRAIN.lr, config.TRAIN.lr_step) File "/home/xu/Project/Deep-Feature-Flow/dff_rfcn/train_end2end.py", line 175, in train_net arg_params=arg_params, aux_params=aux_params, begin_epoch=begin_epoch, num_epoch=end_epoch) File "/home/xu/Project/Deep-Feature-Flow/dff_rfcn/core/module.py", line 977, in fit self.update_metric(eval_metric, data_batch.label) File "/home/xu/Project/Deep-Feature-Flow/dff_rfcn/core/module.py", line 1069, in update_metric self._curr_module.update_metric(eval_metric, labels) File "/home/xu/Project/Deep-Feature-Flow/dff_rfcn/core/module.py", line 670, in update_metric self._exec_group.update_metric(eval_metric, labels) File "/home/xu/Project/Deep-Feature-Flow/dff_rfcn/core/DataParallelExecutorGroup.py", line 481, in update_metric eval_metric.update(labels, texec.outputs) File "/usr/local/lib/python2.7/dist-packages/mxnet/metric.py", line 318, in update metric.update(labels, preds) File "/home/xu/Project/Deep-Feature-Flow/dff_rfcn/core/metric.py", line 56, in update pred_label = mx.ndarray.argmax_channel(pred).asnumpy().astype('int32') File "/usr/local/lib/python2.7/dist-packages/mxnet/ndarray/ndarray.py", line 1972, in asnumpy ctypes.c_size_t(data.size))) File "/usr/local/lib/python2.7/dist-packages/mxnet/base.py", line 251, in check_call raise MXNetError(py_str(_LIB.MXGetLastError())) mxnet.base.MXNetError: [18:36:26] /home/travis/build/dmlc/mxnet-distro/mxnet-build/3rdparty/mshadow/mshadow/././././cuda/tensor_gpu-inl.cuh:58: too large launch parameter: AddTakeGrad[142000,1], [64,1,1]

Stack trace returned 10 entries: [bt] (0) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x37f51a) [0x7fce50ff851a] [bt] (1) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x37fb51) [0x7fce50ff8b51] [bt] (2) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x45f24eb) [0x7fce5526b4eb] [bt] (3) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x48433f4) [0x7fce554bc3f4] [bt] (4) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x484375d) [0x7fce554bc75d] [bt] (5) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x49b315d) [0x7fce5562c15d] [bt] (6) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x2b041a9) [0x7fce5377d1a9] [bt] (7) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x2b0aec7) [0x7fce53783ec7] [bt] (8) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x2ae90f4) [0x7fce537620f4] [bt] (9) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x2aed3c3) [0x7fce537663c3]

xuwentang avatar Mar 06 '19 10:03 xuwentang