benchmark
benchmark copied to clipboard
deeplabv3+ 有时出现异常结束的问题,导致run.sh 退出无法获取结果
训练日志如下:
step 75, loss: 2.736500, step_time_cost: 0.151 s
step 76, loss: 2.795518, step_time_cost: 0.150 s
step 77, loss: 2.817705, step_time_cost: 0.150 s
step 78, loss: 2.724798, step_time_cost: 0.149 s
step 79, loss: 2.779751, step_time_cost: 0.147 s
Training done. Model is saved to /home/crim/benchmark/deeplabv3+/paddle/output/model
*** Aborted at 1557585563 (unix time) try "date -d @1557585563" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGSEGV (@0x58) received by PID 4250 (TID 0x7f2212a06700) from PID 88; stack trace: ***
@ 0x7f22ea9e6390 (unknown)
@ 0x4bc644 PyEval_EvalFrameEx
@ 0x4b9b66 PyEval_EvalCodeEx
@ 0x4c17c6 PyEval_EvalFrameEx
@ 0x4b9b66 PyEval_EvalCodeEx
@ 0x4c17c6 PyEval_EvalFrameEx
@ 0x4b9b66 PyEval_EvalCodeEx
@ 0x4c17c6 PyEval_EvalFrameEx
@ 0x4b9b66 PyEval_EvalCodeEx
@ 0x4c17c6 PyEval_EvalFrameEx
@ 0x4d4e4d (unknown)
@ 0x4bca3c PyEval_EvalFrameEx
@ 0x4d4e4d (unknown)
@ 0x4bca3c PyEval_EvalFrameEx
@ 0x4b9b66 PyEval_EvalCodeEx
@ 0x4d57a3 (unknown)
@ 0x4a587e PyObject_Call
@ 0x4be51e PyEval_EvalFrameEx
@ 0x4c141f PyEval_EvalFrameEx
@ 0x4c141f PyEval_EvalFrameEx
@ 0x4b9b66 PyEval_EvalCodeEx
@ 0x4d5669 (unknown)
@ 0x4eef5e (unknown)
@ 0x4a587e PyObject_Call
@ 0x4c5ef0 PyEval_CallObjectWithKeywords
@ 0x589662 (unknown)
@ 0x7f22ea9dc6ba start_thread
@ 0x7f22ea71241d clone
@ 0x0 (unknown)
可以把set -xe改成 set -x,但是这个问题是否要解决一下?