DeepLoco icon indicating copy to clipboard operation
DeepLoco copied to clipboard

NaN Output When running on Ubuntu 16.04

Open xupei0610 opened this issue 7 years ago • 0 comments

We just complied it as the instruction on Ubuntu 16.04, and run it with argument -arg_file= args/train_llc_args.txt. A NaN output will be got only after several iterations, as shown following.

`Actor Iter 4 Update Net 0: I0808 12:46:07.273452 25515 solver.cpp:228] Iteration 10, loss = 0.0132303 I0808 12:46:07.273478 25515 solver.cpp:244] Train net output #0: loss = 0.0132303 (* 1 = 0.0132303 loss) I0808 12:46:07.273486 25515 sgd_solver.cpp:106] Iteration 10, lr = 0.01 I0808 12:46:07.299680 25515 solver.cpp:228] Iteration 4, loss = inf I0808 12:46:07.299707 25515 solver.cpp:244] Train net output #0: loss = inf (* 1 = inf loss) I0808 12:46:07.299715 25515 sgd_solver.cpp:106] Iteration 4, lr = 0.001

Actor Iter 5 Update Net 0: I0808 12:46:07.632088 25517 solver.cpp:228] Iteration 12, loss = 0.0219066 I0808 12:46:07.632109 25517 solver.cpp:244] Train net output #0: loss = 0.0219066 (* 1 = 0.0219066 loss) I0808 12:46:07.632114 25517 sgd_solver.cpp:106] Iteration 12, lr = 0.01 I0808 12:46:07.651747 25517 solver.cpp:228] Iteration 5, loss = -nan I0808 12:46:07.651769 25517 solver.cpp:244] Train net output #0: loss = -nan (* 1 = -nan loss) I0808 12:46:07.651777 25517 sgd_solver.cpp:106] Iteration 5, lr = 0.001`

We tried it on different Linux machines. The output is similar. But on Windows 10, no NaN will be got.

xupei0610 avatar Aug 08 '18 16:08 xupei0610