nan values in training

Open cxym-yyn opened this issue 3 years ago • 0 comments

Hello, thodan. First of all, thanks for your sharing. I'm trying to training model with 'python train.py --model='ycbv_custom'', but i got nan values in losses. Like: step: 0 total_loss: 9.91135 obj_cls: 3.09097505 frag_cls: 4.15891361 frag_loc: 2.40971112 step: 100 total_loss: nan obj_cls: nan frag_cls: nan frag_loc: nan INFO:tensorflow:global_step/sec: 1.57567 step: 200 total_loss: nan obj_cls: nan frag_cls: nan frag_loc: nan INFO:tensorflow:global_step/sec: 1.66842

And i check the input data with visualization, it works. I trying to fix for days, But I don't where the error is... Can you give me some advice for that?

Jul 04 '22 05:07 cxym-yyn