a-PyTorch-Tutorial-to-Object-Detection icon indicating copy to clipboard operation
a-PyTorch-Tutorial-to-Object-Detection copied to clipboard
verified publisher icon sgrvinod

when traing, the loss becomes nan. why?

Open diamond0910 opened this issue 6 years ago • 2 comments

image

diamond0910 avatar Jan 26 '20 08:01 diamond0910

Hey, Are you using the same dataset? If not then this is happening because the lr is high for your dataset. Reduce the lr. As soon as the loss reaches around thousands it starts showing nan or inf.

Go to train.py and reduce the learning rate(lr).

abhisingh977 avatar Apr 26 '20 23:04 abhisingh977

Upon some tracing I found out that MultiboxLoss returns that nan value, maybe this is the same issue as #54

tripathiarpan20 avatar Dec 30 '20 11:12 tripathiarpan20