goseesomething
goseesomething
The fix is here. https://github.com/Lightning-AI/lightning/issues/15912 
try edit learning rate params. It's because the dataset is difference and the params is not the best conditions, that's why the calculation between mask and pred coming out NAN...
> > Does anyone face the problem? > > 有人遇到這問題嗎? > > 是可以收敛的,你检查一下你数据集的问题,我换了backbone从头训练,大概600多epochs的时候收敛了 感謝回覆~ 後來找到原因了SGD是比較不穩定~的確有時候會不收斂 然後也感謝通知是600多epo左右~我下次再訓會留意這個數字! 然後之前訓沒有找到對的conda list 後來跟著下面凡Issue內容 重整conda list環境就訓得起來了: https://github.com/MhLiao/DB/issues/323 這邊的conda list