FoilHao
FoilHao
I met the same problem. Have you solved it?
Same question here, have you solved it?
Is the 'ship' required adjust to customized class?
Hi, cao1124! I also meet endless loop problem. I still don't understand why this happened and how to solve it. Can you please help me out?
@cao1124 @leohsuofnthu Also I'm puzzled why there is no endless loop problem with author's pretrained weight.
> > @cao1124 @leohsuofnthu Also I'm puzzled why there is no endless loop problem with author's pretrained weight. > > Hi,FoilHao~You need to preprocess the image first ,make sure the...
[2020-04-23 23:57:48,790 INFO train.py line 65 60930] train_dice_coef: 0.0, S Loss: 259086.375, C Loss: 1.6436882019042969 [2020-04-23 23:57:50,109 INFO train.py line 228 60930] Iter -24-976: Loss: 259088.015625 acc: 0.000000% dice: 0.000000%...
Thank you! Your job do help me a lot and I've found the cause for some of my problems
Gradient overflow. Skipping step, loss scaler 0 reducing loss scale to 32768.0 repeats until it is 0 and get an error. Have anyone solved this?
Does your model train normally without fp16? I found my problem is that there are some nans in my dataset leading to nan gradient/gradient overflow. After correcting my data, it...