Loss gets stuck

[javierrodenas]

Hello,

I am experimenting the following issue:

It is getting stuck around 0.006-0.005. Does it happen to you? It doe not matter which kind of dataset I use, the behavior is the same.

Thank you in advance

[TontonTremblay]

I am not sure, what sort of GPU do you have? Which data are you using?

[javierrodenas]

I am using the 2080 RTX Ti (12 GB).

On the other hand, I am using this data currently (some examples):

My experience with this environment is that it does not matter which data I use, the loss is always around 0.005

[javierrodenas]

I was testing different parameters with same type of data:

Green: learning_rate 0.0001 and batch_size 16 Blue: learning_rate 0.001 and batch_size 8

TOTAL TRAIN LOSS

AFFINITY TRAIN LOSS

BELIEF MAPS TRAIN LOSS

The result of train loss is more of less the same and it gets stuck around 0.005 (TOTAL LOSS).

Any recommendation? Did you see this behavior before?

Thank you in advance!

[TontonTremblay]

It might be caused by the slightly symmetrical object. Do you have non symmetrical object to try on?

[javierrodenas]

I can try with non symetrical object.

Another possible issue here could be the color. I comment that because in another github issue I saw a guy who was playing with different color cubes #128. Could my color be an issue? It is white/light grey.

I removed the noise and the loss got to 0.003 (It decreased from 0.005 to 0.003) but it is not able to detect the piece during the predictions

[TontonTremblay]

can you share how the belief maps look like?