kylemin
Stability issues
Hi @kylemin !
I am working myself on an implementation of HEX graphs in Python using PyTorch and I'm having some stability issues : when the scores are very negative and very positive before the HEX layer, the computations diverge too much inside the HEX layer due to the passage to exponentials and I get some NaNs or approximations (some numbers can shrinked to 0 because they are too small to be taken into account given the precision of the system) which make predictions wrong.
Did you have similar numerical stability issues yourself ? And if yes, how did you deal with them ?
Best regards, Arthur
Hi Arthur,
I didn't have stability issues myself, but it totally makes sense that you have those issues. Over/under-flow of exp is a well-known issue so I think I once tried subtracting the maximum element as shown here: link. Maybe you could try this?
Thank you, Kyle
Hi @kylemin, thank you very much for the quick answer !
The problem is that I have strongly diverging score vectors (the vector you get out of the neural network), some are very positive and others very negative.
If tried subtracting the max like you proposed but then the negative ones get even more negative which results in underflow.
Do you have any idea of what the output scores of your neural network (before they get processed by the HEX layer) looked like after training ? That could help me a lot understand my problem ! (I know it has been a long time since you worked on this project so I understand if you don't remember anything of the sort)
Best regards, Arthur
Hi Arthur,
I am not sure how to address your problem and I don't recall how the output scores changed after training. I'll try to log in to my previous school server to see if the information is still there. Thank you!