AdaBound
AdaBound copied to clipboard
Merge with Ranger or over9000
https://github.com/lessw2020/Ranger-Deep-Learning-Optimizer
https://github.com/mgrankin/over9000/issues/4
I strongly believe that AdaBound would be better if it used RAdam instead of Adam. It could merge with Lookahead too and LAMB. Then we would have the best of both worlds and a beautiful example of scientific collaboration.