pytorch-optimizer icon indicating copy to clipboard operation
pytorch-optimizer copied to clipboard
verified publisher icon jettify

Adan optimiser: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models

Open iiSeymour opened this issue 3 years ago • 1 comments

Looks promising https://arxiv.org/abs/2208.06677

iiSeymour avatar Aug 23 '22 16:08 iiSeymour

Hi, @iiSeymour

Here are the results for Adan. rastrigin_Adan rosenbrock_Adan

It seems that many optimizers can reach the optimal point, but the practical performance varies greatly, such as Adam and Adabound. We may prefer to focus on the practical results.

XingyuXie avatar Aug 31 '22 09:08 XingyuXie