pytorch-optimizer icon indicating copy to clipboard operation
pytorch-optimizer copied to clipboard

Implementation of Shampoo inconsistent with the paper

Open siddharth9820 opened this issue 2 years ago • 0 comments

https://github.com/jettify/pytorch-optimizer/blob/910b414565427f0a66e20040475e7e4385e066a5/torch_optimizer/shampoo.py#L130 Shouldn't the second argument be -0.5/order? For example, with order 2, the authors raise the precondition matrices to the -1/4th power.

siddharth9820 avatar May 29 '23 12:05 siddharth9820