optimum-graphcore icon indicating copy to clipboard operation
optimum-graphcore copied to clipboard

Allow setting max-weight-norm optimizer parameter

Open jimypbr opened this issue 3 years ago • 0 comments

[From AlexC in GC] The reference implementation uses a value of 10 for this parameter. However the implementation in GC-Optimum passes None, with no possibility for the user to set this parameter

  • create training command line option --max-weight-norm it can default to None for backwards compatibility
  • in IPUTrainer.create_optimizer pass this argument to LAMB optimizer as part of optimizer_kwargs

jimypbr avatar Jun 22 '22 14:06 jimypbr