optimum-graphcore icon indicating copy to clipboard operation
optimum-graphcore copied to clipboard

Set LayerNorm's eps to a number that is larger than 6e-5

Open gejinchen opened this issue 3 years ago • 2 comments

Currently, many LayerNorm's eps are smaller than 6.1e-5 (smallest fp16 value), which might cause underflow.

gejinchen avatar Jun 29 '22 15:06 gejinchen

@michaelbenayoun this is something that could be done easily as a torch.fx transform?

jimypbr avatar Jul 06 '22 14:07 jimypbr

I would think this is something handled by pytorch, but to answer you @jimypbr, yes definitely!

michaelbenayoun avatar Jul 06 '22 15:07 michaelbenayoun