bert icon indicating copy to clipboard operation
bert copied to clipboard
verified publisher icon google-research

why add exclude_from_weight_decay for norm-related weight?

Open Sanqiang opened this issue 7 years ago • 2 comments

https://github.com/google-research/bert/blob/f39e881b169b9d53bea03d2d341b31707a6c052b/optimization.py#L65

Is there any special reason we add exclude_from_weight_decay for norm-related weight?

Sanqiang avatar Dec 20 '18 02:12 Sanqiang

I have the same question, can someone explain it?

neoql avatar May 08 '20 16:05 neoql

also wonder about this

zheyuye avatar Sep 28 '21 04:09 zheyuye