Gamma, Beta in the model weight

[WANG-KX]

Why the value of gamma_s3, gamma_s2, beta_s3, beta_s2 are all zeros in your provided model weights? If they are all zeros, meaning that they are not functional, right?

[gallenszl]

Hello, gamma and beta are normalization factors, which is initialized as 0 and gradually learns a weight.

[WANG-KX]

I mean in your provided trained weights, they are still zeros.

[gallenszl]

This is a little strange and I will check it later.

[callmeray]

The gradients of gamma_s3, gamma_s2, beta_s3, beta_s2 are None. Is that expected?

[wangpinzhi]

The gradients of gamma_s3, gamma_s2, beta_s3, beta_s2 are None. Is that expected?

I think these parameters are not updated during training, and the parameters from the provided trained weights are all zeros.

[august779188]

I also found this problem ：the gradients of gamma_s3, gamma_s2, beta_s3, beta_s2 are None ，and they are not updated during training.it will lead to poor training results