Cream icon indicating copy to clipboard operation
Cream copied to clipboard

Maybe the potential bug in autoformer

Open crj1998 opened this issue 3 years ago • 0 comments

https://github.com/microsoft/Cream/blob/a857830192d472e6776e9af4bbd988f35ebf1f4d/AutoFormer/model/module/qkv_super.py#L72-L83

In the qkv_super the weight and bias sharing strategy is different. I think the selection of bias is unreasonable and should be modified in the following way.

 def sample_bias(bias, sample_out_dim): 
     sample_bias = torch.cat([sample_bias [i:sample_out_dim:3, :] for i in range(3)], dim =0) 
  
     return sample_bias 

crj1998 avatar Sep 01 '22 09:09 crj1998