Cream
Cream copied to clipboard
Maybe the potential bug in autoformer
https://github.com/microsoft/Cream/blob/a857830192d472e6776e9af4bbd988f35ebf1f4d/AutoFormer/model/module/qkv_super.py#L72-L83
In the qkv_super the weight and bias sharing strategy is different. I think the selection of bias is unreasonable and should be modified in the following way.
def sample_bias(bias, sample_out_dim):
sample_bias = torch.cat([sample_bias [i:sample_out_dim:3, :] for i in range(3)], dim =0)
return sample_bias