Kite0011

Results 3 comments of Kite0011

Parameter fusion in optimizer partition makes lamb behaves differently

> > > @szhengac You are correct, LAMB and LARS implementations that are not aware of ZeRO will not work correctly with ZeRO. This is not a fundamental limitation of...

Parameter fusion in optimizer partition makes lamb behaves differently

> > > > @szhengac You are correct, LAMB and LARS implementations that are not aware of ZeRO will not work correctly with ZeRO. This is not a fundamental limitation...

Cross-Attention?

Hi @lucidrains! Would you mean i can just imply GAU on cross-attention model such as t5? I foud GAU works very well on bert model