LWM pytorch model & ring attention

Thanks for sharing this excellent great work. We want to use pytorch models to try the effect of ring attention. Are there any plans to develop ring attention implementation under pytorch?

Mar 05 '24 09:03 LzhinFdu

What do you have in mind? Is this model suitable for tokenized ecosystem and bridging liquidity and creating a smart algorithm for bridging / blending / mending and growth hacking liquidity across and between multiple TOKENS

Mar 05 '24 09:03 apexspyche

Lucidrains has a pytorch implementation of RingAttention https://github.com/lucidrains/ring-attention-pytorch

Mar 05 '24 10:03 kabachuha

Lucidrains has a pytorch implementation of RingAttention https://github.com/lucidrains/ring-attention-pytorch

Have you tried this repo? I don’t know whether the experimental results are as expected. Seems that the model posted on huggingface cannot use it directly to call ring attention.

Mar 05 '24 14:03 LzhinFdu

What do you have in mind? Is this model suitable for tokenized ecosystem and bridging liquidity and creating a smart algorithm for bridging / blending / mending and growth hacking liquidity across and between multiple TOKENS

I just want to call ring attention when using the trained pytorch LLM for inference.

Mar 05 '24 14:03 LzhinFdu