Add UC Berkleys Large World Models

[Venkman42]

Can someone add support for the Large World Models(Text and Multimodal) from UC Berkley? https://largeworldmodel.github.io/

[Engininja2]

The text models seem to just be llama so you can use convert.py for those

[dtiarks]

The interesting part is probably the implementation of ring attention

[Venkman42]

The interesting part is probably the implementation of ring attention

I haven't totally understood it, but would it help with memory consumption on large contexts?

[dtiarks]

As far as I understood it ring attention is an efficient way to compute attention in many GPUs

[github-actions[bot]]

This issue was closed because it has been inactive for 14 days since being marked as stale.