LongNet
LongNet copied to clipboard
kyegomez
Implementation of plug in and play Attention from "LongNet: Scaling Transformers to 1,000,000,000 Tokens"
Hi, I have installed LongNet in my Ubuntu with 4090. (Is it enough to run LongNet?) but when I type python example.py, there is error... I have tried "pip install...
:~/LongNet$ pip install -r requirements.txt Requirement already satisfied: torch in /home/straughterguthrie/robust/lib/python3.10/site-packages (from -r requirements.txt (line 1)) (2.0.1) Collecting einops Using cached einops-0.6.1-py3-none-any.whl (42 kB) Collecting flash_attn Using cached flash_attn-1.0.8.tar.gz (2.0...
(venv) personalinfo@MacBook-Pro-3 LongNet % python3 train.py 2024-03-05 23:56:10,524 - numexpr.utils - INFO - NumExpr defaulting to 8 threads. 2024-03-05 23:56:17.908409: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available...
## 当组件名为listItem编译到微信小程序无效 系统：windows10专业版 x64； node：v18.20.0; ## v4.08 listItem无效，如下图：  ## v3.99 listItem有效，如下图： 
I ran the example program and got the following error. ``` import torch from long_net.model import LongNetTransformer longnet = LongNetTransformer( num_tokens=20000, dim=512, depth=6, dim_head=64, heads=8, ff_mult=4, ).to("cuda:0") tokens = torch.randint(0,...
**Is your feature request related to a problem? Please describe.** Yes, the current implementation of the DilatedAttention and FlashAttention modules in the Zeta repository does not support multi-GPU configurations effectively,...
