bruicecode

Results 2 issues of bruicecode

LongLoRA implementation

3 comment

### PR types Others ### PR changes Others ### Description 1. 修改了llm/data.py 中处理red pajama中这种只有text的example方法 2. 在LlamaAttention forward使用scaled_dot_product_attention前对qkv进行shift操作，计算后对attn_output进行shift操作 3. 在llm/finetune_generation.py修改了rope scaling，保证模型能够处理2048以上长度的序列 4. 在llm/finetune_generation.py设置了norm embed层可训练

contributor
stale

Train Error

4 comment

(venv) personalinfo@MacBook-Pro-3 LongNet % python3 train.py 2024-03-05 23:56:10,524 - numexpr.utils - INFO - NumExpr defaulting to 8 threads. 2024-03-05 23:56:17.908409: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available...