WangXu

>alibi捕获相对位置，对角线元素应该一致吧，但是为什么baichuan的不一致？是有什么玄机吗？ 

Reduce the amount of gpu memory used in the quantification process

9

I used the example script in the readme to quantize llama3-8b ``` python quant_config = { "zero_point": True, "q_group_size": 16, "w_bit": 4, "version": "GEMM" } model = AutoAWQForCausalLM.from_pretrained(model_path, **{"low_cpu_mem_usage": True},...

An error occurred while deploying the 4bit version of Yi-34B-Chat

1

`AttributeError: 'LlamaTransformerLayerWeight' object has no attribute 'q_weight_'` The 4bit version seems to be only "att_norm_weight_" and "ffn_norm_weight". When will the 4bit version be supported?

使用多卡加载模型，推理时报错

24

您好， 我使用单块A800进行部署推理时正常，但是使用多卡推理会报错： `Task exception was never retrieved future: Traceback (most recent call last): File "/opt/conda/lib/python3.9/site-packages/rpyc/core/stream.py", line 268, in read buf = self.sock.recv(min(self.MAX_IO_CHUNK, count)) ConnectionResetError: [Errno 104] Connection reset by peer...

训练llama3-8b-it报错

2

我严格按照README安装了相关的包 pip install -r requirements.txt pip install git+https://github.com/unslothai/unsloth.git pip install bitsandbytes==0.43.1 pip install peft==0.10.0 pip install torch==2.2.2 pip install xformers==0.0.25.post1 启动参数 ```shell export CUDA_VISIBLE_DEVICES=2,3,4,5,6,7 torchrun --nproc_per_node=6 train.py \ --train_args_file train_args/sft/qlora/llama3-8b-sft-qlora.json...