lcykww

数据集格式

1

`RANK=8 deepspeed --num_gpus=8 --num_nodes=2 train.py \ --base_model --micro_batch_size 4\ --wandb_run_name mora_math_r8 --lora_target_modules q_proj,k_proj,v_proj,o_proj,gate_proj,down_proj,up_proj \ --num_epochs 3 --deepspeed ds.config --wandb_project lora-math --lora_r $RANK --batch_size 128 \ --data_path meta-math/MetaMath \ --save_steps 3000...

RuntimeError: The size of tensor a (4096) must match the size of tensor b (256) at non-singleton dimension 1

2

基础模型：llama-7b-hf 我尝试将微调后的参数合并，但出现了错误 `Traceback (most recent call last): File "/home/kww/test_model.py", line 24, in merged_model = model.merge_and_unload(safe_merge=True) File "/home/kww/MoRA/peft-mora/src/peft/tuners/lora/model.py", line 721, in merge_and_unload return self._unload_and_optionally_merge( File "/home/kww/MoRA/peft-mora/src/peft/tuners/lora/model.py", line 375, in _unload_and_optionally_merge target.merge(safe_merge=safe_merge,...