I got this problem when I follow your restruction to run finetune_visualglm.py
who know how to solve this problem?
**[2023-06-16 17:34:52,083] [INFO] [RANK 0] do_test ...................... False
[2023-06-16 17:34:52,083] [INFO] [RANK 0] iteration .................... 0
Traceback (most recent call last):
File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-search/ganshu01/projects/VisualGLM-6B-main/ganshu_finetune.py", line 188, in
training_main(args, model_cls=model, forward_step_function=forward_step, create_dataset_function=create_dataset_function, collate_fn=data_collator)
File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-search/ganshu01/projects/VisualGLM-6B-main/sat/training/deepspeed_training.py", line 132, in training_main
iteration, skipped = train(model, optimizer,
File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-search/ganshu01/projects/VisualGLM-6B-main/sat/training/deepspeed_training.py", line 276, in train
lm_loss, skipped_iter, metrics = train_step(train_data_iterator,
File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-search/ganshu01/projects/VisualGLM-6B-main/sat/training/deepspeed_training.py", line 350, in train_step
forward_ret = forward_step(data_iterator, model, args, timers, **kwargs)
File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-search/ganshu01/projects/VisualGLM-6B-main/ganshu_finetune.py", line 84, in forward_step
logits = model(input_ids=tokens, image=image, pre_image=pre_image)[0]
File "/home/sankuai/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/sankuai/conda/lib/python3.8/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn
ret_val = func(*args, **kwargs)
File "/home/sankuai/conda/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 1735, in forward
loss = self.module(*inputs, **kwargs)
File "/home/sankuai/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-search/ganshu01/projects/VisualGLM-6B-main/sat/model/official/chatglm_model.py", line 192, in forward
return super().forward(input_ids=input_ids, attention_mask=attention_mask, position_ids=position_ids, past_key_values=past_key_values, **kwargs)
File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-search/ganshu01/projects/VisualGLM-6B-main/sat/model/base_model.py", line 144, in forward
return self.transformer(*args, **kwargs)
File "/home/sankuai/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-search/ganshu01/projects/VisualGLM-6B-main/sat/model/transformer.py", line 451, in forward
hidden_states = self.hooks['word_embedding_forward'](input_ids, output_cross_layer=output_cross_layer, **kw_args)
File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-search/ganshu01/projects/VisualGLM-6B-main/model/visualglm.py", line 20, in word_embedding_forward
image_emb = self.model(**kw_args)
File "/home/sankuai/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-search/ganshu01/projects/VisualGLM-6B-main/model/blip2.py", line 67, in forward
return self.glm_proj(out)
File "/home/sankuai/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, kwargs)
File "/home/sankuai/conda/lib/python3.8/site-packages/torch/nn/modules/linear.py", line 114, in forward
return F.linear(input, self.weight, self.bias)
RuntimeError: mat1 and mat2 shapes cannot be multiplied (1280x1024 and 768x4096)
someone says using half-precision instead of int8. Who know how to modify this in the code?
THUDM
