[BUG] python app.py 运行起来了，可是当推理时报错：Expected all tensors to be on the same device

[chenyujing1234]

python app.py 运行起来了，可是当推理时报错：Expected all tensors to be on the same device 在在这行报错的： outputs = model.inference(inputs, max_new_token.....

[shizhediao]

请问运行的命令是什么，可否把全部的错误信息展示一下。谢谢。

---

What is the command, and can you show all the error messages? Thanks.

[chenyujing1234]

我是在V100上测的。以下是报错的信息： Traceback (most recent call last): File "/home/chenyj/LMFlow/service/app.py", line 68, in predict outputs = model.inference(inputs, max_new_tokens=150,temperature=0.0, do_sample=False) File "/home/chenyj/LMFlow/src/lmflow/models/hf_decoder_model.py", line 458, in inference outputs = self.ds_engine.module.generate( File "/home/zhongm/anaconda3/envs/lmflow/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "/home/zhongm/anaconda3/envs/lmflow/lib/python3.9/site-packages/transformers/generation/utils.py", line 1437, in generate return self.greedy_search( File "/home/zhongm/anaconda3/envs/lmflow/lib/python3.9/site-packages/transformers/generation/utils.py", line 2248, in greedy_search outputs = self( File "/home/zhongm/anaconda3/envs/lmflow/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "/home/zhongm/anaconda3/envs/lmflow/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward output = old_forward(*args, **kwargs) File "/home/zhongm/anaconda3/envs/lmflow/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 687, in forward outputs = self.model( File "/home/zhongm/anaconda3/envs/lmflow/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "/home/zhongm/anaconda3/envs/lmflow/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 577, in forward layer_outputs = decoder_layer( File "/home/zhongm/anaconda3/envs/lmflow/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "/home/zhongm/anaconda3/envs/lmflow/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward output = old_forward(*args, **kwargs) File "/home/zhongm/anaconda3/envs/lmflow/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 289, in forward hidden_states = self.input_layernorm(hidden_states) File "/home/zhongm/anaconda3/envs/lmflow/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "/home/zhongm/anaconda3/envs/lmflow/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward output = old_forward(*args, **kwargs) File "/home/zhongm/anaconda3/envs/lmflow/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 91, in forward return self.weight * hidden_states RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1!

[shizhediao]

Could you add CUDA_VISIBLE_DEVICES=0 to run inference on a single GPU?

[yaoguany]

可以参考PR-174里tests/models/test_hf_decoder_model.py 104行到107行的改动把ram optimized load取消，是deepspeed/accelerate的问题

[chenyujing1234]

按这个方法可以了

[shizhediao]

ok, glad to hear that the issue has been solved.