yhq

> HI @zwd003 This error occurred during the deployment process. How to solve it? Thanks! > > (RayWorkerWrapper pid=52311) ERROR 05-11 18:04:33 worker_base.py:145] File "/opt/vllm/vllm/model_executor/models/deepseek_v2.py", line 156, in forward (RayWorkerWrapper...

> HI @zwd003 This error occurred during the deployment process. How to solve it? Thanks! > > (RayWorkerWrapper pid=52311) ERROR 05-11 18:04:33 worker_base.py:145] File "/opt/vllm/vllm/model_executor/models/deepseek_v2.py", line 156, in forward (RayWorkerWrapper...

I run `python -m flexgen.flex_opt --gpu-batch-size 1 --overlap false --model facebook/opt-6.7b --path _DUMMY_ --prompt-len 512 --gen-len 512` and `python flex_opt.py --gpu-batch-size 1 --overlap false --hh-ratio 0.2 --hh-all --model facebook/opt-6.7b --path...

I'm running starcoder on 2*A10, The command is as follows: python -m vllm.entrypoints.api_server --model /model/starchat/starcoder-codewovb-wlmhead-mg2hf41 --tensor-parallel-size 2 --gpu-memory-utilization 0.90 --host 0.0.0.0 --port 8081 --max-num-batched-tokens 5120