lmdeploy icon indicating copy to clipboard operation
lmdeploy copied to clipboard
Published 2 months ago • verified publisher icon InternLM

[Bug] Unable to use Ctrl+C to normally end service on the Ascend platform

Open jiajie-yang opened this issue 1 year ago • 2 comments

Checklist

  • [X] 1. I have searched related issues but cannot get the expected help.
  • [X] 2. The bug has not been fixed in the latest version.
  • [X] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.

Describe the bug

On the Ascend platform (such as ascend910b), after the lmdeploy serve api_server command is used to start the service, the service cannot be stopped gracefully by pressing Ctrl+C. Either an error is reported or the process is stuck.

Reproduction

ASCEND_RT_VISIBLE_DEVICES=3,4 lmdeploy serve api_server \
    models/InternVL2-8B \
    --tp 2 \
    --server-port 23333 \
    --device ascend \
    --cache-max-entry-count 0.8 \
    --log-level DEBUG

Environment

[W compiler_depend.ts:623] Warning: expandable_segments currently defaults to false. You can enable this feature by `export PYTORCH_NPU_ALLOC_CONF = expandable_segments:True`. (function operator())
sys.platform: linux
Python: 3.10.14 (main, May  6 2024, 19:36:58) [GCC 11.2.0]
CUDA available: False
MUSA available: False
numpy_random_seed: 2147483648
GCC: gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0
PyTorch: 2.1.0
PyTorch compiling details: PyTorch built with:
  - GCC 10.2
  - C++ Version: 201703
  - Intel(R) MKL-DNN v3.1.1 (Git Hash 64f6bcbcbab628e96f33a62c3e975f8535a7bde4)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: NO AVX
  - Build settings: BLAS_INFO=open, BUILD_TYPE=Release, CXX_COMPILER=/opt/rh/devtoolset-10/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-invalid-partial-specialization -Wno-unused-private-field -Wno-aligned-allocation-unavailable -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=open, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=2.1.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_EIGEN_FOR_BLAS=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=OFF, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, 

TorchVision: 0.16.0
LMDeploy: 0.6.1+2e49fc3
transformers: 4.44.1
gradio: Not Found
fastapi: 0.115.0
pydantic: 2.9.2
triton: Not Found

Error traceback

2024-10-09 02:35:34,096 - lmdeploy - DEBUG - utils.py:220 - <SamplingLogits> take time: 2.02 ms
2024-10-09 02:35:34,097 - lmdeploy - DEBUG - utils.py:220 - <UpdateRunning> take time: 0.05 ms
^CException in thread Thread-1:
Traceback (most recent call last):
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
Process ForkServerProcess-19:6:
Process ForkServerProcess-19:5:
    self.run()
Process ForkServerProcess-19:3:
Process ForkServerProcess-19:7:
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/utils/multiprocess_util.py", line 91, in run
Process ForkServerProcess-19:4:
Process ForkServerProcess-19:8:
Process ForkServerProcess-19:2:
    key, func, args, kwargs = self.task_q.get(timeout=TIMEOUT)
  File "<string>", line 2, in get
Process ForkServerProcess-19:9:
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
    kind, result = conn.recv()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
    buf = self._recv(4)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 383, in _recv
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
    func(*args, **kwargs)
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
    func(*args, **kwargs)
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
    func(*args, **kwargs)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
    key, func_name, detail = resource_proxy[TASK_QUEUE].get()
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
    key, func_name, detail = resource_proxy[TASK_QUEUE].get()
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
    key, func_name, detail = resource_proxy[TASK_QUEUE].get()
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
    func(*args, **kwargs)
  File "<string>", line 2, in get
  File "<string>", line 2, in get
  File "<string>", line 2, in get
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
    key, func_name, detail = resource_proxy[TASK_QUEUE].get()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
    kind, result = conn.recv()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
    kind, result = conn.recv()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
    kind, result = conn.recv()
  File "<string>", line 2, in get
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
    kind, result = conn.recv()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
Traceback (most recent call last):
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
KeyboardInterrupt
KeyboardInterrupt
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
KeyboardInterrupt
KeyboardInterrupt
    raise EOFError
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
Traceback (most recent call last):
Traceback (most recent call last):
EOFError
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
    func(*args, **kwargs)
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
    key, func_name, detail = resource_proxy[TASK_QUEUE].get()
  File "<string>", line 2, in get
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
    kind, result = conn.recv()
Traceback (most recent call last):
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
    func(*args, **kwargs)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
    func(*args, **kwargs)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
    key, func_name, detail = resource_proxy[TASK_QUEUE].get()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
    key, func_name, detail = resource_proxy[TASK_QUEUE].get()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
  File "<string>", line 2, in get
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
    func(*args, **kwargs)
  File "<string>", line 2, in get
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
    kind, result = conn.recv()
KeyboardInterrupt
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
    key, func_name, detail = resource_proxy[TASK_QUEUE].get()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
    kind, result = conn.recv()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "<string>", line 2, in get
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
    kind, result = conn.recv()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
KeyboardInterrupt
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
KeyboardInterrupt
KeyboardInterrupt
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
Process ForkServerProcess-3:
Process ForkServerProcess-8:
Process ForkServerProcess-4:
Process ForkServerProcess-7:
Process ForkServerProcess-9:
Process ForkServerProcess-5:
Process ForkServerProcess-6:
Process ForkServerProcess-2:
    self.run()
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/utils/multiprocess_util.py", line 91, in run
    key, func, args, kwargs = self.task_q.get(timeout=TIMEOUT)
  File "<string>", line 2, in get
Traceback (most recent call last):
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
Traceback (most recent call last):
Traceback (most recent call last):
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
Traceback (most recent call last):
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
    func(*args, **kwargs)
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
    func(*args, **kwargs)
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
    func(*args, **kwargs)
    kind, result = conn.recv()
Traceback (most recent call last):
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
    key, func_name, detail = resource_proxy[TASK_QUEUE].get()
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
    key, func_name, detail = resource_proxy[TASK_QUEUE].get()
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
    key, func_name, detail = resource_proxy[TASK_QUEUE].get()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 250, in recv
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "<string>", line 2, in get
  File "<string>", line 2, in get
  File "<string>", line 2, in get
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
    func(*args, **kwargs)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    buf = self._recv_bytes()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
    kind, result = conn.recv()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
    kind, result = conn.recv()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
    kind, result = conn.recv()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
    key, func_name, detail = resource_proxy[TASK_QUEUE].get()
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
    func(*args, **kwargs)
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
    func(*args, **kwargs)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
    func(*args, **kwargs)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
    func(*args, **kwargs)
  File "<string>", line 2, in get
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
    key, func_name, detail = resource_proxy[TASK_QUEUE].get()
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
    key, func_name, detail = resource_proxy[TASK_QUEUE].get()
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
    key, func_name, detail = resource_proxy[TASK_QUEUE].get()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
    buf = self._recv(4)
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
    key, func_name, detail = resource_proxy[TASK_QUEUE].get()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
    kind, result = conn.recv()
  File "<string>", line 2, in get
  File "<string>", line 2, in get
  File "<string>", line 2, in get
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
  File "<string>", line 2, in get
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 383, in _recv
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
    kind, result = conn.recv()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
    kind, result = conn.recv()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
    kind, result = conn.recv()
KeyboardInterrupt
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
    kind, result = conn.recv()
KeyboardInterrupt
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
KeyboardInterrupt
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt
    raise EOFError
EOFError
INFO:     Shutting down
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [1381088]
Traceback (most recent call last):
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/managers.py", line 810, in _callmethod
    conn = self._tls.connection
AttributeError: 'ForkAwareLocal' object has no attribute 'connection'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/util.py", line 300, in _run_finalizers
    finalizer()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/util.py", line 224, in __call__
    res = self._callback(*self._args, **self._kwargs)
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 53, in wrapper
    return func(cls, *args, **kwargs)
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 202, in finalize
    cls.cann_kb_mgr.finalize()
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/cann_kb_manager/knowledge_bank_manager.py", line 42, in finalize
    self.__static_mgr.close()
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/cann_kb_manager/base_manager.py", line 333, in close
    for name in copy.deepcopy(self.write_kb.keys()):
  File "<string>", line 2, in keys
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/managers.py", line 814, in _callmethod
    self._connect()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/managers.py", line 801, in _connect
    conn = self._Client(self._token.address, authkey=self._authkey)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 502, in Client
    c = SocketClient(address)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 630, in SocketClient
    s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
^C
^C^C^C^C

jiajie-yang avatar Oct 09 '24 03:10 jiajie-yang

I have to manually send a SIGINT signal to the lmdeploy process to gracefully end the service.

jiajie-yang avatar Oct 09 '24 03:10 jiajie-yang

@jiajie-yang Can you comment this line: https://github.com/InternLM/lmdeploy/blob/231e5bbcc5e1ea5f253002488587eacacd8f5e55/lmdeploy/pytorch/engine/model_agent.py#L630 And then try it again?

CyCle1024 avatar Oct 10 '24 11:10 CyCle1024

@jiajie-yang Can you comment this line:

https://github.com/InternLM/lmdeploy/blob/231e5bbcc5e1ea5f253002488587eacacd8f5e55/lmdeploy/pytorch/engine/model_agent.py#L630

And then try it again?

I tried it. It worked! Thanks a lot!

jiajie-yang avatar Oct 11 '24 09:10 jiajie-yang