InternLM
[Bug] lmdeploy部署的微调后的qwen2vl模型，能正常部署加载，但调用时候报错
Checklist
- [x] 1. I have searched related issues but cannot get the expected help.
- [x] 2. The bug has not been fixed in the latest version.
- [x] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
Describe the bug
ERROR: Exception in ASGI application Traceback (most recent call last): File "/gpfs01/gyh20_2/anaconda3/envs/lmdeploy-env/lib/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi result = await app( # type: ignore[func-returns-value] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/gpfs01/gyh20_2/anaconda3/envs/lmdeploy-env/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in call return await self.app(scope, receive, send) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/gpfs01/gyh20_2/anaconda3/envs/lmdeploy-env/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in call await super().call(scope, receive, send) File "/gpfs01/gyh20_2/anaconda3/envs/lmdeploy-env/lib/python3.12/site-packages/starlette/applications.py", line 112, in call await self.middleware_stack(scope, receive, send) File "/gpfs01/gyh20_2/anaconda3/envs/lmdeploy-env/lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in call raise exc File "/gpfs01/gyh20_2/anaconda3/envs/lmdeploy-env/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in call await self.app(scope, receive, _send) File "/gpfs01/gyh20_2/anaconda3/envs/lmdeploy-env/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in call await self.app(scope, receive, send) File "/gpfs01/gyh20_2/anaconda3/envs/lmdeploy-env/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in call await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) File "/gpfs01/gyh20_2/anaconda3/envs/lmdeploy-env/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app raise exc File "/gpfs01/gyh20_2/anaconda3/envs/lmdeploy-env/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app await app(scope, receive, sender) File "/gpfs01/gyh20_2/anaconda3/envs/lmdeploy-env/lib/python3.12/site-packages/starlette/routing.py", line 714, in call await self.middleware_stack(scope, receive, send) File "/gpfs01/gyh20_2/anaconda3/envs/lmdeploy-env/lib/python3.12/site-packages/starlette/routing.py", line 734, in app await route.handle(scope, receive, send) File "/gpfs01/gyh20_2/anaconda3/envs/lmdeploy-env/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle await self.app(scope, receive, send) File "/gpfs01/gyh20_2/anaconda3/envs/lmdeploy-env/lib/python3.12/site-packages/starlette/routing.py", line 76, in app await wrap_app_handling_exceptions(app, request)(scope, receive, send) File "/gpfs01/gyh20_2/anaconda3/envs/lmdeploy-env/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app raise exc File "/gpfs01/gyh20_2/anaconda3/envs/lmdeploy-env/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app await app(scope, receive, sender) File "/gpfs01/gyh20_2/anaconda3/envs/lmdeploy-env/lib/python3.12/site-packages/starlette/routing.py", line 73, in app response = await f(request) ^^^^^^^^^^^^^^^^ File "/gpfs01/gyh20_2/anaconda3/envs/lmdeploy-env/lib/python3.12/site-packages/fastapi/routing.py", line 301, in app raw_response = await run_endpoint_function( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/gpfs01/gyh20_2/anaconda3/envs/lmdeploy-env/lib/python3.12/site-packages/fastapi/routing.py", line 212, in run_endpoint_function return await dependant.call(**values) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/gpfs01/gyh20_2/anaconda3/envs/lmdeploy-env/lib/python3.12/site-packages/lmdeploy/serve/openai/api_server.py", line 451, in chat_completions_v1 async for res in result_generator: File "/gpfs01/gyh20_2/anaconda3/envs/lmdeploy-env/lib/python3.12/site-packages/lmdeploy/serve/async_engine.py", line 663, in generate prompt_input = await self._get_prompt_input(prompt, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/gpfs01/gyh20_2/anaconda3/envs/lmdeploy-env/lib/python3.12/site-packages/lmdeploy/serve/vl_async_engine.py", line 89, in _get_prompt_input results = await self.vl_encoder.wrap_for_pytorch(results, self.chat_template, self.tokenizer, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/gpfs01/gyh20_2/anaconda3/envs/lmdeploy-env/lib/python3.12/site-packages/lmdeploy/vl/engine.py", line 81, in wrap_for_pytorch result = self.model.to_pytorch(messages, chat_template, tokenizer, sequence_start) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/gpfs01/gyh20_2/anaconda3/envs/lmdeploy-env/lib/python3.12/site-packages/lmdeploy/vl/model/qwen2.py", line 108, in to_pytorch return self.to_pytorch_aux(messages, prompt, IMAGE_TOKEN, tokenizer, sequence_start) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/gpfs01/gyh20_2/anaconda3/envs/lmdeploy-env/lib/python3.12/site-packages/lmdeploy/vl/model/base.py", line 187, in to_pytorch_aux segs = prompt.split(IMAGE_TOKEN) ^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'split' INFO: 127.0.0.1:38726 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
Reproduction
Environment
cuda 12.4
lmdeploy 2.7.4.post1
qwenvl
Error traceback
感谢反馈，这边提到lmdeploy版本 lmdeploy 2.7.4.post1 是否有误？lmdeploy没有这个版本，能否提供 pip list信息方便复现定位问题
感谢反馈，这边提到lmdeploy版本
lmdeploy 2.7.4.post1是否有误？lmdeploy没有这个版本，能否提供 pip list信息方便复现定位问题
我说错了，是lmdeploy 2.7.2.post1，对应的Pip list如下：
Package Version
accelerate 1.5.2 addict 2.4.0 aiohappyeyeballs 2.6.1 aiohttp 3.11.14 aiosignal 1.3.2 annotated-types 0.7.0 anyio 4.9.0 attrs 25.3.0 autocommand 2.2.2 av 14.2.0 backports.tarfile 1.2.0 certifi 2025.1.31 charset-normalizer 3.4.1 click 8.1.8 cloudpickle 3.1.1 datasets 3.4.1 dill 0.3.8 diskcache 5.6.3 distro 1.9.0 einops 0.8.1 fastapi 0.115.11 filelock 3.18.0 fire 0.7.0 flash_attn 2.7.4.post1 flash_attn 2.7.4.post1 frozenlist 1.5.0 fsspec 2024.12.0 h11 0.14.0 httpcore 1.0.7 httpx 0.28.1 huggingface-hub 0.29.3 idna 3.10 importlib_metadata 8.0.0 inflect 7.3.1 interegular 0.3.3 jaraco.collections 5.1.0 jaraco.context 5.3.0 jaraco.functools 4.0.1 jaraco.text 3.12.1 Jinja2 3.1.6 jiter 0.9.0 jsonschema 4.23.0 jsonschema-specifications 2024.10.1 lark 1.2.2 llvmlite 0.44.0 lmdeploy 0.7.2.post1 markdown-it-py 3.0.0 MarkupSafe 3.0.2 mdurl 0.1.2 mmengine-lite 0.10.7 more-itertools 10.3.0 mpmath 1.3.0 msgpack 1.1.0 multidict 6.2.0 multiprocess 0.70.16 nest-asyncio 1.6.0 networkx 3.4.2 numba 0.61.0 numpy 1.26.4 nvidia-cublas-cu12 12.4.5.8 nvidia-cuda-cupti-cu12 12.4.127 nvidia-cuda-nvrtc-cu12 12.4.127 nvidia-cuda-runtime-cu12 12.4.127 nvidia-cudnn-cu12 9.1.0.70 nvidia-cufft-cu12 11.2.1.3 nvidia-curand-cu12 10.3.5.147 nvidia-cusolver-cu12 11.6.1.9 nvidia-cusparse-cu12 12.3.1.170 nvidia-ml-py 12.570.86 nvidia-nccl-cu12 2.21.5 nvidia-nvjitlink-cu12 12.4.127 nvidia-nvtx-cu12 12.4.127 openai 1.66.5 outlines 0.0.46 packaging 24.2 pandas 2.2.3 partial-json-parser 0.2.1.1.post5 peft 0.14.0 pillow 11.1.0 pip 25.0 platformdirs 4.3.6 propcache 0.3.0 protobuf 6.30.1 psutil 7.0.0 pyairports 2.1.1 pyarrow 19.0.1 pycountry 24.6.1 pydantic 2.10.6 pydantic_core 2.27.2 Pygments 2.19.1 pynvml 12.0.0 python-dateutil 2.9.0.post0 pytz 2025.1 PyYAML 6.0.2 qwen-vl-utils 0.0.10 ray 2.44.1 referencing 0.36.2 regex 2024.11.6 requests 2.32.3 rich 13.9.4 rpds-py 0.23.1 safetensors 0.5.3 sentencepiece 0.2.0 setuptools 76.1.0 shortuuid 1.0.13 six 1.17.0 sniffio 1.3.1 starlette 0.46.1 sympy 1.13.1 termcolor 2.5.0 tiktoken 0.9.0 timm 1.0.15 tokenizers 0.21.1 tomli 2.0.1 torch 2.5.1 torchvision 0.20.1 tqdm 4.67.1 transformers 4.49.0 triton 3.1.0 typeguard 4.3.0 typing_extensions 4.12.2 tzdata 2025.1 urllib3 2.3.0 uvicorn 0.34.0 wheel 0.45.1 xxhash 3.5.0 yapf 0.43.0 yarl 1.18.3 zipp 3.19.2
@cuikaiGitHub
我这边用 lmdeploy v0.7.2.post1 版本尝试了一下，结果如下图，没有复现报错。从您反馈的错误信息来看，很可能与输入的 prompt 以及图片相关，如果方便的话能否提供测试用的 prompt 以及图片信息或者描述相关信息？如果不方便提供，请再检查一下确保输入时图片不为空
prompt_reg_lic = f'''请逐步分析车辆登记证图片，按模块准确提取车辆相关信息。 ### 一. 首先提取“注册登记摘要信息栏”中的信息： 提取以下字段： - 1. 机动车所有人/身份证明名称/号码 - 2. 登记机关 - 3. 登记日期 - 4. 机动车登记编号 ### 二. 其次提取“注册登记机动车信息栏”中的信息： 提取以下字段： - 5. 车辆类型 - 6. 车辆品牌 - 7. 车辆型号 - 8. 车身颜色 - 9. 车辆识别代号/车架号（注意！遵循车辆识别代号！仔细核对每一位数字和字母，每一位数字和字母都很重要！！！） - 10. 国产/进口 - 11. 发动机号 - 12. 发动机型号 - 13. 燃料种类 - 14. 排量/功率 - 15. 制造厂名称 - 16. 转向形式 - 17. 轮距 - 18. 轮胎数 - 19. 轮胎规格 - 20. 钢板弹簧片数 - 21. 轴距 - 22. 轴数 - 23. 外廓尺寸 - 24. 货厢内部尺寸 - 25. 总质量 - 26. 核定载质量 - 27. 核定载客 - 28. 准牵引总质量 - 29. 驾驶室载客 - 30. 使用性质 - 31. 车辆获得方式 - 32. 车辆出厂日期 - 33. 发证机关章 - 34. 发证日期
### 三. 提取“转移登记摘要信息栏”中的信息
第一步判断字段是否有值，如果值为空，则填写"无值"。
第二步提取值不为空的字段：
- 机动车所有人/身份证明名称/号码
- 登记机关
- 登记日期
- 机动车登记编号
输出格式为：字段：值\n
注意：严格按照图片内容，不要编造、修改图片内容。
'''
prompt_vin=''' 分析车辆登记证图片，精确提取以下字段, 车辆识别代号/车架号 (此字段极其重要，请仔细核对)
输出格式为：字段：值\n
'''
