在沐曦C500上部署InternVL2.5-26B模型失败

[Itsanewday]

参考 https://lmdeploy.readthedocs.io/en/latest/get_started/maca/get_started.html 下载提供的lmdeploy镜像，并采用docker compose 部署. 文件如下

version: "3.8"

x-common: &common pull_policy: always # always, never, missing, build restart: unless-stopped stop_signal: SIGINT stop_grace_period: 1m logging: driver: "json-file" options: max-file: "10" max-size: "100m"

services: lmdeploy-internvl25-26B: image: localhost:5000/lmdeploy:maca container_name: lmdeploy-internvl25-26B shm_size: 100gb environment: - CUDA_VISIBLE_DEVICES=4,5 - GLOO_SOCKET_IFNAME=lo devices: - "/dev/dri:/dev/dri" - "/dev/mxcd:/dev/mxcd" - "/dev/infiniband:/dev/infiniband" group_add: - "video" volumes: - /tmp:/tmp - /mnt/data0/models:/models/ ports: - 20003:23333 entrypoint: ["/bin/bash", "-c"] command: [ "lmdeploy serve api_server --backend pytorch --device cuda --cache-block-seq-len 16 /models/OpenGVLab/InternVL2_5-26B --model-name internvl2 --tp 2 --cache-max-entry-count 0.9 " ] <<: *common

但部署失败，日志显示如下： torch.cuda.DeferredCudaCallError: CUDA call failed lazily at initialization with error: device >= 0 && device < num_gpus INTERNAL ASSERT FAILED at "/workspace/framework/mcPytorch/aten/src/ATen/cuda/CUDAContext.cpp":49, please report a bug to PyTorch. device=1, num_gpus=

tp=1时可以部署，不过推理时显存不足