DeepSeek-R1-Qwen-7B model setup error -> unknown pre-tokenizer type: 'deepseek-r1-qwen'

[scripnichenko]

Scenario

Trying to set-up the Node with DeepSeek: "chat": "https://huggingface.co/gaianet/DeepSeek-R1-Distill-Qwen-7B-GGUF/resolve/main/DeepSeek-R1-Distill-Qwen-7B-Q2_K.gguf",

Logs log/start-llamaedge.log

[2025-01-27 23:48:41.030] [info] [WASI-NN] GGML backend: LLAMA_COMMIT 924518e2
[2025-01-27 23:48:41.030] [info] [WASI-NN] GGML backend: LLAMA_BUILD_NUMBER 4466
...
[2025-01-27 23:48:41.135] [info] [WASI-NN] llama.cpp: print_info: file format = GGUF V3 (latest)
[2025-01-27 23:48:41.135] [info] [WASI-NN] llama.cpp: print_info: file type   = Q2_K - Medium
[2025-01-27 23:48:41.135] [info] [WASI-NN] llama.cpp: print_info: file size   = 2.80 GiB (3.16 BPW)
[2025-01-27 23:48:41.278] [error] [WASI-NN] llama.cpp: llama_model_load: error loading model:    
error loading model vocabulary: unknown pre-tokenizer type: 'deepseek-r1-qwen'
[2025-01-27 23:48:41.278] [error] [WASI-NN] llama.cpp: llama_model_load_from_file: failed to load model
[2025-01-27 23:48:41.315] [error] [WASI-NN] GGML backend: load: unable to init model.
common_init_from_params: failed to load model 'DeepSeek-R1-Distill-Qwen-7B-Q2_K.gguf'
[2025-01-27 23:48:41.315] [error] llama_core::graph in /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/llama-core-0.26.2/src/graph.rs:246: Backend Error: WASI-NN Backend Error: Caller module passed an invalid argument
[2025-01-27 23:48:41.315] [error] rag_api_server in src/main.rs:520: Failed to initialize the core context. Backend Error: WASI-NN Backend Error: Caller module passed an invalid argument
Error: Operation("Failed to initialize the core context. Backend Error: WASI-NN Backend Error: Caller module passed an invalid argument")

Remediation plan

The original full-precision model (likely FP16/BF16) was converted to a reduced precision format using the llama.cpp toolkit and expected LLAMA_BUILD_NUMBER >= b4519 when current version is 4466.

Contributing

@juntao @longzhi please let me know if you are open for contributions.

[alabulei1]

Hi @scripnichenko

Yes, you're correct. We need to upgrade the plugin to support DeepSeek-R1-Qwen-7B. Right now you can run DeepSeek-R1-Llama-7B. Stay tuned.

[scripnichenko]

@alabulei1 are you refering to gaianet/DeepSeek-R1-Distill-Llama-8B-GGUF ?

[alabulei1]

Yes, it is.