llama.cpp
llama.cpp copied to clipboard
Published
2 months ago
•
ggml-org
ggml-org
Misc. bug: Native API failed. Native API returns: 20 (UR_RESULT_ERROR_DEVICE_LOST)
Name and Version
llama-server --version version: 4688 (a18f481f) built with MSVC 19.42.34436.0 for
Operating systems
Windows
Which llama.cpp modules do you know to be affected?
llama-server
Command line
llama-server.exe -m G:\OllamaModels\blobs\sha256-6e9f90f02bb3b39b59e81916e8cfce9deb45aeaeb9a54a5be4414486b907dc1e -np 4 -ngl 40 --port 3000
Problem description & steps to reproduce
I think the bug can be 100% repeated, when running the llama-server for a while, sometimes, it will be broken due to "The GGUF Native API failed. Native API returns: 20 (UR_RESULT_ERROR_DEVICE_LOST)" The issue can also be reproduced with llama-cli.exe
UR_RESULT_ERROR_DEVICE_LOST seems to say that GPU device didn't to respond for a timeout, the symptom looks like a 3D Game that have often encountered, I tried to increased TdrDelay to 3c(hex) in the registry key as following:
\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\GraphicsDrivers
then restarted PC to apply the change, but the issue persists,
Would you have some fix on the specific issue? such as trying to restart something with llama-cpp / server itself?
By the way, I don't think the issue would be related to specific model files or configs, you can try it out .
First Bad Commit
No response
Relevant log output
build: 4688 (a18f481f) with MSVC 19.42.34436.0 for
system info: n_threads = 16, n_threads_batch = 16, total_threads = 24
system_info: n_threads = 16 (n_threads_batch = 16) / 24 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | LLAMAFILE = 1 | OPENMP = 1 | AARCH64_REPACK = 1 |
main: HTTP server is listening, hostname: 127.0.0.1, port: 3000, http threads: 23
main: loading model
get_memory_info: [warning] ext_intel_free_memory is not supported (export/set ZES_ENABLE_SYSMAN=1 to support), use total memory as free memory
srv load_model: loading model 'G:\OllamaModels\blobs\sha256-6e9f90f02bb3b39b59e81916e8cfce9deb45aeaeb9a54a5be4414486b907dc1e'
llama_model_load_from_file_impl: using device SYCL0 (Intel(R) UHD Graphics 770) - 14797 MiB free
llama_model_loader: loaded meta data with 26 key-value pairs and 579 tensors from G:\OllamaModels\blobs\sha256-6e9f90f02bb3b39b59e81916e8cfce9deb45aeaeb9a54a5be4414486b907dc1e (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = qwen2
llama_model_loader: - kv 1: general.type str = model
llama_model_loader: - kv 2: general.name str = DeepSeek R1 Distill Qwen 14B
llama_model_loader: - kv 3: general.basename str = DeepSeek-R1-Distill-Qwen
llama_model_loader: - kv 4: general.size_label str = 14B
llama_model_loader: - kv 5: qwen2.block_count u32 = 48
llama_model_loader: - kv 6: qwen2.context_length u32 = 131072
llama_model_loader: - kv 7: qwen2.embedding_length u32 = 5120
llama_model_loader: - kv 8: qwen2.feed_forward_length u32 = 13824
llama_model_loader: - kv 9: qwen2.attention.head_count u32 = 40
llama_model_loader: - kv 10: qwen2.attention.head_count_kv u32 = 8
llama_model_loader: - kv 11: qwen2.rope.freq_base f32 = 1000000.000000
llama_model_loader: - kv 12: qwen2.attention.layer_norm_rms_epsilon f32 = 0.000010
llama_model_loader: - kv 13: general.file_type u32 = 15
llama_model_loader: - kv 14: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 15: tokenizer.ggml.pre str = qwen2
llama_model_loader: - kv 16: tokenizer.ggml.tokens arr[str,152064] = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv 17: tokenizer.ggml.token_type arr[i32,152064] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv 18: tokenizer.ggml.merges arr[str,151387] = ["臓 臓", "臓臓 臓臓", "i n", "臓 t",...
llama_model_loader: - kv 19: tokenizer.ggml.bos_token_id u32 = 151646
llama_model_loader: - kv 20: tokenizer.ggml.eos_token_id u32 = 151643
llama_model_loader: - kv 21: tokenizer.ggml.padding_token_id u32 = 151643
llama_model_loader: - kv 22: tokenizer.ggml.add_bos_token bool = true
llama_model_loader: - kv 23: tokenizer.ggml.add_eos_token bool = false
llama_model_loader: - kv 24: tokenizer.chat_template str = {% if not add_generation_prompt is de...
llama_model_loader: - kv 25: general.quantization_version u32 = 2
llama_model_loader: - type f32: 241 tensors
llama_model_loader: - type q4_K: 289 tensors
llama_model_loader: - type q6_K: 49 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type = Q4_K - Medium
print_info: file size = 8.37 GiB (4.87 BPW)
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
load: special tokens cache size = 22
load: token to piece cache size = 0.9310 MB
print_info: arch = qwen2
print_info: vocab_only = 0
print_info: n_ctx_train = 131072
print_info: n_embd = 5120
print_info: n_layer = 48
print_info: n_head = 40
print_info: n_head_kv = 8
print_info: n_rot = 128
print_info: n_swa = 0
print_info: n_embd_head_k = 128
print_info: n_embd_head_v = 128
print_info: n_gqa = 5
print_info: n_embd_k_gqa = 1024
print_info: n_embd_v_gqa = 1024
print_info: f_norm_eps = 0.0e+00
print_info: f_norm_rms_eps = 1.0e-05
print_info: f_clamp_kqv = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale = 0.0e+00
print_info: n_ff = 13824
print_info: n_expert = 0
print_info: n_expert_used = 0
print_info: causal attn = 1
print_info: pooling type = 0
print_info: rope type = 2
print_info: rope scaling = linear
print_info: freq_base_train = 1000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn = 131072
print_info: rope_finetuned = unknown
print_info: ssm_d_conv = 0
print_info: ssm_d_inner = 0
print_info: ssm_d_state = 0
print_info: ssm_dt_rank = 0
print_info: ssm_dt_b_c_rms = 0
print_info: model type = 14B
print_info: model params = 14.77 B
print_info: general.name = DeepSeek R1 Distill Qwen 14B
print_info: vocab type = BPE
print_info: n_vocab = 152064
print_info: n_merges = 151387
print_info: BOS token = 151646 '<锝渂egin鈻乷f鈻乻entence锝?'
print_info: EOS token = 151643 '<锝渆nd鈻乷f鈻乻entence锝?'
print_info: EOT token = 151643 '<锝渆nd鈻乷f鈻乻entence锝?'
print_info: PAD token = 151643 '<锝渆nd鈻乷f鈻乻entence锝?'
print_info: LF token = 198 '膴'
print_info: FIM PRE token = 151659 '<|fim_prefix|>'
print_info: FIM SUF token = 151661 '<|fim_suffix|>'
print_info: FIM MID token = 151660 '<|fim_middle|>'
print_info: FIM PAD token = 151662 '<|fim_pad|>'
print_info: FIM REP token = 151663 '<|repo_name|>'
print_info: FIM SEP token = 151664 '<|file_sep|>'
print_info: EOG token = 151643 '<锝渆nd鈻乷f鈻乻entence锝?'
print_info: EOG token = 151662 '<|fim_pad|>'
print_info: EOG token = 151663 '<|repo_name|>'
print_info: EOG token = 151664 '<|file_sep|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = true)
get_memory_info: [warning] ext_intel_free_memory is not supported (export/set ZES_ENABLE_SYSMAN=1 to support), use total memory as free memory
get_memory_info: [warning] ext_intel_free_memory is not supported (export/set ZES_ENABLE_SYSMAN=1 to support), use total memory as free memory
load_tensors: offloading 40 repeating layers to GPU
load_tensors: offloaded 40/49 layers to GPU
load_tensors: SYCL0 model buffer size = 6245.35 MiB
load_tensors: CPU_Mapped model buffer size = 2320.69 MiB
..........................................................................................
llama_init_from_model: n_seq_max = 4
llama_init_from_model: n_ctx = 4096
llama_init_from_model: n_ctx_per_seq = 1024
llama_init_from_model: n_batch = 2048
llama_init_from_model: n_ubatch = 512
llama_init_from_model: flash_attn = 0
llama_init_from_model: freq_base = 1000000.0
llama_init_from_model: freq_scale = 1
llama_init_from_model: n_ctx_per_seq (1024) < n_ctx_train (131072) -- the full capacity of the model will not be utilized
GGML_SYCL_DEBUG: 0
GGML_SYCL_FORCE_MMQ: no
GGML_SYCL_F16: no
Found 1 SYCL devices:
| | | | |Max | |Max |Global | |
| | | | |compute|Max work|sub |mem | |
|ID| Device Type| Name|Version|units |group |group|size | Driver version|
|--|-------------------|---------------------------------------|-------|-------|--------|-----|-------|---------------------|
| 0| [level_zero:gpu:0]| Intel UHD Graphics 770| 12.2| 32| 512| 32| 15515M| 1.6.31441|
llama_kv_cache_init: kv_size = 4096, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 48, can_shift = 1
llama_kv_cache_init: SYCL0 KV buffer size = 640.00 MiB
llama_kv_cache_init: CPU KV buffer size = 128.00 MiB
llama_init_from_model: KV self size = 768.00 MiB, K (f16): 384.00 MiB, V (f16): 384.00 MiB
llama_init_from_model: CPU output buffer size = 2.32 MiB
llama_init_from_model: SYCL0 compute buffer size = 926.08 MiB
llama_init_from_model: SYCL_Host compute buffer size = 18.01 MiB
llama_init_from_model: graph nodes = 1686
llama_init_from_model: graph splits = 116 (with bs=512), 3 (with bs=1)
common_init_from_params: setting dry_penalty_last_n to ctx_size = 4096
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
srv init: initializing slots, n_slots = 4
slot init: id 0 | task -1 | new slot n_ctx_slot = 1024
slot init: id 1 | task -1 | new slot n_ctx_slot = 1024
slot init: id 2 | task -1 | new slot n_ctx_slot = 1024
slot init: id 3 | task -1 | new slot n_ctx_slot = 1024
main: model loaded
main: chat template, chat_template: {% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% set ns = namespace(is_first=false, is_tool=false, is_output_first=true, system_prompt='') %}{%- for message in messages %}{%- if message['role'] == 'system' %}{% set ns.system_prompt = message['content'] %}{%- endif %}{%- endfor %}{{bos_token}}{{ns.system_prompt}}{%- for message in messages %}{%- if message['role'] == 'user' %}{%- set ns.is_tool = false -%}{{'<锝淯ser锝?' + message['content']}}{%- endif %}{%- if message['role'] == 'assistant' and message['content'] is none %}{%- set ns.is_tool = false -%}{%- for tool in message['tool_calls']%}{%- if not ns.is_first %}{{'<锝淎ssistant锝?<锝渢ool鈻乧alls鈻乥egin锝?<锝渢ool鈻乧all鈻乥egin锝?' + tool['type'] + '<锝渢ool鈻乻ep锝?' + tool['function']['name'] + '\n' + '' + '\n' + tool['function']['arguments'] + '\n' + '' + '<锝渢ool鈻乧all鈻乪nd锝?'}}{%- set ns.is_first = true -%}{%- else %}{{'\n' + '<锝渢ool鈻乧all鈻乥egin锝?' + tool['type'] + '<锝渢ool鈻乻ep锝?' + tool['function']['name'] + '\n' + '' + '\n' + tool['function']['arguments'] + '\n' + '' + '<锝渢ool鈻乧all鈻乪nd锝?'}}{{'< 锝渢ool鈻乧alls鈻乪nd锝?<锝渆nd鈻乷f鈻乻entence锝?'}}{%- endif %}{%- endfor %}{%- endif %}{%- if message['role'] == 'assistant' and message['content'] is not none %}{%- if ns.is_tool %}{{'<锝渢ool鈻乷utputs鈻乪nd锝?' + message['content'] + '<锝渆nd鈻乷f鈻乻entence锝?'}}{%- set ns.is_tool = false -%}{%- else %}{% set content = message['content'] %}{% if '</think>' in content %}{% set content = content.split('</think>')[-1] %}{% endif %}{{'<锝淎ssistant锝?' + content + '<锝渆nd鈻乷f鈻乻entence锝?'}}{%- endif %}{%- endif %}{%- if message['role'] == 'tool' %}{%- set ns.is_tool = true -%}{%- if ns.is_output_first %}{{'<锝渢ool鈻乷utputs鈻乥egin锝?<锝渢ool鈻乷utput鈻乥egin锝?' + message['content'] + '<锝渢ool鈻乷utput鈻乪nd锝?'}}{%- set ns.is_output_first = false %}{%- else %}{{'\n<锝渢ool鈻乷utput鈻乥egin锝?' + message['content'] + '<锝渢ool鈻乷utput鈻乪nd锝?'}}{%- endif %}{%- endif %}{%- endfor -%}{% if ns.is_tool %}{{'<锝渢ool鈻乷utputs鈻乪nd锝?'}}{% endif %}{% if add_generation_prompt and not ns.is_tool %}{{'<锝淎ssistant锝?'}}{% endif %}, example_format: 'You are a helpful assistant
<锝淯ser锝?Hello<锝淎ssistant锝?Hi there<锝渆nd鈻乷f鈻乻entence锝?<锝淯ser锝?How are you?<锝淎ssistant锝?'
main: server is listening on http://127.0.0.1:3000 - starting the main loop
srv update_slots: all slots are idle
slot launch_slot_: id 0 | task 0 | processing task
slot update_slots: id 0 | task 0 | new prompt, n_ctx_slot = 1024, n_keep = 0, n_prompt_tokens = 198
slot update_slots: id 0 | task 0 | kv cache rm [0, end)
slot update_slots: id 0 | task 0 | prompt processing progress, n_past = 198, n_tokens = 198, progress = 1.000000
slot update_slots: id 0 | task 0 | prompt done, n_past = 198, n_tokens = 198
Native API failed. Native API returns: 20 (UR_RESULT_ERROR_DEVICE_LOST)
Exception caught at file:D:\a\llama.cpp\llama.cpp\ggml\src\ggml-sycl\ggml-sycl.cpp, line:327, func:operator()
SYCL error: CHECK_TRY_ERROR((*stream).memcpy((char *)tensor->data + offset, host_buf, size) .wait()): Meet error in this line code!
in function ggml_backend_sycl_buffer_set_tensor at D:\a\llama.cpp\llama.cpp\ggml\src\ggml-sycl\ggml-sycl.cpp:327
D:\a\llama.cpp\llama.cpp\ggml\src\ggml-sycl\..\ggml-sycl\common.hpp:111: SYCL error
