ggml-org
Misc. bug: jinja template with exception ruins the template
Name and Version
version: 4529 (12c2bdf2) built with MSVC 19.29.30157.0 for
Operating systems
Windows
Which llama.cpp modules do you know to be affected?
llama-server
Command line
Problem description & steps to reproduce
I used ministral, its template has "raise_exception" if it doesn't like some stuff. i believe it results in a broken templating when the jinja rules are broken. maybe i'm wrong in the cause but the templated stuff is broken anyways.
for example this request: [ { "content": "Say: 1", "role": "user" }, { "content": "Say: 1", "role": "user" }, { "content": "Say: 1", "role": "user" }, { "content": "Say: 2", "role": "assistant" }, { "content": "Say: 2", "role": "assistant" }, { "content": "Say: 2", "role": "assistant" } ]
produces this:
[INST]Say: 1[/INST]Say: 1[/INST]Say: 1[/INST]Say: 2[INST]Say: 2[INST]Say: 2"
which is a bug (because the formatting is messed up) due to the definitions in the template denying using 2 user roles in a row. And at that things are silently broken on the inside.
i believe if such thing happens, there should be a fall back to the old mechanism, that was working fine templating such queries, doesn't matter what model believes should be allowed. after all, we should have a way to experiment with models, not to be dictated by the model on input format with no way to override it.
i guess some people would prefer to see the error at that. so i believe we need a switch to allow templating anything we wish, just the way it used always to be and another way to return the error for template preferences being violated.
also, i've noticed that usual example template is broken too, for some reason.
template: {%- if messages[0]["role"] == "system" %} {%- set system_message = messages[0]["content"] %} {%- set loop_messages = messages[1:] %} {%- else %} {%- set loop_messages = messages %} {%- endif %} {%- if not tools is defined %} {%- set tools = none %} {%- endif %} {%- set user_messages = loop_messages | selectattr("role", "equalto", "user") | list %} {#- This block checks for alternating user/assistant messages, skipping tool calling messages #} {%- set ns = namespace() %} {%- set ns.index = 0 %} {%- for message in loop_messages %} {%- if not (message.role == "tool" or message.role == "tool_results" or (message.tool_calls is defined and message.tool_calls is not none)) %} {%- if (message["role"] == "user") != (ns.index % 2 == 0) %} {{- raise_exception("After the optional system message, conversation roles must alternate user/assistant/user/assistant/...") }} {%- endif %} {%- set ns.index = ns.index + 1 %} {%- endif %} {%- endfor %} {{- bos_token }} {%- for message in loop_messages %} {%- if message["role"] == "user" %} {%- if tools is not none and (message == user_messages[-1]) %} {{- "[AVAILABLE_TOOLS][" }} {%- for tool in tools %} {%- set tool = tool.function %} {{- '{"type": "function", "function": {' }} {%- for key, val in tool.items() if key != "return" %} {%- if val is string %} {{- '"' + key + '": "' + val + '"' }} {%- else %} {{- '"' + key + '": ' + val|tojson }} {%- endif %} {%- if not loop.last %} {{- ", " }} {%- endif %} {%- endfor %} {{- "}}" }} {%- if not loop.last %} {{- ", " }} {%- else %} {{- "]" }} {%- endif %} {%- endfor %} {{- "[/AVAILABLE_TOOLS]" }} {%- endif %} {%- if loop.last and system_message is defined %} {{- "[INST]" + system_message + "\n\n" + message["content"] + "[/INST]" }} {%- else %} {{- "[INST]" + message["content"] + "[/INST]" }} {%- endif %} {%- elif (message.tool_calls is defined and message.tool_calls is not none) %} {{- "[TOOL_CALLS][" }} {%- for tool_call in message.tool_calls %} {%- set out = tool_call.function|tojson %} {{- out[:-1] }} {%- if not tool_call.id is defined or tool_call.id|length != 9 %} {{- raise_exception("Tool call IDs should be alphanumeric strings with length 9!") }} {%- endif %} {{- ', "id": "' + tool_call.id + '"}' }} {%- if not loop.last %} {{- ", " }} {%- else %} {{- "]" + eos_token }} {%- endif %} {%- endfor %} {%- elif message["role"] == "assistant" %} {{- message["content"] + eos_token}} {%- elif message["role"] == "tool_results" or message["role"] == "tool" %} {%- if message.content is defined and message.content.content is defined %} {%- set content = message.content.content %} {%- else %} {%- set content = message.content %} {%- endif %} {{- '[TOOL_RESULTS]{"content": ' + content|string + ", " }} {%- if not message.tool_call_id is defined or message.tool_call_id|length != 9 %} {{- raise_exception("Tool call IDs should be alphanumeric strings with length 9!") }} {%- endif %} {{- '"call_id": "' + message.tool_call_id + '"}[/TOOL_RESULTS]' }} {%- else %} {{- raise_exception("Only user and assistant roles are supported, with the exception of an initial optional system message!") }} {%- endif %} {%- endfor %}
First Bad Commit
it didn't happen in version b4404, don't know when jinja stuff was introduced
Relevant log output
main: chat template, chat_template: {%- if messages[0]["role"] == "system" %}
{%- set system_message = messages[0]["content"] %}
{%- set loop_messages = messages[1:] %}
{%- else %}
{%- set loop_messages = messages %}
{%- endif %}
{%- if not tools is defined %}
{%- set tools = none %}
{%- endif %}
{%- set user_messages = loop_messages | selectattr("role", "equalto", "user") | list %}
{#- This block checks for alternating user/assistant messages, skipping tool calling messages #}
{%- set ns = namespace() %}
{%- set ns.index = 0 %}
{%- for message in loop_messages %}
{%- if not (message.role == "tool" or message.role == "tool_results" or (message.tool_calls is defined and message.tool_calls is not none)) %}
{%- if (message["role"] == "user") != (ns.index % 2 == 0) %}
{{- raise_exception("After the optional system message, conversation roles must alternate user/assistant/user/assistant/...") }}
{%- endif %}
{%- set ns.index = ns.index + 1 %}
{%- endif %}
{%- endfor %}
{{- bos_token }}
{%- for message in loop_messages %}
{%- if message["role"] == "user" %}
{%- if tools is not none and (message == user_messages[-1]) %}
{{- "[AVAILABLE_TOOLS][" }}
{%- for tool in tools %}
{%- set tool = tool.function %}
{{- '{"type": "function", "function": {' }}
{%- for key, val in tool.items() if key != "return" %}
{%- if val is string %}
{{- '"' + key + '": "' + val + '"' }}
{%- else %}
{{- '"' + key + '": ' + val|tojson }}
{%- endif %}
{%- if not loop.last %}
{{- ", " }}
{%- endif %}
{%- endfor %}
{{- "}}" }}
{%- if not loop.last %}
{{- ", " }}
{%- else %}
{{- "]" }}
{%- endif %}
{%- endfor %}
{{- "[/AVAILABLE_TOOLS]" }}
{%- endif %}
{%- if loop.last and system_message is defined %}
{{- "[INST]" + system_message + "\n\n" + message["content"] + "[/INST]" }}
{%- else %}
{{- "[INST]" + message["content"] + "[/INST]" }}
{%- endif %}
{%- elif (message.tool_calls is defined and message.tool_calls is not none) %}
{{- "[TOOL_CALLS][" }}
{%- for tool_call in message.tool_calls %}
{%- set out = tool_call.function|tojson %}
{{- out[:-1] }}
{%- if not tool_call.id is defined or tool_call.id|length != 9 %}
{{- raise_exception("Tool call IDs should be alphanumeric strings with length 9!") }}
{%- endif %}
{{- ', "id": "' + tool_call.id + '"}' }}
{%- if not loop.last %}
{{- ", " }}
{%- else %}
{{- "]" + eos_token }}
{%- endif %}
{%- endfor %}
{%- elif message["role"] == "assistant" %}
{{- message["content"] + eos_token}}
{%- elif message["role"] == "tool_results" or message["role"] == "tool" %}
{%- if message.content is defined and message.content.content is defined %}
{%- set content = message.content.content %}
{%- else %}
{%- set content = message.content %}
{%- endif %}
{{- '[TOOL_RESULTS]{"content": ' + content|string + ", " }}
{%- if not message.tool_call_id is defined or message.tool_call_id|length != 9 %}
{{- raise_exception("Tool call IDs should be alphanumeric strings with length 9!") }}
{%- endif %}
{{- '"call_id": "' + message.tool_call_id + '"}[/TOOL_RESULTS]' }}
{%- else %}
{{- raise_exception("Only user and assistant roles are supported, with the exception of an initial optional system message!") }}
{%- endif %}
{%- endfor %}
, example_format: '[INST]You are a helpful assistant
Hello[/INST]Hi there</s>[INST]How are you?[/INST]'
==================================
main: chat template, chat_template: {%- if messages[0]["role"] == "system" %}
{%- set system_message = messages[0]["content"] %}
{%- set loop_messages = messages[1:] %}
{%- else %}
{%- set loop_messages = messages %}
{%- endif %}
{%- if not tools is defined %}
{%- set tools = none %}
{%- endif %}
{%- set user_messages = loop_messages | selectattr("role", "equalto", "user") | list %}
{#- This block checks for alternating user/assistant messages, skipping tool calling messages #}
{%- set ns = namespace() %}
{%- set ns.index = 0 %}
{%- for message in loop_messages %}
{%- if not (message.role == "tool" or message.role == "tool_results" or (message.tool_calls is defined and message.tool_calls is not none)) %}
{%- if (message["role"] == "user") != (ns.index % 2 == 0) %}
{{- raise_exception("After the optional system message, conversation roles must alternate user/assistant/user/assistant/...") }}
{%- endif %}
{%- set ns.index = ns.index + 1 %}
{%- endif %}
{%- endfor %}
{{- bos_token }}
{%- for message in loop_messages %}
{%- if message["role"] == "user" %}
{%- if tools is not none and (message == user_messages[-1]) %}
{{- "[AVAILABLE_TOOLS][" }}
{%- for tool in tools %}
{%- set tool = tool.function %}
{{- '{"type": "function", "function": {' }}
{%- for key, val in tool.items() if key != "return" %}
{%- if val is string %}
{{- '"' + key + '": "' + val + '"' }}
{%- else %}
{{- '"' + key + '": ' + val|tojson }}
{%- endif %}
{%- if not loop.last %}
{{- ", " }}
{%- endif %}
{%- endfor %}
{{- "}}" }}
{%- if not loop.last %}
{{- ", " }}
{%- else %}
{{- "]" }}
{%- endif %}
{%- endfor %}
{{- "[/AVAILABLE_TOOLS]" }}
{%- endif %}
{%- if loop.last and system_message is defined %}
{{- "[INST]" + system_message + "\\n\\n" + message["content"] + "[/INST]" }}
{%- else %}
{{- "[INST]" + message["content"] + "[/INST]" }}
{%- endif %}
{%- elif (message.tool_calls is defined and message.tool_calls is not none) %}
{{- "[TOOL_CALLS][" }}
{%- for tool_call in message.tool_calls %}
{%- set out = tool_call.function|tojson %}
{{- out[:-1] }}
{%- if not tool_call.id is defined or tool_call.id|length != 9 %}
{{- raise_exception("Tool call IDs should be alphanumeric strings with length 9!") }}
{%- endif %}
{{- ', "id": "' + tool_call.id + '"}' }}
{%- if not loop.last %}
{{- ", " }}
{%- else %}
{{- "]" + eos_token }}
{%- endif %}
{%- endfor %}
{%- elif message["role"] == "assistant" %}
{{- message["content"] + eos_token}}
{%- elif message["role"] == "tool_results" or message["role"] == "tool" %}
{%- if message.content is defined and message.content.content is defined %}
{%- set content = message.content.content %}
{%- else %}
{%- set content = message.content %}
{%- endif %}
{{- '[TOOL_RESULTS]{"content": ' + content|string + ", " }}
{%- if not message.tool_call_id is defined or message.tool_call_id|length != 9 %}
{{- raise_exception("Tool call IDs should be alphanumeric strings with length 9!") }}
{%- endif %}
{{- '"call_id": "' + message.tool_call_id + '"}[/TOOL_RESULTS]' }}
{%- else %}
{{- raise_exception("Only user and assistant roles are supported, with the exception of an initial optional system message!") }}
{%- endif %}
{%- endfor %}
, example_format: '[INST]You are a helpful assistant
Hello[/INST]Hi there</s>[INST]How are you?[/INST]'
main: server is listening on http://127.0.0.1:8085 - starting the main loop
que start_loop: processing new tasks
que start_loop: update slots
srv update_slots: all slots are idle
srv kv_cache_cle: clearing KV cache
que start_loop: waiting for new tasks
formatted_chat: '[INST]Say: 1[/INST]Say: 1[/INST]Say: 1[/INST]Say: 2</s>[INST]Say: 2</s>[INST]Say: 2</s>'
