ggml-org
Misc. bug: server provides strutured output for response_format: json_object, but not for response_format: json_schema
Name and Version
on latest commit ce8784bdb153ff7794dde5a50b0ebfa51baa6171
but have been noticing it for several days now
Operating systems
No response
Which llama.cpp modules do you know to be affected?
No response
Problem description & steps to reproduce
I have been trying to follow the steps for structured json output using json_schema in server here
however, I was not able to get any combination of json_schema to work. I was able to get json_object to work, doing effectively the same thing, but since this differs from the OpenAI API (which I suppose server is striving for) I suppose it's a bug. The official server docs also mention json_schema is supported (see the above link)
Does not work, does not apply any structured json schema at all, and does not indicate any failure or warning (simply returns unstructured output), taken directly from OpenAI's docs (click "curl" for the curl version)
"response_format": {
"type": "json_schema",
"json_schema": {
"name": "math_reasoning",
"schema": {
"type": "object",
"properties": {
"steps": {
"type": "array",
"items": {
"type": "object",
"properties": {
"explanation": {
"type": "string"
},
"output": {
"type": "string"
}
},
"required": [
"explanation",
"output"
],
"additionalProperties": false
}
},
"final_answer": {
"type": "string"
}
},
"required": [
"steps",
"final_answer"
],
"additionalProperties": false
},
"strict": true
}
}
however, "json_object" works as expected
"response_format": {
"type": "json_object",
"schema": {
"type": "object",
"properties": {
"steps": {
"type": "array",
"items": {
"type": "object",
"properties": {
"explanation": {
"type": "string"
},
"output": {
"type": "string"
}
},
"required": [
"explanation",
"output"
],
"additionalProperties": false
}
},
"final_answer": {
"type": "string"
}
},
"required": [
"steps",
"final_answer"
],
"additionalProperties": false
}
}
First Bad Commit
No response
Relevant log output
No response
@andysalerno This issue was already addressed in #9527. Here are the commands of a basic example and outputs when I run it for both json_schema and json_object.
Command for json_schema:
curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -H "Authorization: Bearer no-key" -d '{
"model": "tinyllama",
"messages": [
{"role": "system", "content": "Extract the event information."},
{"role": "user", "content": "Alice and Bob are going to a science fair on Friday."}
],
"response_format": {
"type": "json_schema",
"json_schema": {
"name": "calendar event",
"schema": {
"type": "object",
"properties": {
"participants": {
"type": "array",
"items": {
"type": "string",
"additionalProperties": false
}
},
"name":{
"type": "string"
},
"date":{
"type": "string"
}
},
"required": ["date", "name", "participants"],
"additionalProperties": false
},
"strict": true
}
}
}' | jq ".choices[0].message.content" | jq -r "." | jq .
Output:
{
"participants": [
"Alice",
"Bob"
],
"name": "Science fair",
"date": "Friday, March 14, 2021 at 03:00 PM (UTC)"
}
Command for json_object:
curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -H "Authorization: Bearer no-key" -d '{
"model": "tinyllama",
"messages": [
{"role": "system", "content": "Extract the event information."},
{"role": "user", "content": "Alice and Bob are going to a science fair on Friday."}
],
"response_format": {
"type": "json_object",
"schema": {
"type": "object",
"properties": {
"participants": {
"type": "array",
"items": {
"type": "string",
"additionalProperties": false
}
},
"name": {
"type": "string"
},
"date": {
"type": "string"
}
},
"required": [
"participants",
"name",
"date"
],
"additionalProperties": false
}
}
}' | jq ".choices[0].message.content" | jq -r "." | jq .
Output:
{
"participants": [
"Alice",
"Bob"
],
"name": "Science Fair",
"date": "18-11-2021"
}
Both examples here follow the schema defined in the request. Please let me know if the docs need to be updated. It would help if you could share some terminal output or logs.
It does indeed seem to not be working.
This response_format:
{
"type": "json_object",
"schema": { ... },
}
works fine. But I could not get "type": "json_schema" to control the output.
