spring-projects
Structured Output with few shot JSON Example not working
Hi Team, First of all thank you so much for your effort on creating this Spring AI module. I am currently exploring Structured output entity feature to get the structured out in the format I wanted using few a Few Shot JSON example in the prompt itself.
Its throwing an error java.lang.IllegalArgumentException: The template string is not valid
Complete error details are given below.
Environment
Local Environment
Prompt
Extract the key information from the following text delimited by triple backticks and format it in JSON.
I need details like name, booking date, flight information (flight number, origin, destination, departure/arrival times),s
luggage details, ticket price, and seat number.
Here is an example output of the JSON format:\n
{
"name": "John Doe",
"booking_date": "January 11, 2024",
"flight_info": {
"flight_number": "123",
"origin_airport_code": "JFK",
"origin_city": "New York",
"destination_airport_code": "LAX",
"destination_city": "Los Angeles",
"departure_time": "8:00 AM",
"arrival_time": "11:30 AM"
},
"luggage": {
"carry_on": "1",
"checked_bag": "1"
},
"ticket_price": {
"value": "450.00",
"currency": "DOLLAR"
},
"seat_number": "14A"
}
Text: ```Emily Thompson booked a flight on October 10, 2024. She will be flying from New York (JFK) to Los Angeles (LAX) on flight number AA123.The departure time is 8:00 AM, and the arrival time is 11:30 AM. She has a carry-on bag and a checked bag. Her ticket price was $450.00, and she will be seated in 14A.```'
String template file : flight_details_fewshot.st
Extract the key information from the following text delimited by triple backticks and format it in JSON.
I need details like name, booking date, flight information (flight number, origin, destination, departure/arrival times),s
luggage details, ticket price, and seat number.
Here is an example output of the JSON format:\n
{jsonexample}
Text: ```{input}```
Controller:
@Value("classpath:/prompt-templates/structured_outputs/flight_details_fewshot.st")
private Resource flightBookingFewShot;
@PostMapping("/v1/structured_outputs/entity/fewshot")
public Object entityFewShot(@RequestBody @Valid UserInput userInput) {
log.info("userInput message : {} ", userInput);
String jsonExample = "{\n" +
" \"name\": \"John Doe\",\n" +
" \"booking_date\": \"January 11, 2024\",\n" +
" \"flight_info\": {\n" +
" \"flight_number\": \"123\",\n" +
" \"origin_airport_code\": \"JFK\",\n" +
" \"origin_city\": \"New York\",\n" +
" \"destination_airport_code\": \"LAX\",\n" +
" \"destination_city\": \"Los Angeles\",\n" +
" \"departure_time\": \"8:00 AM\",\n" +
" \"arrival_time\": \"11:30 AM\"\n" +
" },\n" +
" \"luggage\": {\n" +
" \"carry_on\": \"1\",\n" +
" \"checked_bag\": \"1\"\n" +
" },\n" +
" \"ticket_price\": {\n" +
" \"value\": \"450.00\",\n" +
" \"currency\": \"DOLLAR\"\n" +
" },\n" +
" \"seat_number\": \"14A\"\n" +
"}";
var promptTemplate = new PromptTemplate(flightBookingFewShot);
var message = promptTemplate.createMessage(Map.of("input", userInput.prompt(), "jsonexample", jsonExample));
var promptMessage = new Prompt(List.of(message));
log.info("Prompt : \n {}", promptMessage);
var requestSpec = chatClient.prompt(promptMessage);
var booking = requestSpec.call().entity(FlightBooking.class);
log.info("booking : {} ", booking);
return booking;
}
Error
{
"timestamp": "2025-03-21T09:49:51.580+00:00",
"status": 500,
"error": "Internal Server Error",
"trace": "java.lang.IllegalArgumentException: The template string is not valid.\n\tat org.springframework.ai.chat.prompt.PromptTemplate.<init>(PromptTemplate.java:86)\n\tat org.springframework.ai.chat.client.advisor.api.AdvisedRequest.toPrompt(AdvisedRequest.java:171)\n\tat org.springframework.ai.chat.client.DefaultChatClient$DefaultChatClientRequestSpec$1.aroundCall(DefaultChatClient.java:680)\n\tat org.springframework.ai.chat.client.advisor.DefaultAroundAdvisorChain.lambda$nextAroundCall$1(DefaultAroundAdvisorChain.java:98)\n\tat io.micrometer.observation.Observation.observe(Observation.java:565)\n\tat org.springframework.ai.chat.client.advisor.DefaultAroundAdvisorChain.nextAroundCall(DefaultAroundAdvisorChain.java:98)\n\tat org.springframework.ai.chat.client.DefaultChatClient$DefaultCallResponseSpec.doGetChatResponse(DefaultChatClient.java:493)\n\tat org.springframework.ai.chat.client.DefaultChatClient$DefaultCallResponseSpec.lambda$doGetObservableChatResponse$1(DefaultChatClient.java:482)\n\tat io.micrometer.observation.Observation.observe(Observation.java:565)\n\tat org.springframework.ai.chat.client.DefaultChatClient$DefaultCallResponseSpec.doGetObservableChatResponse(DefaultChatClient.java:482)\n\tat org.springframework.ai.chat.client.DefaultChatClient$DefaultCallResponseSpec.doSingleWithBeanOutputConverter(DefaultChatClient.java:456)\n\tat org.springframework.ai.chat.client.DefaultChatClient$DefaultCallResponseSpec.entity(DefaultChatClient.java:451)\n\tat com.llm.structuredoutputs.StructuredOutputsController.entityFewShot(StructuredOutputsController.java:147)\n\tat java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)\n\tat java.base/java.lang.reflect.Method.invoke(Method.java:580)\n\tat org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:255)\n\tat org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:188)\n\tat org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:118)\n\tat org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:926)\n\tat org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:831)\n\tat org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)\n\tat org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1089)\n\tat org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:979)\n\tat org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1014)\n\tat org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:914)\n\tat jakarta.servlet.http.HttpServlet.service(HttpServlet.java:590)\n\tat org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:885)\n\tat jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:195)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)\n\tat org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)\n\tat org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)\n\tat org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)\n\tat org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)\n\tat org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167)\n\tat org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)\n\tat org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:483)\n\tat org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115)\n\tat org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)\n\tat org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)\n\tat org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344)\n\tat org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:384)\n\tat org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)\n\tat org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:905)\n\tat org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1741)\n\tat org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)\n\tat org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1190)\n\tat org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)\n\tat org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\nCaused by: org.stringtemplate.v4.compiler.STException\n\tat org.stringtemplate.v4.compiler.Compiler.reportMessageAndThrowSTException(Compiler.java:224)\n\tat org.stringtemplate.v4.compiler.Compiler.compile(Compiler.java:154)\n\tat org.stringtemplate.v4.STGroup.compile(STGroup.java:514)\n\tat org.stringtemplate.v4.ST.<init>(ST.java:162)\n\tat org.stringtemplate.v4.ST.<init>(ST.java:156)\n\tat org.springframework.ai.chat.prompt.PromptTemplate.<init>(PromptTemplate.java:80)\n\t... 60 more\n",
"message": "The template string is not valid.",
"path": "/springai/v1/structured_outputs/entity/fewshot"
}
Spring AI Version
set('springAiVersion', "1.0.0-M6")
Expected output
{
"name": "Li Wei",
"booking_date": "November 5, 2024",
"flight_info": {
"flight_number": "CA456",
"origin_airport_code": "PEK",
"origin_city": "Beijing",
"destination_airport_code": "PVG",
"destination_city": "Shanghai",
"departure_time": "10:00 AM",
"arrival_time": "12:30 PM"
},
"luggage": {
"carry_on": "1",
"checked_bag": "1"
},
"ticket_price": {
"value": "3200.00",
"currency": "YUAN"
},
"seat_number": "22C"
}
Working example without the entity() function call.
When I run the same example but just use the chatClient.prompt(promptMessage).call.content(), its working.
@PostMapping("/v1/structured_outputs/fewshot")
public Object chat1(@RequestBody @Valid UserInput userInput) {
log.info("userInput message : {} ", userInput);
var promptTemplate = new PromptTemplate(flightBookingFewShot);
var message = promptTemplate.createMessage(Map.of("input", userInput.prompt(), "jsonexample", CommonUtil.flightJson()));
var promptMessage = new Prompt(List.of(message));
var requestSpec = chatClient.prompt(promptMessage);
log.info("requestSpec : {} ", requestSpec);
return requestSpec.call().content();
// var responseSpec = requestSpec.call().entity(FlightBooking.class);
// return responseSpec;
}
This kind of interaction is pretty common to drive the LLM to map the right values into JSON properties so that the application can take necessary action on them.
Fixing this would be a really helpful in dealing with Structured outputs.
Thanks, Dilip Sundarraj
I also encountered this problem. The problem occures in org.springframework.ai.chat.client.advisor.api.AdvisedRequest.toPrompt() method:
public Prompt toPrompt() {
ArrayList<Message> messages = new ArrayList(this.messages());
String processedSystemText = this.systemText();
if (StringUtils.hasText(processedSystemText)) {
if (!CollectionUtils.isEmpty(this.systemParams())) {
processedSystemText = (new PromptTemplate(processedSystemText, this.systemParams())).render();
}
messages.add(new SystemMessage(processedSystemText));
}
String formatParam = (String)this.adviseContext().get("formatParam");
String processedUserText = StringUtils.hasText(formatParam) ? this.userText() + System.lineSeparator() + "{spring_ai_soc_format}" : this.userText();
if (StringUtils.hasText(processedUserText)) {
Map<String, Object> userParams = new HashMap(this.userParams());
if (StringUtils.hasText(formatParam)) {
userParams.put("spring_ai_soc_format", formatParam);
}
if (!CollectionUtils.isEmpty(userParams)) {
processedUserText = (new PromptTemplate(processedUserText, userParams)).render();
}
messages.add(new UserMessage(processedUserText, this.media()));
}
ChatOptions var6 = this.chatOptions();
if (var6 instanceof FunctionCallingOptions functionCallingOptions) {
if (!this.functionNames().isEmpty()) {
functionCallingOptions.setFunctions(new HashSet(this.functionNames()));
}
if (!this.functionCallbacks().isEmpty()) {
functionCallingOptions.setFunctionCallbacks(this.functionCallbacks());
}
if (!CollectionUtils.isEmpty(this.toolContext())) {
functionCallingOptions.setToolContext(this.toolContext());
}
}
return new Prompt(messages, this.chatOptions());
}
Under the hood .entity() method uses AdvisorsApi, adding "formatParam" Adviser to AdviseContext for ChatClient. We have:
String processedUserText = StringUtils.hasText(formatParam) ? this.userText() + System.lineSeparator() + "{spring_ai_soc_format}" : this.userText();
-
formatParam- helper message, obtained from AdviseContext by the "formatParam" key -
this.userText()- your user's message: your JSON; -
{spring_ai_soc_format}- a special template used to substitute a message about how LLM should structure its answer, stored in userParams As a result, we get text that contains our JSON along with {}, as well as a line-glue{spring_ai_soc_format}.
Then
processedUserText = (new PromptTemplate(processedUserText, userParams)).render();
is called and PromptTemplate tries to substitute the string, but our JSON is treated as a placeholder.
Well, I think that the main problem of this method is userText concatenation before the format message substitution. Is there any reason why it is made this way? I think it would be better to allow the client to configure how the template is applied to userText.
As stated in other open issues, you can escape { with {{ and } with }} (or \} etc.) , but then LLM may report that the data is in an invalid format: invalid JSON structure. So you have to add another helper message about treating {{}} as {}, but that's a crutch. Maybe this problem can be solved by using ModelApi and Prompt natively, but I'd like to use ChatClientApi. I think it makes sense that I use structured input to get structured output...
This is another example of the problem caused by inability to turn off ST templating, which has prompted several other issues.
JSON few shot examples are a very common requirement. Being unable to pass them in makes ChatClient unsuited to many problems.
There should be an option to disable all template rendering so literal strings can be passed in using ChatClient.
What about using < and > as delimiter instead of { and }, which I'd assume to be used more frequently in prompts?
Or some even more esoteric delimiters?
Or provide some API tweak like bringing back the .builder() as currently described in the 1.0.0-SNAPSHOT docs, or just as constructor args (allowing to use PromptTemplate with JSON explicitly at least)?
Example using a custom StringTemplate renderer with '<' and '>' delimiters
PromptTemplate promptTemplate = PromptTemplate.builder()
.renderer(StTemplateRenderer.builder().startDelimiterToken('<').endDelimiterToken('>').build())
Happy to provide a PR if that's helpful?
Is this still a problem in the current 1.0.0-SNAPSHOT?
The ChatClient supports passing a custom TemplateRenderer to allow using different delimiters and support JSON in the prompt. More info here. It could be a NoOpTemplateRenderer if you don't need any templating at all. Or you could use the StTemplateRenderer and customise the delimiters to something else than curly braces, as shared by @fabapp2.
Hi @ThomasVitale — just saw your #2780, which seems to solve the issue for me in M8. Sorry I missed that earlier! It looks like the docs are still referencing the old .builder() API. Thanks for the PR!