Add validation mechanism that does not require JSR-380 annotations

[johnsonr]

We presently support validation only via JSR-380 annotations. While this is valuable, such annotations might not exist on classes we use, and we might not control those classes. Also, validation may be too complex for JSR-280 annotations.

We need to provide a way of defining a validator that works programmatically and produces a compatible result, via InvalidLlmReturnType exception.

This is distinct from the problem of validating UserContent.

[igordayen]

Could using ValidatorUtils / SmartValidator from spring framework be considered. Custom annotations could also be developed with delegation to existing spring utils or special utils can be developed by implementing spring validator interface.

[igordayen]

Design notes:

validation-options-comparison.md

[johnsonr]

I'd rather not increase API dependence on Spring. Although this is certainly a potential path.

[igordayen]

I'd rather not increase API dependence on Spring. Although this is certainly a potential path.

noted, understood

[m-kostira]

Hi guys, pitching in:

I'm using the programatic API, and in some situations it would be nice to have a custom validation function applied to the generated object/text, so that an error message can be returned back to the llm as a hint how to fix the generation. E.g something like:

ai
   .withOutputValidator(result -> {
        // custom validation logic
        if (!validate(result)) {
            throw new Exception("Error..."); // framework propagates the error message to the LLM and it retries while avoiding making the same error 
        }
    })
    .createObject(promptString, Result.class);

Again, this is nice to have, not a deal breaker. I sometimes get around this by exposing a tool method to accept the result, and doing validation inside the method.

[igordayen]

Hi guys, pitching in:

I'm using the programatic API, and in some situations it would be nice to have a custom validation function applied to the generated object/text, so that an error message can be returned back to the llm as a hint how to fix the generation. E.g something like:

hello @m-kostira - thanks for suggestions. may i inquire: you would like to validate "raw" LLM response prior to object creation?

[m-kostira]

I'd like to validate the deserialized object rather than the raw text response.

On Thu, 4 Dec 2025, 01:43 Igor Dayen, @.***> wrote:

igordayen left a comment (embabel/embabel-agent#978) https://github.com/embabel/embabel-agent/issues/978#issuecomment-3609297059

Hi guys, pitching in:

I'm using the programatic API, and in some situations it would be nice to have a custom validation function applied to the generated object/text, so that an error message can be returned back to the llm as a hint how to fix the generation. E.g something like:

hello @m-kostira https://github.com/m-kostira - thanks for suggestions. may i inquire: you would like to validate "raw" LLM response prior to object creation?

— Reply to this email directly, view it on GitHub https://github.com/embabel/embabel-agent/issues/978#issuecomment-3609297059, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADRFXMXRFMAR3TJY4R7A2KT375YQTAVCNFSM6AAAAACKEZECLKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTMMBZGI4TOMBVHE . You are receiving this because you were mentioned.Message ID: @.***>

[igordayen]

I'd like to validate the deserialized object rather than the raw text response. @m-kostira

would be helpful if you re-confirm proper understanding of the ask:

1. LLM returns - sample output: July is the hottest month in New York, average tempreture is 78F {name: July, ranking:hottest}
2. so, you would like to validate: {name: July, ranking:hottest} adheres to schema prior to transforming to record MonthItem(name:String, ranking:String) ?

Would be also helpful if you can provide feedback on proposals:

• implicit validation by matching validation condition signature with action signature
• explicit validation by demarcating action with validation annotation
• hybrid, both

Thank you.

[m-kostira]

So my use case is something similar to the following: the user wants to plan a vacation and tells the LLM their preferred destinations are Greece and Italy and they can only take vacation in August, June, or September.

Then the user asks the LLM to come up with some destinations with cheap plane tickets for a one-week stay. Let's say the output is an object, basically a round trip flight. Even if the output adheres to the schema, I want to be able to verify whether the flight dates are in the requested months, whether the destinations are in Greece/Italy vs somewhere else like Spain or Turkey.

So basically we're not validating the schema, rather we're validating the reasoning according to some custom business logic. IMO that can be achieved by a custom validator method supplied by the client code.

Regarding the proposals, I'm using Embabel via the programatic API, so I don't really have any preference how it's implemented with regards to actions/goals/annotations, as long as there is a low level way to supply the validator method, like in my snippet in the previous comment.

Thanks!

Edit: come to think of it, maybe there is already functionality that supports this in Spring AI? E.g. if we can add a custom advisor in the advisor chain, maybe..

[igordayen]

hello @m-kostira , getting more clarity now. thank you.

Currently framework filters out reasoning/thinking, purifies object construction content.

Thinking about options.

1. Define record Fllight {flight#, from-city, to-city, from-country, to-country, flightt-date, month-from, month-to}, and validate structured object
2. Framework Enhancement, so "withResultValidation" will signal framework to preserve reasoning content. Reasoning content is optional and unstructured, though.
3. Consider streaming, feature is in pilot testing to be releaed as GA soon. see and run:

embabel-agent/embabel-agent-autoconfigure/models/embabel-agent-openai-autoconfigure/src/test/kotlin/com/embabel/agent/config/models/openai/LLMStreamingIT.kt

Streaming framework streams sequence of StreamingEvents as either {Object or Thinking}, thus preserving reasoning. As stated in [2], reasoning is optional (models may or may not support it) and unstructured.

@johnsonr , could you please advise. Thank you.

[harinda05]

I prefer Option 2 - where action-level validation happens. This can probably be used as an extension for guardrail implementation as well! issue-984 and issue-983

However, the question remains, where the validations should happen. From a guardrail perspective, for LLM responses, in my opinion, the guardrail should be applied at the LLM response level before it gets deserialized to the given type class. Need to verify the feasibility of handling that as non-invasive as possible to the current framework.

[igordayen]

However, the question remains, where the validations should happen. From a guardrail perspective, for LLM responses, in my opinion, the guardrail should be applied at the LLM response level before it gets deserialized to the given type class. Need to verify the feasibility of handling that as non-invasive as possible to the current framework.

@harinda05 could you please review discussion above, initiated by @m-kostira on the same venue and avise?

[harinda05]

@m-kostira 's requirement seems to be perfectly aligned with the action level validation, where basically, the intermediary actions should be validated before reaching the goal. However, where this validation should be done is debatable.

As I can understand, the requirement is basically we're not validating the schema, rather we're validating the reasoning according to some custom business logic.

IMO it all depends on how the actions of the agent are defined! In some cases, it could be enough to validate the constructed object. And in some cases, it might make sense to validate the LLm response itself (ex, in guardrails for LLM responses).

But in neither case do we validate the reasoning of the LLM! This is what I am a bit sceptical about. Do we need to validate the reasoning of an LLM action at all?

[igordayen]

1. action validation execution timing: before action, handled by framework
2. scope of Validation: any objects in Action Context
3. reasoning - as unstructured text - is available in streaming API.

Hope it helps.

Thanks.

[harinda05]

Ah sorry! I mistook reasoning for something else, for example COT(chain of thought) of the model. I guess reasoning here refers to the naked string response LLM produces!