Skip to content

GH-5832: Validate all results in StructuredOutputValidationAdvisor#5846

Open
suryateja-g13 wants to merge 1 commit intospring-projects:mainfrom
suryateja-g13:GH-5832
Open

GH-5832: Validate all results in StructuredOutputValidationAdvisor#5846
suryateja-g13 wants to merge 1 commit intospring-projects:mainfrom
suryateja-g13:GH-5832

Conversation

@suryateja-g13
Copy link
Copy Markdown

Thank you for taking time to contribute this pull request!

Problem

StructuredOutputValidationAdvisor.validateOutputSchema() called chatResponse.getResult(), which only retrieves the first result. When a ChatResponse contains multiple results, all subsequent results were silently skipped, allowing invalid JSON in later generations to bypass schema validation entirely.

This was tracked as a TODO comment in the source:

// TODO: should we consider validation for multiple results?
String json = chatClientResponse.chatResponse().getResult().getOutput().getText();

Solution

Replace the single-result check with getResults() iteration. The first failing result short-circuits the loop and triggers a retry with an augmented prompt. All results must pass for the response to be accepted.

Changes

  • StructuredOutputValidationAdvisor.java — replaced getResult() with getResults() iteration in validateOutputSchema(); null/empty results check updated accordingly
  • StructuredOutputValidationAdvisorTests.java — updated existing testAdviseCallWithNullResult to stub getResults() (the old getResult() stub became unused); added two new tests:
    • testValidationWithMultipleResultsAllValid — response with 2 valid results passes without retry
    • testValidationWithMultipleResultsSecondInvalid — response where second result is invalid triggers retry

Checklist

  • Added Signed-off-by line to commit (DCO)
  • Rebased on latest main
  • Unit tests added/updated
  • Build passes (mvn package -pl spring-ai-client-chat)

Fixes #5832

…ationAdvisor

Previously, validateOutputSchema() called getResult() which only returns
the first ChatResponse result. Subsequent results were silently ignored,
allowing invalid JSON in later generations to bypass schema validation
entirely. This was tracked as a TODO comment in the source.

Replace the single-result check with getResults() iteration: the first
failing result short-circuits and triggers a retry with an augmented
prompt. All results must be valid for the response to be accepted.

Fixes spring-projectsGH-5832 (spring-projects#5832)

Signed-off-by: Gorre Surya <sgorre92@gmail.com>
@redinside-dev
Copy link
Copy Markdown

LGTM. Clean iteration over getResults() with short-circuit on first failure. Good test coverage with testValidationWithMultipleResultsAllValid and testValidationWithMultipleResultsSecondInvalid. Ship it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

StructuredOutputValidationAdvisor validates only first result, ignoring remaining results

2 participants