Skip to content

feat: enforce output_model validation in EventLoopNode (fixes #5929)#6005

Open
Waryjustice wants to merge 1 commit intoaden-hive:mainfrom
Waryjustice:enforce-output-model-validation
Open

feat: enforce output_model validation in EventLoopNode (fixes #5929)#6005
Waryjustice wants to merge 1 commit intoaden-hive:mainfrom
Waryjustice:enforce-output-model-validation

Conversation

@Waryjustice
Copy link
Contributor

Summary

Closes #5929.

NodeSpec.output_model and max_validation_retries were defined and tested, but never read by the runtime. A node declaring output_model=OrderSchema behaved identically to one without it, silently passing malformed LLM outputs into shared memory.

This PR wires output_model into the ACCEPT path of EventLoopNode.execute().

Changes

\core/framework/graph/event_loop_node.py\

  • Import \OutputValidator\ from \ ramework.graph.validator\
  • Initialise _validation_retry_count = 0\ alongside the other loop counters
  • After the missing-keys guard in the ACCEPT path, call \OutputValidator.validate_with_pydantic(accumulator.to_dict(), output_model):
    • ValidationError + retries remaining → inject [Output validation failed]: \ into conversation history and \continue\ the loop (RETRY)
    • ValidationError + retries exhausted → ESCALATE with a structured error message, log via
      untime_logger, and return a failed \NodeResult\
    • Validation passes → fall through to the existing memory-write path unchanged

\core/tests/test_pydantic_validation.py\

  • New \TestRuntimeValidationPath\ class with 8 tests that mirror the exact validation logic in the ACCEPT path:
    • Valid outputs pass silently
    • Invalid outputs produce properly-formatted feedback
    • Injected message matches the [Output validation failed]:\ prefix
    • Default retry budget (2) is tracked correctly
    • \max_validation_retries=0\ escalates on the first failure
    • Nodes without \output_model\ skip validation (zero behaviour change)
    • Type-coercion failures and constraint violations are caught
    • Successful validation returns the correct model instance

Behaviour unchanged for existing nodes

Nodes that do not set \output_model\ take the \if ctx.node_spec.output_model is not None\ fast path and are completely unaffected.

Wire NodeSpec.output_model into the set_output handling path so that
structured-output validation is actually enforced at runtime.

Previously, output_model and max_validation_retries were defined on
NodeSpec (and tested for existence) but were never read by the runtime.
A node declaring output_model=OrderSchema behaved identically to one
without it, silently passing malformed LLM outputs into shared memory.

Changes:
- Import OutputValidator in event_loop_node.py
- Track _validation_retry_count in the main execute() loop
- In the ACCEPT path (after missing-keys guard), call
  OutputValidator.validate_with_pydantic(accumulator.to_dict(), output_model)
  * On ValidationError: inject '[Output validation failed]: <feedback>'
    into conversation history and RETRY (up to max_validation_retries)
  * On exhausted retries: ESCALATE with structured error detail and
    log via runtime_logger (log_step + log_node_complete)
  * On success: fall through to the normal memory-write path unchanged
- Nodes without output_model are completely unaffected (zero behaviour change)
- No new dependencies — pydantic and OutputValidator already present

Extend core/tests/test_pydantic_validation.py with TestRuntimeValidationPath:
tests that mirror the exact validation logic executed in the ACCEPT path,
covering: valid pass-through, invalid feedback generation, injection message
format, retry budget tracking, zero-retries escalation, type coercion
failures, constraint violations, and default-value preservation.

Fixes aden-hive#5929

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions
Copy link

github-actions bot commented Mar 7, 2026

PR Closed - Requirements Not Met

This PR has been automatically closed because it doesn't meet the requirements.

PR Author: @Waryjustice
Found issues: #5929 (assignees: none)
Problem: The PR author must be assigned to the linked issue.

To fix:

  1. Assign yourself (@Waryjustice) to one of the linked issues
  2. Re-open this PR

Exception: To bypass this requirement, you can:

  • Add the micro-fix label or include micro-fix in your PR title for trivial fixes
  • Add the documentation label or include doc/docs in your PR title for documentation changes

Micro-fix requirements (must meet ALL):

Qualifies Disqualifies
< 20 lines changed Any functional bug fix
Typos & Documentation & Linting Refactoring for "clean code"
No logic/API/DB changes New features (even tiny ones)

Why is this required? See #472 for details.

@github-actions github-actions bot closed this Mar 7, 2026
@bryanadenhq bryanadenhq reopened this Mar 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: Enforce output_model validation in EventLoopNode — NodeSpec field defined but never connected

3 participants