Skip to content

Add structured output format validation helpers#386

Open
ECNUACRush wants to merge 2 commits into
gepa-ai:mainfrom
ECNUACRush:feat/issue-265-format-validation
Open

Add structured output format validation helpers#386
ECNUACRush wants to merge 2 commits into
gepa-ai:mainfrom
ECNUACRush:feat/issue-265-format-validation

Conversation

@ECNUACRush

Copy link
Copy Markdown

Summary

Fixes #265.

This PR adds evaluator wrappers for structured-output validation:

  • require_json_output
  • require_xml_output
  • require_regex_match
  • require_format

When validation fails, the wrapper returns score=0.0 and includes the parse or validation error in side_info, so GEPA's reflection step can learn from format failures.

This is related to #379/#384, but intentionally limited to evaluator-side format failure feedback. It does not implement rejected-proposal memory.

Tests

  • python -m pytest -q
  • python -m ruff check src/gepa/utils/format_validation.py src/gepa/utils/__init__.py tests/test_format_validation.py docs/docs/guides/faq.md
  • python -m pyright --pythonversion 3.12 src/gepa/utils/format_validation.py
  • mkdocs build -f docs/mkdocs.yml

@semanticdiff-com

semanticdiff-com Bot commented Jul 2, 2026

Copy link
Copy Markdown

Review changes with  SemanticDiff

Changed Files
File Status
  docs/docs/guides/faq.md Unsupported file format
  src/gepa/utils/__init__.py  0% smaller
  src/gepa/utils/format_validation.py  0% smaller
  tests/test_format_validation.py  0% smaller

@mvanhorn

mvanhorn commented Jul 2, 2026

Copy link
Copy Markdown

Heads up: I opened #387 for this same issue a few minutes after you filed this, without seeing yours first. Yours is more complete (XML support plus the FAQ docs), so I'm closing mine in favor of it.

One small edge case my tests covered that might be worth folding in: a require_format validator that raises an exception rather than returning False. My version catches that and treats it as a format failure (score 0 with the error in feedback) instead of letting it propagate out of the evaluator. Take it or leave it. Thanks for picking this up.

@ECNUACRush

Copy link
Copy Markdown
Author

Thanks for the heads up, and no worries. I appreciate you closing yours. @mvanhorn

Good catch on the require_format exception case. I’ll add handling for it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: format validation as first-class metric with automatic failure penalization

2 participants