Skip to content

feat: add format-validation decorators for evaluators#387

Closed
mvanhorn wants to merge 1 commit into
gepa-ai:mainfrom
mvanhorn:fix/265-format-validator-utility
Closed

feat: add format-validation decorators for evaluators#387
mvanhorn wants to merge 1 commit into
gepa-ai:mainfrom
mvanhorn:fix/265-format-validator-utility

Conversation

@mvanhorn

@mvanhorn mvanhorn commented Jul 2, 2026

Copy link
Copy Markdown

Summary

Evaluators had no reusable way to hard-fail malformed model output before scoring. Each evaluator re-implemented JSON parsing and format checks.

Changes

Add gepa.utils.format_validator with three decorators: require_json_output (parse JSON, optional shallow schema check, pass parsed value through), require_format (arbitrary predicate), and require_regex_match. On failure the wrapped evaluator is skipped and a (0.0, {feedback}) result is returned. Re-exported from gepa.utils.

Testing

Added tests/test_format_validator.py covering success, malformed JSON, schema mismatch, and regex paths.

Fixes #265

@semanticdiff-com

semanticdiff-com Bot commented Jul 2, 2026

Copy link
Copy Markdown

Review changes with  SemanticDiff

Changed Files
File Status
  src/gepa/utils/__init__.py  0% smaller
  src/gepa/utils/format_validator.py  0% smaller
  tests/test_format_validator.py  0% smaller

@mvanhorn

mvanhorn commented Jul 2, 2026

Copy link
Copy Markdown
Author

Closing in favor of #386, which was opened a few minutes earlier and covers the same feature more completely (adds require_xml_output and docs). Left a note there.

@mvanhorn mvanhorn closed this Jul 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: format validation as first-class metric with automatic failure penalization

1 participant