Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 10 additions & 11 deletions docs/docs/guides/faq.md
Original file line number Diff line number Diff line change
Expand Up @@ -254,26 +254,25 @@ result = optimize_anything(..., config=config)

Yes, and this is one of GEPA's strongest use cases for smaller models. Dropbox reduced gemma-3-12b's malformed JSON rate from **40% to under 3%** while simultaneously improving relevance quality, by optimizing the prompt to enforce structured output compliance.

The key is to **penalize format failures in your metric** so GEPA learns this is a hard constraint:
The key is to **penalize format failures in your metric** so GEPA learns this is a hard constraint. GEPA includes evaluator wrappers for common structured-output formats:

```python
from gepa.utils import require_json_output


@require_json_output(schema={"score": int, "reasoning": str})
def evaluator(data, response):
import json
# Hard penalty for format violations
try:
parsed = json.loads(response)
except json.JSONDecodeError:
return 0.0, {"Output": response, "Error": "Malformed JSON — failed to parse"}

score = compute_quality_score(parsed, data)
# response is already parsed JSON here. If parsing or schema validation
# fails, GEPA returns score=0.0 with the parse error in side_info.
score = compute_quality_score(response, data)
return score, {
"Output": parsed,
"Output": response,
"Expected": data["expected"],
"Score": score,
}
```

Including the parse error in `side_info` gives GEPA's reflection LM the signal it needs to add explicit formatting instructions to the prompt.
`require_json_output` automatically returns `score=0.0` with the malformed output and validation error in `side_info`. The parse error gives GEPA's reflection LM the signal it needs to add explicit formatting instructions to the prompt. For non-JSON formats, use `require_regex_match`, `require_xml_output`, or `require_format` with a custom validator.

### Does GEPA support async optimization?

Expand Down
17 changes: 17 additions & 0 deletions src/gepa/utils/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,10 @@

**Stdio capture** — thread-safe stdout/stderr capture during evaluation:
``StreamCaptureManager``, ``ThreadLocalStreamCapture``.

**Format validation** — evaluator wrappers for structured outputs:
``require_json_output``, ``require_xml_output``, ``require_regex_match``,
``require_format``.
"""

# Code execution utilities for fitness functions that evaluate generated code
Expand All @@ -20,6 +24,13 @@
execute_code,
get_code_hash,
)
from .format_validation import (
FormatValidationError,
require_format,
require_json_output,
require_regex_match,
require_xml_output,
)
from .stdio_capture import (
StreamCaptureManager,
ThreadLocalStreamCapture,
Expand Down Expand Up @@ -56,6 +67,12 @@
"TimeLimitError",
"execute_code",
"get_code_hash",
# Format validation utilities
"FormatValidationError",
"require_format",
"require_json_output",
"require_regex_match",
"require_xml_output",
# Stdio capture utilities
"StreamCaptureManager",
"ThreadLocalStreamCapture",
Expand Down
Loading
Loading