Feat: Add held-out set evaluation for final candidate selection

[UgurKap]

Summary

This PR adds held-out-set support to gepa.optimize() so final candidate selection can happen on a split that did not drive candidate survival during optimization.

The motivation is that valset already participates in the optimization loop through Pareto tracking and candidate selection. Re-using that same split for final selection can bias selection toward candidates that survived search pressure on the Pareto/validation set, even if they generalize worse on unseen data.

Changes

• add held_out support to gepa.optimize()
• add HeldOutSetEvaluationPolicy
• evaluate the current valset leader on the held-out set
• select the final best candidate by held-out score when configured
• track held-out scores and held-out evaluation counts in GEPAResult
• add held-out callback/event coverage
• add tests for held-out evaluation flow, callbacks, persistence, warnings, logging semantics, and result semantics
• add documentation for the held-out selection policy
• Unrelated, but was a bug: fix GEPAResult JSON round-trip deserialization for integer-keyed maps such as held_out_scores (i.e. reading the candidate_ids back from the disk turns them into strings)

Key Decisions / Assumptions

• scope is limited to gepa.optimize(); optimize_anything was left unchanged
• if held_out is provided and val_evaluation_policy is None, GEPA auto-selects HeldOutSetEvaluationPolicy
• "full_eval" still means FullEvaluationPolicy()
• if "full_eval" or FullEvaluationPolicy() is used together with held_out, GEPA raises an error
• "heldout_eval" is available as an explicit held-out-aware option
• "heldout_eval" without a held_out split is allowed but emits a warning and falls back to valset-only semantics
• best_score_on_valset remains the true best valset score
• policy_selected_valset_score is logged only when held_out is configured
• best_score_on_held_out is the held-out score of the policy-selected candidate
• aggregate-valset-best reporting is preserved separately from policy-selected-best reporting
• valset callback semantics remain valset-focused even under held-out selection

Validation

• uv run pytest tests/test_held_out_eval_policy.py
• uv run pyright src/
• smoke-tested full_eval and heldout_eval behavior locally on a short AIME slice

[semanticdiff-com]

Review changes with  

Changed Files

File	Status
src/gepa/core/engine.py	29% smaller
src/gepa/api.py	27% smaller
src/gepa/core/result.py	16% smaller
src/gepa/logging/utils.py	9% smaller
docs/docs/api/strategies/HeldOutSetEvaluationPolicy.md	Unsupported file format
docs/docs/guides/experiment-tracking.md	Unsupported file format
docs/docs/guides/faq.md	Unsupported file format
docs/docs/guides/held-out-selection.md	Unsupported file format
docs/docs/guides/index.md	Unsupported file format
docs/docs/guides/quickstart.md	Unsupported file format
docs/mkdocs.yml	0% smaller
src/gepa/core/callbacks.py	0% smaller
src/gepa/core/state.py	0% smaller
src/gepa/proposer/reflective_mutation/reflective_mutation.py	0% smaller
src/gepa/strategies/eval_policy.py	0% smaller
tests/test_callbacks.py	0% smaller
tests/test_candidate_selector.py	0% smaller
tests/test_held_out_eval_policy.py	0% smaller

[Shangyint]

Thanks for the PR. However, this does not serve a good purpose for the following reasons:

1. This is already achievable currently via callbacks
2. Valset is unseen data in the way that the reflection proposer will not see any of them, unless we set trainset == valset
3. Empirically, not using another held-out set works well for multiple domains; this PR could be doing premature optimization (that also at a large cost)

[UgurKap]

Thanks for the PR. However, this does not serve a good purpose for the following reasons:

1. This is already achievable currently via callbacks

2. Valset _is_ unseen data in the way that the reflection proposer will not see any of them, unless we set `trainset == valset`

3. Empirically, not using another held-out set works well for multiple domains; this PR could be doing premature optimization (that also at a large cost)

Hello @Shangyint , thank you for the comment. Sorry, about not putting in more context. I had previously talked with @LakshyAAAgrawal , and showed why this approach could be good here: https://uguralikaplan.com/blog/gepa-is-better-than-you-think/ (Essentially showing pareto set cannot measure generalization capability at least for AIME-2026). I didn't consider if this would be possible with the callbacks though.