Add a run_dir admission manifest for candidate acceptance/rejection audits

[sunghunkwag]

Summary

GEPA already exposes a lot of useful observability through callbacks, run_dir artifacts, and W&B/MLflow tables. I think one small missing artifact would make GEPA runs easier to audit and reproduce from local files alone:

a compact admission_manifest.jsonl record for each proposed candidate, accepted or rejected, keyed by candidate hash and parent hash.

This would be especially useful for prompt/code/agent-architecture optimization runs where the important question after the fact is not only "what was the best candidate?", but "why was this mutation admitted into the candidate pool, what evidence admitted it, and what evidence was only used for rejection/debugging?"

What I found in the current code/docs

• GEPA already has deterministic candidate hashing for evaluation-cache keys: src/gepa/core/state.py.
• run_dir already writes human-readable run_log.json and candidates.json: src/gepa/core/state.py.
• Accepted candidates are appended through update_state_with_new_program(...): src/gepa/core/state.py.
• Reflective proposal acceptance/rejection is centralized in _accept_reflective_proposal(...): src/gepa/core/engine.py.
• Rejected candidates currently emit CandidateRejectedEvent with old_score, new_score, and reason, but not the proposed candidate, parent candidate hash, subsample ids, proposal kind, or candidate hash: src/gepa/core/engine.py.
• Accepted candidates emit CandidateAcceptedEvent and then ValsetEvaluatedEvent, which includes validation scores and coverage: src/gepa/core/engine.py.
• The experiment tracking docs already document a proposals table for accepted/rejected LM calls, including status, candidate_idx, parent_ids, minibatch scores, prompt, raw LM output, and proposed text: docs/docs/guides/experiment-tracking.md.

So the requested change is not "add logging from scratch." It is a narrower, local, machine-readable admission ledger that joins the candidate/proposal/validation/budget facts that are currently spread across callbacks, state, and tracker tables.

Why this is different from related open issues

I saw several related issues and tried to avoid duplicating them:

• GEPA logs #172 asks for a broader detailed logging data model. This proposal is a smaller audit artifact, one JSONL record per proposal decision.
• Persistent opt_state #227 is about persistent adapter/opt state. This proposal does not require making rejected candidates part of optimizer state.
• feat: built-in anti-overfit guard to prevent prompt copying training data #261 is about preventing prompt copying/overfit. This proposal records the evidence boundary around admission; it does not implement an anti-overfit policy.
• feat: OnlineEvaluationPolicy — test-before-train windowed evaluation #270 is about online test-before-train evaluation. This proposal should work with any EvaluationPolicy by recording which val ids were evaluated and which policy was used.
• Cached eval still counts toward metric budget #316 is about cached evals counting toward budget. This proposal would make cache/metric-call accounting auditable, but does not itself change the accounting.
• Per-proposal reflection LLM cost tracking #331 is per-proposal reflection LM cost tracking. This proposal can include that field if available, but is about candidate admission evidence.
• GEPA doesn't remember rejected proposals — re-sampling the same parent repeats the same failed mutation #379 is about remembering rejected proposals and feeding them back into reflection. This proposal can support that later, but it is observational: rejected proposals remain rejected unless another feature chooses to use them.

Proposed artifact

When run_dir is set, GEPA could append records like this to admission_manifest.jsonl:

{
  "schema_version": 1,
  "iteration": 12,
  "proposal_kind": "reflective_mutation",
  "decision": "accepted",
  "decision_reason": "strict_improvement: new_sum > old_sum",
  "candidate_idx": 5,
  "candidate_hash": "sha256:...",
  "parent_candidate_ids": [2],
  "parent_candidate_hashes": ["sha256:..."],
  "components_updated": ["instructions"],
  "subsample_ids": ["train_17", "train_42", "train_81"],
  "subsample_score_before": 1.2,
  "subsample_score_after": 2.0,
  "val_evaluation_policy": "FullEvaluationPolicy",
  "evaluated_val_ids": ["val_0", "val_1", "val_2"],
  "val_average_score": 0.73,
  "is_best_program": true,
  "metric_calls_before_decision": 120,
  "metric_calls_after_decision": 146,
  "metric_calls_charged": 26,
  "cache_hits": null,
  "cache_misses": null,
  "tracker_row_ref": {
    "table": "proposals",
    "candidate_idx": 5
  }
}

For rejected proposals:

{
  "schema_version": 1,
  "iteration": 13,
  "proposal_kind": "reflective_mutation",
  "decision": "rejected",
  "decision_reason": "strict_improvement: new_sum <= old_sum",
  "candidate_idx": null,
  "candidate_hash": "sha256:...",
  "parent_candidate_ids": [2],
  "parent_candidate_hashes": ["sha256:..."],
  "components_updated": ["instructions"],
  "subsample_ids": ["train_5", "train_8", "train_13"],
  "subsample_score_before": 2.0,
  "subsample_score_after": 1.3,
  "val_evaluation_policy": null,
  "evaluated_val_ids": [],
  "val_average_score": null,
  "is_best_program": false,
  "metric_calls_before_decision": 146,
  "metric_calls_after_decision": 152,
  "metric_calls_charged": 6
}

The full prompt/raw LM output can remain in the existing experiment tracker table. The manifest can store hashes or tracker references by default to avoid making a local audit file unexpectedly sensitive.

Possible implementation path

1. Expose the existing candidate hash helper or add a small public/internal helper, rather than duplicating hash logic outside state.py.
2. Add a lightweight AdmissionRecord helper or callback event around _accept_reflective_proposal(...).
3. Write JSONL atomically under run_dir when a proposal decision is made.
4. Include merge decisions as the same schema with proposal_kind: "merge" and two parent hashes.
5. Keep this opt-in if there is concern about local artifacts, e.g. EngineConfig(write_admission_manifest=True) or enabled only when run_dir is set.

Suggested tests

• A synthetic reflective proposal that is accepted writes one manifest line with non-null candidate_idx, candidate hash, parent hash, minibatch ids, and evaluated val ids.
• A synthetic reflective proposal that is rejected writes one manifest line with candidate_idx: null and is not present in candidates.json.
• A merge accept/reject path writes the same schema with two parents.
• A custom partial EvaluationPolicy records partial evaluated_val_ids and total_valset_size/coverage.
• With cache_evaluation=True, the manifest records actual charged metric calls separately from candidate/example pairs served from cache, if that accounting is available.

External reference

I have been using a small bounded verifier harness here: https://github.com/sunghunkwag/rsi-metaforge-core

The relevant pattern is not the whole harness, but the evidence discipline around hidden evals, train-fit rejection, rollback-sensitive admission, and explicit claim boundaries:

• README framing: https://github.com/sunghunkwag/rsi-metaforge-core#readme
• Evidence boundary: https://github.com/sunghunkwag/rsi-metaforge-core/blob/main/EVIDENCE.md

That repo is not a GEPA replacement and not an unrestricted self-improvement claim. It is just a useful reference for why an admission/rollback ledger becomes valuable once candidate-generating systems start optimizing prompts, code, or agent architectures under evaluator gates.

[OtotaO]

+1 on this. The part that makes it compelling is that the record is replayable for free: every field
can be reconstructed from facts GEPA already holds, so a verifier can recompute the manifest and diff
it against what was written without rerunning the optimizer.

Concretely, the verify side is just:

def verify_record(record, candidate, parents):
    assert record["candidate_hash"] == _candidate_hash(candidate)
    assert record["parent_candidate_hashes"] == [_candidate_hash(p) for p in parents]
    # subsample + decision fields are exact replays of what the engine had in hand

I went and traced the seams rather than guess, so here's the plan I'd implement. Corrections very
welcome where I've misread something.

Reflective accept/reject. The decision lives in GEPAEngine._accept_reflective_proposal: the
boolean is self.acceptance_criterion.should_accept(proposal, state), and at that point the proposal
(a CandidateProposal) plus state give me everything for the identity/lineage half of the record
without any extra work. candidate_hash comes from the module-level
_candidate_hash(proposal.candidate) in state.py, parent_candidate_hashes from
_candidate_hash(state.program_candidates[i]) over proposal.parent_program_ids, the subsample scores
from the subsample_scores_before/after the method already reduces to old_sum/new_sum,
proposal_kind from proposal.tag, and the reject reason on the reject branch.

The one boundary I want to flag honestly: the val-level fields (val_average_score,
evaluated_val_ids, is_best_program, the assigned candidate index) are not in scope at the
decision. They're produced later inside _run_full_eval_and_add (the _evaluate_on_valset →
update_state_with_new_program → ValsetEvaluatedEvent path), which only runs on the accept branch.
So the record is naturally two-phase: write identity/lineage/subsample/decision at the seam, then
backfill the val fields at the full-val step (and on a reject they stay pending, with candidate_idx == -1, matching what the existing proposals table already does).

components_updated is the one field with no first-class home. I'd recover it by parsing the
prompt:/raw_lm_output: keys off proposal.metadata, the same way _log_proposal_lm_calls does
today, unless you'd rather I add a dedicated field upstream.

Merges. You're right that they're handled inline in run(): there's no _accept_merge_proposal
analog, the call is just new_sum >= max(parent_sums) under the tag == "merge" guard. The nice thing
is both parents are fully reachable there. proposal.parent_program_ids == [id1, id2] index into
state.program_candidates, so I can hash both parent dicts and the merged proposal.candidate right
at the site. Reflective and merge both funnel their accepts through _run_full_eval_and_add +
CandidateAcceptedEvent, so for accepted records what I'd do is stamp them from a small shared helper
at that common chokepoint (threading proposal_kind + parent hashes through), rather than duplicate
the write. That keeps the two paths symmetric and stops the manifest drifting between them. The
reject records don't share a seam (reflective rejects at the CandidateRejectedEvent branch, merges
at their own inline < max(parent_sums) branch), so I'd write those at each decision point. If you'd
rather it all open-coded to mirror the current inline style, easy to flip.

Budget fields. total_metric_calls / per-discovery cost / num_full_val_evals all come straight
off state.total_num_evals and friends (already surfaced in GEPAResult), so those are real.
cache_hits/cache_misses I'd leave null: as far as I can tell EvaluationCache keeps only _cache
with no cumulative hit/miss counter, so there's nothing to read without adding counters upstream (happy
to PR those separately if you want them).

For persistence I'd match GEPAState._atomic_write_json (tmp-write + os.replace, json.dump(..., indent=2, default=json_default)), the same helper save() uses for run_log.json /
candidates.json, so the manifest lands with the same atomic-write guarantees.

One thing that's genuinely your call: the opt-in surface. My default would be a
write_admission_manifest: bool = False field on EngineConfig (mirrored as a flat kwarg in
api.py's optimize(), since the two entry points are parallel), which means a small third edit to
add the param to GEPAEngine.__init__ and forward it from both build sites. But if you'd rather just
infer it from run_dir being set and avoid a new flag entirely, that's an equally clean surface. Tell
me which you prefer and I'll wire it that way.

Separately, and very much subordinate to the above: is tamper-evidence of the manifest itself in
scope for you, or out? The hashes already make each record self-checking against the candidate it
names, but they don't chain records together or carry a signature an external reviewer could check
offline. If that's ever interesting I can sketch it: I maintain a small signed-receipt library
(SUM) built on exactly that shape (Ed25519 over JCS-canonical bytes,
detached JWS), so it'd be a thin, opt-in add rather than new infrastructure. Otherwise I'll happily
leave it out and keep this a purely replayable audit log.

[sunghunkwag]

Thanks for tracing this against the current code. Your read of the reflective and merge seams matches mine, and "replayable for free" is the core property I was aiming for.

For a v1, my preferences would be:

• Make it explicitly opt-in: write_admission_manifest: bool = False in EngineConfig, mirrored in api.py::optimize(), and require run_dir when enabled. I would not infer it from run_dir alone because existing users should not start emitting a potentially sensitive artifact unexpectedly.
• Keep the manifest as JSONL, but do not backfill a line in place. Build the decision snapshot at _accept_reflective_proposal; for an accepted proposal, finalize the validation fields after _run_full_eval_and_add and append the complete record once. Rejects can append once immediately with validation fields null/empty and candidate_idx: null. The shared finalization helper for reflective and merge accepts sounds right.
• Make the phase explicit: the recorded decision is the subsample acceptance decision; validation fields are post-decision enrichment, not a second admission gate.
• I would derive a deterministic components_changed from the candidate-versus-parent data rather than parse prompt: / raw_lm_output: metadata. Metadata may be absent for adapters or custom proposers, while a candidate diff remains replayable. If ?components selected for mutation? is also needed, that should be a separate optional first-class field.
• Leave cache hit/miss counters null or omit them in v1. Adding counters can be a separate change.
• Use a dedicated one-record JSONL append helper rather than _atomic_write_json, which atomically replaces a whole JSON document. Serialize one complete line, append and flush it, and test handling of a truncated final line.
• A minimal replay verifier/test that recomputes candidate and parent hashes and checks the recorded decision inputs would be valuable.

I checked SUM; the Ed25519/JCS/detached-JWS design is relevant and well aligned with this kind of receipt. However, I would keep signatures, record chaining, and tamper-evidence out of #381 and treat them as a separate opt-in follow-up after the base schema is stable.

[OtotaO]

This is a great spec, thanks. I'll open a PR that implements exactly this:

• write_admission_manifest: bool = False on EngineConfig, mirrored in optimize(), and required
  run_dir when enabled. Agreed on not inferring it from run_dir alone, existing runs shouldn't start
  emitting a new artifact unexpectedly.
• Append-once, no in-place backfill. Build the decision snapshot at _accept_reflective_proposal; for
  an accepted proposal, finalize the validation fields after _run_full_eval_and_add and append the
  complete record once; rejects append immediately with validation fields null/empty and candidate_idx: null. One shared finalization helper across the reflective and merge accept paths.
• Phase made explicit in the schema: the recorded decision is the subsample acceptance decision, and
  the validation fields are post-decision enrichment, not a second admission gate.
• components_changed derived from a candidate-versus-parent diff rather than parsing
  prompt:/raw_lm_output: metadata. You're right that metadata can be absent for custom
  adapters/proposers, while a diff stays replayable. If "components selected for mutation" is also wanted
  later, I'll add it as a separate optional first-class field.
• cache hit/miss left null/omitted in v1; counters can be their own change.
• A dedicated single-record JSONL append helper (serialize one complete line, append and flush) rather
  than _atomic_write_json, plus a test that a truncated final line is handled gracefully.
• A minimal replay test that recomputes the candidate and parent hashes and re-checks the recorded
  decision inputs.

Agreed on keeping signatures, record chaining, and tamper-evidence out of this PR. I'll keep the base
schema clean and we can treat that as a separate opt-in follow-up once it's stable. I'll get a draft up
and tag you.

[aranya-chatterjee]

is this still up for grabs or has @Ototoo already opened a PR?

[OtotaO]

@aranya-chatterjee yep, the PR is up: #382. It implements the full v1 spec from this thread (opt-in
write_admission_manifest, append-once JSONL with no in-place backfill, one shared finalize helper
across the reflective and merge accept paths, components_changed from a candidate-vs-parent diff, a
dedicated single-record append helper, and a replay verifier test). It's marked ready for review and CI
is green, so I'm just waiting on @sunghunkwag to take a look.

Happy to have another set of eyes on it if you're interested.