Add opt-in admission manifest: per-decision JSONL audit ledger (#381)

[OtotaO]

Implements the admission_manifest.jsonl proposed in #381: one JSON line per candidate-admission decision (accepted or rejected), keyed by candidate and parent hashes, so a run can be audited and reproduced from local files alone. Every field is derivable from facts GEPA already holds, so the manifest is replayable for free. Opt-in, off by default, zero-cost when off.

Follows the design we converged on in the issue thread.

What it does

• Opt-in, requires run_dir. write_admission_manifest: bool = False on EngineConfig, mirrored as a flat kwarg in optimize(); raises if run_dir is unset, so existing runs never start emitting a new artifact unexpectedly.
• Append-once, no in-place backfill. The decision snapshot is built at the subsample-acceptance seam (_accept_reflective_proposal); for an accepted candidate the validation fields are finalized after _run_full_eval_and_add and the complete record is appended once; rejects append immediately with validation: null and candidate_idx: null. One shared finalization helper covers the reflective and merge accept paths.
• Explicit phase. The recorded decision is the subsample acceptance decision (decision_basis: "subsample_acceptance"); the validation block is post-decision enrichment, not a second admission gate.
• components_changed from a candidate-versus-parent diff, not metadata parsing (metadata can be absent for custom adapters/proposers, while a diff stays replayable).
• Dedicated single-record JSONL append helper (serialize one complete line, append, flush), not _atomic_write_json. read_manifest() tolerates a truncated final line from an interrupted append.
• cache hit/miss left null in v1; signatures, record chaining, and tamper-evidence are intentionally out of scope as a possible separate opt-in follow-up.

verify_record() recomputes the candidate and parent hashes and re-derives the decision from the recorded subsample scores.

Example record (real output from the test run)

An accepted reflective decision:

{
  "schema_version": 1,
  "iteration": 1,
  "proposal_kind": "reflective_mutation",
  "decision": "accepted",
  "decision_reason": "subsample score 1.0 > 0.0 (accepted by StrictImprovementAcceptance)",
  "decision_basis": "subsample_acceptance",
  "candidate_hash": "f554b8d7...098adf",
  "candidate_idx": 1,
  "parent_candidate_ids": [0],
  "parent_candidate_hashes": ["0abb9c2d...8bb4a1"],
  "components_changed": ["current_candidate"],
  "subsample_ids": [0],
  "subsample_score_before": 0.0,
  "subsample_score_after": 1.0,
  "metric_calls_before_decision": 3,
  "validation": {
    "val_average_score": 1.0,
    "evaluated_val_ids": [0],
    "is_best_program": true,
    "metric_calls_after_validation": 4,
    "metric_calls_charged_validation": 1
  },
  "cache_hits": null,
  "cache_misses": null
}

A rejected decision is the same shape with decision: "rejected", candidate_idx: null, and validation: null.

Files

• src/gepa/logging/admission_manifest.py (new): the schema builder, the append and read helpers, and verify_record.
• src/gepa/core/engine.py: emit at the reflective and merge accept/reject seams via one shared finalization helper; the opt-in flag on __init__.
• src/gepa/optimize_anything.py, src/gepa/api.py: the EngineConfig field and the optimize() flat kwarg.
• tests/test_admission_manifest.py (new, 16 tests).

Tests and checks

• 16 new tests: accepted / rejected / merge record shapes, the truncated-final-line case, replay verification, and an end-to-end optimize_anything run whose accepted records replay against candidates.json.
• Full pytest tests/ passes locally under the dev extra; pyright is clean; no new ruff findings; no new dependencies (the module uses only the standard library and existing GEPA internals).
• Off by default, so existing behavior and cost are unchanged.

Open questions for you

• Opt-in surface: I went with the EngineConfig field mirrored in optimize(), per your preference over inferring from run_dir.
• Module placement: I put it under gepa/logging/; happy to move it (for example to core/) if you would rather it sit with the other run_dir artifacts.

Opening as a draft for your review. @sunghunkwag

Addresses #381.

[semanticdiff-com]

Review changes with  

Changed Files

File	Status
src/gepa/api.py	0% smaller
src/gepa/core/engine.py	0% smaller
src/gepa/logging/admission_manifest.py	0% smaller
src/gepa/optimize_anything.py	0% smaller
tests/test_admission_manifest.py	0% smaller

[sunghunkwag]

Thanks, this matches the v1 shape well. I checked the diff and the focused tests.

One small replayability issue I found: with acceptance_criterion="improvement_or_equal", an equal-score acceptance writes a reason like subsample score 0.0 > 0.0, even though the actual rule is >=. Also, verify_record(record, candidate, parents) defaults to strict improvement and returns false for that record unless the caller separately passes acceptance="improvement_or_equal", while the manifest record itself does not store the acceptance strategy.

Since the feature is meant to be replayable from local artifacts, I think the manifest should record the acceptance criterion/operator, and the decision reason should use the actual operator. Minor doc nit: api.optimize(..., write_admission_manifest=...) should also be documented in the parameter list.

Otherwise this looks aligned with the v1 spec from #381.

[OtotaO]

Good catch, and you are right that this is a real replayability gap, not just a cosmetic reason string. Both halves trace to the same root: the record did not carry the rule that produced the decision, so a verifier could not re-derive an equal-score acceptance without being told the strategy out of band. Fixed in the latest push.

What changed:

1. Every record now stores two new fields: acceptance_criterion (the public name, matching optimize(acceptance_criterion=...): "strict_improvement", "improvement_or_equal", or "merge_gate" for the merge path) and acceptance_operator (">" or ">=", or null for a custom criterion). I record the literal operator alongside the name on purpose, so a third party can replay from the JSONL alone without knowing GEPA's class names or which strategy the run used.

2. verify_record(record, candidate, parents) now reads acceptance_operator from the record and re-derives the decision with it. No acceptance= argument needed. The recorded operator is authoritative; the acceptance= kwarg stays only as an override for records written before the field existed. A custom criterion records a null operator, so its score decision is treated as out of scope for replay while hashes and components_changed are still verified.

3. The reflective-accept reason now uses the criterion's actual operator instead of a hardcoded >, so an equal-score acceptance under improvement_or_equal reads subsample score 0.0 >= 0.0 (accepted by improvement_or_equal). A custom criterion gets a neutral phrasing rather than a misleading comparison.

4. Added write_admission_manifest to the optimize() parameter docstring.

Tests: added the exact case you flagged (equal-score accept under improvement_or_equal replaying True with no override), plus the record-is-authoritative-over-a-stale-override case, the custom-criterion null-operator case, and an end-to-end improvement_or_equal run asserting >= is recorded. Full tests/ shows no new failures versus the branch baseline.

One note: I kept schema_version at 1 since this schema has not shipped yet, so the two fields are an additive part of v1 rather than a new version. Happy to bump it if you would rather treat each pre-merge schema edit as a version.

Open questions from the PR description still stand (the opt-in surface, and gepa/logging/ vs gepa/core/ placement). Let me know if you would like the criterion recorded as just the operator, just the name, or both as it is now.

[sunghunkwag]

Confirmed on the latest head. The equal-score improvement_or_equal case now records acceptance_criterion="improvement_or_equal" and acceptance_operator=">=", the reason string uses >=, and verify_record(...) replays it successfully without an out-of-band override.

I think keeping both the public criterion name and the literal operator is the right shape: the name preserves semantic context, while the operator keeps replay independent of GEPA class names. Keeping schema_version=1 also seems fine to me since this has not shipped yet.

From the #381 perspective, this resolves my replayability concern. I only see the latest GitHub Actions runs as action_required/not reported on the PR, so once those are approved and green this looks good to me.

[OtotaO]

Thanks, glad it lines up on your end too. Agreed on all three points: keeping both the public criterion name and the literal operator gives the semantic context plus replay that does not depend on GEPA's class names, and holding schema_version at 1 makes sense while this is pre-merge.

On CI: the runs are now green across every job (Build Package 3.10 through 3.14, Run Tests 3.10 through 3.14, Check Ruff Fix, and Type Check), so that gate looks cleared from my side too. Nothing pending on my end. Happy to rebase or adjust if anything comes up before you merge.

Thanks again for the careful review.

[OtotaO]

Hi @LakshyAAAgrawal, flagging this one for a merge look whenever you have a moment.

Where it stands:

• It implements Add a run_dir admission manifest for candidate acceptance/rejection audits #381 (sunghunkwag's request for a replayable run_dir admission manifest): an opt-in, per-decision JSONL ledger of candidate accept/reject decisions, with verify_record() so each decision replays from the artifact alone.
• sunghunkwag reviewed it, found a real replayability gap (an equal-score acceptance under improvement_or_equal was recording the wrong operator), I pushed a fix that records acceptance_criterion and acceptance_operator on every record, and they confirmed on the latest head that it resolves their concern.
• CI is green across every test, lint, and build job (Build Package and Run Tests on 3.10 through 3.14, Check Ruff Fix, and Type Check; the docs deploy job is skipped as expected).
• I kept schema_version at 1 since the schema has not shipped, so the two new fields are additive within v1 rather than a new version. Happy to bump it if you would rather version each pre-merge edit.

Two open questions from the PR description are still yours to call: the opt-in surface, and gepa/logging/ vs gepa/core/ placement. Glad to rebase or adjust either before merge. No rush, I just did not want it to go stale.

Thanks for the library, it has been a pleasure to build on.