feat: example-level flip gate — P→F blocks promotion regardless of gains (Closes #1446) - #1449
Merged
Merged
Conversation
…ains (#1446) Child B of #1308. Child A (#1355) shipped the frozen probe-set data layer and nothing consumed it; the promotion decision was still "did the aggregate score go up?" — the exact question AgentDevel (arXiv:2601.04620) shows ships regressions, since an aggregate can rise while specific previously-working cases break. Their ablation: 14.8% regression rate with 4 bad releases without example-level flip gating, 3.1% with 0 with it. scripts/evolution_flip_gate.py classifies every probe into four cells — P->F regression, F->P fix, P->P held, F->F still broken — and blocks when P->F exceeds the threshold no matter how many F->P gains accompany it. Gains do not buy regressions; that is the entire point of the gate. Three decisions worth stating: - Default max_regressions is 0. One broken previously-working example is worth a human look. Callers needing slack ask for it explicitly. - Unrecognised grader output reads as FAIL, not pass. _passed accepts bools, status strings, and dicts with passed/pass/ok/success/status; anything else fails closed, because a result that reads as a pass turns a regression into a silent P->P. - A probe present before and absent after BLOCKS by default. The probe set is frozen by contract (#1355), so a missing probe means either the set was mutated or the run did not finish — both make the comparison unsound, and the missing probe may be the one that broke. Overridable with --allow-missing. CLI exits 0 promote / 1 block / 2 bad input, so a caller can gate on it directly. Verified end to end, not just via the unit tests. Deferred as the issue specifies: wiring the verdict into merge verification is Child C (#1447), version metadata is Child D (#1448). 26 tests covering the coercion table, all four cells, missing-probe handling, threshold behaviour and every CLI exit path; ruff clean. Closes #1446
This was referenced Jul 28, 2026
Closed
4 tasks
Lexus2016
added a commit
that referenced
this pull request
Jul 28, 2026
Child C of #1308. The flip-gate engine (#1446, PR #1449) classifies per-probe P→F regressions but nothing consumed the verdict — the merge gate would still self-merge a PR that broke previously-working probes. This wires the verdict into check_merge_policy_with_quality(): - check_flip_gate() runs evolution_flip_gate.compare_runs/flip_verdict and returns FLIP_GATE_BLOCK violations naming the regressed probes - A block stops the autonomous self-merge - --flip-gate-override REASON downgrades to a recorded warning (auditable) - Skills without probe sets are unaffected (opt-in per skill) - Fail-closed: missing probe data or unavailable module blocks the merge Success criteria met: ✓ P→F regression above threshold prevents the merge ✓ Blocking comment names specific probes that flipped ✓ Override is possible and leaves a recorded reason ✓ Skill without probe set merges as before Closes #1447 Co-authored-by: Hermes Evolution <evolution@hermes.ai>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1446 — Child B of #1308, consuming the frozen probe sets Child A (#1355) shipped.
Why
Child A gave the pipeline a probe-set data layer and nothing consumed it. The promotion decision was still "did the aggregate score go up?" — the exact question AgentDevel (arXiv:2601.04620) shows ships regressions, because an aggregate can rise while specific previously-working cases break.
Their ablation is the argument: 14.8% regression rate with 4 bad releases without example-level flip gating, 3.1% with 0 with it.
What it does
Every probe lands in one of four cells:
Promotion blocks when P→F exceeds the threshold no matter how many F→P gains sit beside it. Gains do not buy regressions — that is the whole gate.
Three decisions worth calling out
Default
max_regressionsis 0. One broken previously-working example is worth a human look. Slack must be asked for explicitly.Unrecognised grader output reads as FAIL, not pass.
_passedaccepts bools, status strings, and dicts withpassed/pass/ok/success/status; anything else fails closed. A result that read as a pass would turn a regression into a silent P→P — the failure this gate exists to catch.A probe present before and absent after BLOCKS by default. The probe set is frozen by contract (#1355), so a missing probe means the set was mutated or the run did not finish. Both make the comparison unsound, and the missing probe may be the one that broke. Overridable with
--allow-missing.Verified end to end
Exit 0 promote / 1 block / 2 bad input, so a caller can gate on it directly.
Deferred, as the issue specifies
Wiring the verdict into merge verification is Child C (#1447); version metadata on promoted skills is Child D (#1448). This slice only produces the table and the verdict.
26 tests — the coercion table, all four cells, missing-probe handling, threshold behaviour, every CLI exit path. Ruff clean.