[IMPROVEMENT+REPLACEMENT] Adopt hidden-verifier + continuous partial-credit replay grading (consolidates LHTB + Senior-SWE-Bench)

[Lexus2016]

Problem statement

This issue consolidates two related research signals (one [IMPROVEMENT], one [REPLACEMENT]) that converge on the same mechanism change. Filed together to avoid two issues on the same work.

Two independent sources this cycle converge on one conclusion: binary pass/fail grading destroys ranking signal and hides the dominant agent failure mode.

• LHTB (Long-Horizon Terminal-Bench): across 782 runs of 18 frontier models (best = Grok 4.5 at 0.51 avg, 13/46 solved), 79% of unresolved runs time out while still actively making progress — the dominant failure is state-loss disguised as productivity, not capability. Binary pass/fail collapses the entire model ranking to noise (ten of fifteen models solve zero), but under LHTB's continuous partial-credit banding (graded reward 0→1, "solved" only at ≥0.95) >50% of runs land in a 0–0.25 band that does rank models meaningfully.
• Senior-SWE-Bench (Snorkel/Princeton/UW-Madison, Jul 16): top models (Claude Opus 4.8) only "tastefully" solve 24% of senior tasks. Under binary scoring that reads as "agent fails"; under partial-credit it reads as "agent does 60% of a senior task" — the actually useful signal for prioritization.

Hermes's cron-driven evolution loop already records trajectories but grades pass/fail. Switching to replayed-artifact, partial-credit grading would expose the exact "dies-still-working" failure class that LHTB proves dominates real agent failure, and would change the signal that feeds the entire downstream funnel — every issue, analysis, and merge decision is downstream of how runs are scored.

This is the single highest-leverage replacement on this cycle.

Proposed solution (two layers)

Layer A — posture/discipline (zero-cost, [IMPROVEMENT]):

1. Hide the verifiers from the agent — run grading in a separate eval harness the agent process cannot cat; grade from artifacts on replay.
2. Never trust agent-reported "done" — always re-execute the produced artifact and score fraction-of-checks-passed.

Layer B — mechanism change ([REPLACEMENT]):
3. Replace the "did the cron/eval pass? Y/N" path with a graded-replay grader in the evolution pipeline that emits score ∈ [0,1] per task, not just pass|fail.
4. Adoption of LHTB-style continuous partial-credit banding: graded reward 0→1, "solved" threshold at ≥0.95.

Acceptance criteria

• Verifiers/grading harness are not readable by the agent during a run (hidden verifier).
• Runs are re-graded from replayed artifacts, not from agent self-report.
• The grader emits a continuous score ∈ [0,1] per task (not boolean), stored alongside the existing pass/fail flag for backward compat.
• At least one historical run re-scored under the new grader shows a non-trivial partial-credit distribution (i.e., the banding actually ranks where pass/fail did not).
• Downstream consumers (issue generation, analysis) can consume the continuous score.

Sources

• LHTB: https://zli12321.github.io/LHTB (arXiv:2607.08964) — [IMPROVEMENT] signal, priority 1.44 (High impact, Low effort)
• Senior-SWE-Bench: https://snorkel.ai/blog/senior-swe-bench-evaluating-coding-agents-like-senior-engineers — [REPLACEMENT] signal, priority 1.28 (High impact, Medium effort)
• Frontier standing: behind

[Lexus2016]

Deferred at implementation (2026-07-26): larger than one cycle — needs decomposition.

This issue is still wanted (highest strategic value in the cycle), but on inspection it is a multi-layer mechanism change (compat=0.5, safety=0.5) that does not fit a single autonomous ≤200-line slice:

• Layer A (posture): hidden-verifier discipline + replay-from-artifacts.
• Layer B (mechanism): graded-replay grader emitting score ∈ [0,1], LHTB-style banding (solved @ ≥0.95), and downstream consumer wiring (issue generation + analysis must consume the continuous score).

Existing infrastructure that a first slice can build on: scripts/evolution_trajectory_logger.py and scripts/evolution_trajectory_store.py already capture trajectories. The gap is the grader + score schema + consumer integration.

Proposed decomposition (each ≤0.6 effort, ≤200 lines):

1. Slice 1 — score schema + trajectory-store field: add a continuous score ∈ [0,1] field to the trajectory store (backward-compat: default to 1.0 if pass else 0.0), with a migration note. No grader yet — just the data slot + a stub reader. (~1h)
2. Slice 2 — graded-replay grader skeleton: a new scripts/evolution_partial_credit_grader.py that re-opens a stored trajectory and emits a fractional score from a per-task check contract (fraction-of-checks-passed). Wire it behind a flag so pass/fail stays the default. (~3h)
3. Slice 3 — hidden-verifier posture: move the check contract out of any agent-readable path; grade strictly from replayed artifacts. (~2h)
4. Slice 4 — downstream consumer: teach issue-generation/analysis to read the continuous score and rank accordingly. (~2h)

Marking needs-split; a future cycle can pick up Slice 1.

[Lexus2016]

Cross-reference — five open issues now target the promotion/verification decision from different angles. Reviewed together during today's backlog pass; they are not duplicates, but implementing them independently would produce overlapping gates over the same decision point.

Issue	What it governs
#1446 (Child B of #1308)	the mechanism — per-example P→F / F→P table + block verdict. Implemented, PR #1449.
#1447 (Child C of #1308)	where it binds — wiring the verdict into merge verification, plus the implementation-blind critic
#1441	the statistic — per-task binomial test with baseline-regression veto and pooled baseline
#1440	the hierarchy — which signal to trust when they disagree (formal verifiers > execution feedback > learned judges > intrinsic)
#1444	the audit — post-hoc protocol-validity check that the benchmark itself was not gamed
#1302	the scoring — hidden verifier + continuous partial credit feeding all of the above

Suggested sequencing, since each layer needs the one below it to mean anything:

1. [INCREMENT] Flip-gate engine: per-example P→F / F→P table + block verdict (Child B of #1308) #1446 → [INCREMENT] Wire the flip gate into merge verification + implementation-blind critic (Child C of #1308) #1447 — a gate that blocks. Without a binding decision point the rest is measurement with no consequence.
2. [IMPROVEMENT] Promotion gate: per-task binomial test with baseline-regression veto and pooled baseline #1441 — makes the block statistically sound. [INCREMENT] Flip-gate engine: per-example P→F / F→P table + block verdict (Child B of #1308) #1446 blocks on a raw count with a threshold; a binomial test against a pooled baseline is what distinguishes a real regression from sampling noise, and it slots directly into flip_verdict.
3. [IMPROVEMENT+REPLACEMENT] Adopt hidden-verifier + continuous partial-credit replay grading (consolidates LHTB + Senior-SWE-Bench) #1302 — improves the inputs. Both [INCREMENT] Flip-gate engine: per-example P→F / F→P table + block verdict (Child B of #1308) #1446 and [IMPROVEMENT] Promotion gate: per-task binomial test with baseline-regression veto and pooled baseline #1441 consume pass/fail per example; partial credit and a hidden verifier make those inputs harder to game and more informative.
4. [IMPROVEMENT] Verification hierarchy for self-improvement loops: formal verifiers > execution feedback > learned judges > intrinsic signals #1440 — arbitrates disagreement. Only meaningful once there is more than one signal feeding the decision.
5. [IMPROVEMENT] HackDetect: post-hoc protocol-validity audit for evolution pipeline benchmarks #1444 — audits the whole thing after the fact. Needs a stable protocol to audit.

One caution for whoever picks these up: #1441 and #1446 both own part of flip_verdict. #1441 should replace its threshold arithmetic rather than wrap it, or the pipeline ends up with two disagreeing definitions of "regression".