ooo run: ~40% of executions fail on verifier false negatives and environment faults — raise run success to ≥80%

[Q00]

Measurement (local event store, 2026-07-01 → 08-04, ~/.ouroboros/ouroboros.db)

• Per execution: 51 completed / 35 failed (59%).
• Per AC session, 181 failures by cause:

Cause	Count	Share
Fat-harness verifier failed (unsupported evidence claims: files_touched/tests_passed …)	~100	55%
Verify gate failed: verify_command exited with status 1	19	10%
Provider proxy 502/503 (auth_unavailable, unknown provider for model)	~19	10%
Claude Agent SDK is not installed (SDK runtime selected, SDK absent)	15	8%
Codex configuration drift, misc	~5	3%

• Of 57 distinct ACs rejected by the verifier, only 13 ever completed on a later attempt (avg 2.4 attempts) — the rejection is systematic, not a phrasing problem.
• codex_cli persists 0 execution.tool.completed events against 6,830 execution.tool.started, so a rejected leaf cannot be audited after the fact.

Reproduced false negatives (deterministic, _verify_atomic_evidence_against_runtime_messages fed codex-shaped item.started/item.completed events)

A leaf whose transcript shows Edit hello.py and /bin/zsh -lc 'python3 -m pytest --doctest-modules -q hello.py' (real July case exec_33b50bc500a8) is rejected when:

• the file was written through a shell command (cat > f <<EOF, sed -i, scaffolding, git apply) — no file_change item, no authenticated Bash lease → files_touched unsupported;
• the Codex completion carries no exit_code, or an empty/truncated aggregated_output → tests_passed unsupported even with 1 passed in the output;
• task_cwd differs from the worktree the runtime reports → files_touched unsupported.

Root cause: the verifier's ground truth is the shape of the transcript, not the workspace or the harness's own observation. Each runtime encodes tools differently, so every new host needs another matcher (#2245 is one such patch).

Levers

• L1a ground files_touched in a before/after workspace snapshot the dispatcher takes itself (unforgeable, runtime-independent; unchanged files still rejected).
• L1b when the transcript cannot prove tests_passed, the harness re-runs the claimed test command with a bounded timeout and uses its own exit code (extends fix(orchestrator): normalize paths and unwrap shell wrappers in fat-harness evidence matching (release blocker) #1591).
• L1c persist codex tool completions for diagnosability.
• L2 preflight the selected runtime (SDK importable, backend reachable) before dispatching any AC, and stop classifying proxy auth_unavailable/unknown provider as transient.
• L3 verify_command exit≠0 — needs per-case data; out of scope here.

Removing 70% of the L1+L2 failure families lifts 59% → ~85%. Measured by 10 live ouroboros run executions of a fixed cli-todo seed before and after, with an independent product smoke as ground truth.

Each lever lands as its own single-subsystem PR referencing this issue.

[ouroboros-agent]

This looks like a multi-PR effort that needs design discussion before implementation. Labeling needs-design for human planning.

Scope assessment:

• L1a/L1b change the trust boundary across src/ouroboros/orchestrator/evidence/verification.py, evidence claim/test detection, parallel_executor.py, and workspace snapshot/verify-command execution paths.
• L1c changes Codex tool completion capture and persistence through the runtime adapter and execution_event_emitter.py, with downstream harness projection/delivery consumers.
• L2 spans runtime selection/preflight, SDK/provider capability probes, backend reachability, failure taxonomy, and retry policy.
• Complexity is high and cross-subsystem; the requested >=80% outcome also requires a before/after live-run measurement plan.
• Suggested approach: agree on the dispatcher-owned evidence contract and measurement fixture first, then land L1a, L1b, L1c, and L2 as independently tested PRs as the issue proposes.

ouroboros-agent[bot]

[Q00]

Baseline: 10 live ouroboros run executions on main (d5c2f9f), runtime claude-cli, fixed cli-todo seed (4 ACs, 3 levels, non-mutating verify commands)

run	harness	seconds	product smoke (independent, 4 probes)
1	FAIL	162	3/4
2	FAIL	147	4/4
3	FAIL	175	3/4
4	FAIL	207	4/4
5	FAIL	166	4/4
6	FAIL	135	4/4
7	FAIL	140	4/4
8	FAIL	148	3/4
9	FAIL	166	4/4
10	FAIL	140	3/4

Harness success: 0/10. Product fully working: 6/10, partially (3/4): 4/10. Every level reported 4/4 ACs succeeded in every run; every failure is the same final-settlement rule:

Final acceptance rejected because the workspace changed after verify_command completed; replaying verify_command is not permitted.

AC1/AC3 pass their verify gate at level 1, then AC2 edits habit_tracker.py and AC4 adds test_habit_tracker.py; at settlement their cached workspace digest no longer matches and they are discarded without re-checking. This is the normal shape of any multi-level seed, so on this seed the rule alone takes the success rate to 0%. #2314 replays the verify command once at settlement instead.

The transcript-verifier false negatives (#2312, #2313) did not fire in this baseline because claude-cli delivers Edit/Bash tool results in full; they are the dominant family on codex_cli in the July event store (0 persisted tool completions, 76 verifier rejections).

"After" run (same seed, same runtime, stack #2312 → #2313 → #2314) is in progress.

[Q00]

After: same seed, same runtime (claude-cli), stack #2312 → #2313 → #2314 (e097941)

run	harness	seconds	product smoke
1	PASS	158	3/4*
2	PASS	141	4/4
3	PASS	148	3/4*
4	PASS	181	3/4*
5	PASS	146	4/4
6	PASS	156	3/4*
7	PASS	166	4/4
8	PASS	173	3/4*
9	PASS	152	3/4*
10	PASS	142	4/4

Harness success: 0/10 → 10/10 (baseline in the comment above). Wall-clock per run unchanged (baseline mean 159 s, after 156 s), so the settlement replay and the harness-side test re-execution add no measurable time on this seed.

* The smoke script's 3/4 rows are a scoring artefact: re-running the four probes by hand on run 1's worktree gives 4/4 (add/list, done, exit-2, and 6 passed for the generated pytest suite). The harness verdict is the metric here; the smoke exists only to show the product was real in the baseline failures.

Attribution on this seed is entirely #2314 (every baseline failure was the settlement workspace_mutated rule; #2312/#2313 target the codex_cli transcript-verifier family from the July event store, which claude-cli does not trigger). #2316 covers the environment families (SDK preflight, non-transient auth/provider 5xx).

Caveat on generality: one seed, one runtime, N=10. A codex_cli run of the same seed is the next measurement to attribute #2312/#2313.