test: deterministic boundary fault-injection matrix for daemon HTTP and subprocess seams

[thymikee]

Part of #1412 (Track E — gate integrity). From external review: Track A deliberately excluded the daemon HTTP and subprocess boundaries — but they are this repo's riskiest seams. Both are already injectable (exec.ts is the single subprocess seam; the provider/transport seam from #1389 covers the runner side), so faults can be injected deterministically, no chaos randomness.

Deliverables

• An enumerated fault matrix — faults: timeout, cancellation mid-request, partial/malformed response, process death mid-request, lost response AFTER a mutation dispatched, filesystem failure on artifact write, optional-optimization failure (e.g. direct-iOS selector fast path errors) — crossed with representative command classes (mutation, read, artifact-producing, session-lifecycle). Completeness enforced satisfies-style over both axes, matching the guarantee-matrix idiom.
• Invariants asserted per cell:
  • no lease/session/claim leaks (inventory clean after recovery);
  • no unsafe mutation replay (a lost-response mutation is never silently re-dispatched);
  • typed error identity survives wrapping — code, hint, diagnosticId, logPath all present at the CLI surface;
  • best-effort probes degrade instead of failing the request;
  • deadlines stay bounded (no hang; assert against the injected budget).
• Runs deterministically in the unit/integration lanes (fake providers + injected transport), not on live devices.

Relationship

Complements, not duplicates: the ADR 0011 guarantee matrix owns semantic guarantees per dispatch path; the torture lane (#1416) owns interleavings; this owns boundary faults. Cells already proven by existing subprocess-stub tests get credited, not rewritten.

Acceptance

• Matrix completeness is a compile/test error, not a review convention.
• At least the lost-response-after-mutation and process-death rows exist for press/fill and open/close before this closes.

[thymikee]

Relevance check (asked alongside #1429, which is now up as #1714).

Still worth doing — but it is not gate work, and should not be bundled with #1429.

Three things I checked against current main:

1. The seams this needs still exist and are still injectable. src/utils/exec.ts is still the single subprocess seam, and src/daemon/client/daemon-client-transport.ts still routes through loadNodeHttpRequester, so faults can be injected deterministically as the issue assumed. The refactor: extract platform device inventory runtime #1699/docs: adopt request-bound platform runtime #1697 platform-runtime refactors moved code around them but did not remove either seam.

2. The risk is live, not theoretical. fix(daemon): report the device claim retained by a failed close #1647 ("report the device claim retained by a failed close", merged 2026-08-06) is exactly a no lease/claim leaks after a boundary failure bug — the first invariant this matrix would assert. That is the strongest argument for keeping it open.

3. Existing coverage is thin. Grepping the daemon and client tests for the fault shapes this enumerates (cancellation mid-request, process death mid-request, lost response after a dispatched mutation) finds essentially nothing — the subprocess-stub project covers command-level behaviour, not boundary faults. So there is little to "credit, not rewrite"; most cells would be new.

Recommendation: leave open, sequence separately. The 2026-07-28 descope on #1412 already reclassified this as product testing to be sequenced against feature work rather than gate work, and that call still looks right — the bulk of the cost here is building the harness (fake daemon + injected transport + lease/claim inventory inspection), not the matrix itself. Under the standing "one issue per PR" rule it wants its own PR.

One scoping note if it gets picked up: the acceptance criterion is already narrow and good ("lost-response-after-mutation and process-death rows for press/fill and open/close"). Building those two rows first — harness included — and letting the compile-enforced completeness check start with the rest of the matrix explicitly waived would land the risky part early and avoid a large speculative build.

No changes made to this issue's scope; nothing in #1714 touches it.

---

Generated by Claude Code

[thymikee]

Recording cutover supplied four concrete first cells for the artifact/session-lifecycle side of this issue: (1) cancellation immediately after native acquisition, (2) durable manifest persistence failure after native start, (3) exact-owner binding resolving after its recovery deadline, and (4) a presence probe returning unknown/timeout rather than proven missing. Land these as one vertical slice through the existing DurableCaptureResource and platform-host interfaces, asserting no unowned native resource, primary-error preservation, secondary cleanup diagnostics, retained evidence on uncertainty, and correct replacement-start admission. Do not build the full matrix or a new fault framework first; reuse the current injected adapters, and generalize only if a second command family needs the same fault adapter.

[thymikee]

Amendment from the #1781 audit (2026-08-17), verified against HEAD 856ff38:

• The subprocess seam is already injectable with no production change: withCommandExecutorOverride in src/utils/exec.ts (AsyncLocalStorage override, 9 existing test call sites). A "fail the Nth call" driver can wrap it directly.
• There is no fs facade anywhere; modules import node:fs as a namespace, so vi.spyOn(fs, 'writeFileSync' | 'renameSync' | 'fsyncSync' | …) is the mechanism (already used read-side in device-claims.test.ts, race-side in session-script-writer.test.ts).
• Six tmp→rename write-then-publish modules have no write/rename-side errno injection at all: durable-capture-resource-store.ts (backs screen-recording and app-log — the app-log store has no boundary test), device-claims.ts, session-script-writer.ts, daemon-shutdown-report.ts, runner-lease.ts, session-store.ts. The only existing template, screen-recording-boundary-faults.test.ts, injects at the store-wrapper level, not as an OS errno.
• Socket seams (runner-usbmux-protocol.ts, daemon-client-transport.ts) are not enumerated as a fault axis in the current matrix.

Suggested addition to the matrix: fs write-side EIO/ENOSPC/EMFILE for the six modules above (mid-sequence fault → assert consistent terminal state and no torn publish), plus the two socket seams. Pairs with the resource-leak oracle in #1781 B1, which is what makes injected faults observable.

[thymikee]

#1781 wave 3 item B1 (the resource-leak oracle this issue's amendment pairs with) is up as #1859 (draft until CI is green).

What B1 covers: test/integration/support/daemon-leak-oracle.ts + calls at the end of the three real-subprocess daemon lanes (smoke-daemon-clean, smoke-daemon-http, daemon-replace-exit-flush). After shutdown it asserts (a) no daemon-owned process survives — ownership = PPID descendant, PGID = daemon pid, AGENT_DEVICE_STATE_DIR=<dir> in the environment, state-dir path as an argv token, propagated down the tree; never a global process count — and (b) the isolated state dir holds only classified artifacts (allowlist, no *.tmp, no daemon.json/.lock without a live daemon, no lifecycle: "open" capture descriptor / app-log.pid after shutdown). Red-proven against the pre-fix commits of #1324 (simctl recordVideo ppid 1 / pgid = dead daemon, 0-byte mp4) and #1109 (agent-browser daemon + 12 Chrome processes survive SIGTERM); green on main for both scenarios. assertNoDaemonLeaks({ stateDir, daemonPids, phase }) is the single entry point a fault driver can call after each injected fault.

What remains for #1431: the fs write-side errno matrix (EIO/ENOSPC/EMFILE across the six tmp→rename modules), the socket seams (runner-usbmux-protocol.ts, daemon-client-transport.ts), the exec.ts withCommandExecutorOverride fail-the-Nth-call driver, and satisfies-style completeness over faults × command classes. B1 deliberately does not build any of that; it only makes the leak invariant observable.

One candidate cell surfaced by the oracle (main a70cdee36, not a regression — it matches #1109's accepted "within the idle window"): SIGTERM of a daemon holding an open web session leaves the agent-browser daemon + Chrome fleet alive until the 5-minute idle window elapses, and a fresh daemon on the same state dir does not reap it (the startup reaper skips fleets with recent activity). Graceful daemon shutdown has no web-session close step, unlike the recording/app-log/perf steps in teardownSessionResources. Explicit close is clean.