Symptom
Validate (no e2e) intermittently fails with no failed step — the job dies mid-run and gh run view --log-failed reports the work-step log is absent (log not found). Re-runs pass. Signature = null work-step + missing log = the runner was killed (OOM), not a test assertion.
Evidence
- 1 of the last 5 CI failures had this signature (job
89231127491 in the sampled window).
- Consistent with the documented runner-loss/OOM pattern — the job runs ~8–9 min then vanishes with no step-level failure.
Root cause (hypothesis)
Runner memory exhaustion under cargo/test parallelism during the coverage-instrumented validate --no-e2e build. Resource-shaped, not a logic failure.
Fix directions
- Cap build/test concurrency (cargo jobs / nextest threads) to fit the runner's memory, or move the job to a larger runner.
- Measure peak RSS to size it deliberately.
Why this matters before merge queue
A merge queue runs merge_group checks in addition to pull_request checks — added concurrent load. This OOM is exactly the mode that extra load worsens, and it has no test to mark flaky (retries can't help). So headroom here should precede queue adoption (see the merge-queue issue).
Acceptance
Symptom
Validate (no e2e)intermittently fails with no failed step — the job dies mid-run andgh run view --log-failedreports the work-step log is absent (log not found). Re-runs pass. Signature = null work-step + missing log = the runner was killed (OOM), not a test assertion.Evidence
89231127491in the sampled window).Root cause (hypothesis)
Runner memory exhaustion under
cargo/test parallelism during the coverage-instrumentedvalidate --no-e2ebuild. Resource-shaped, not a logic failure.Fix directions
Why this matters before merge queue
A merge queue runs
merge_groupchecks in addition topull_requestchecks — added concurrent load. This OOM is exactly the mode that extra load worsens, and it has no test to mark flaky (retries can't help). So headroom here should precede queue adoption (see the merge-queue issue).Acceptance
Validate (no e2e)completes reliably without runner loss across repeated runs.