Skip to content

test-infra: 'Validate (no e2e)' dies with no failed step (runner OOM) #629

Description

@mdorman

Symptom

Validate (no e2e) intermittently fails with no failed step — the job dies mid-run and gh run view --log-failed reports the work-step log is absent (log not found). Re-runs pass. Signature = null work-step + missing log = the runner was killed (OOM), not a test assertion.

Evidence

  • 1 of the last 5 CI failures had this signature (job 89231127491 in the sampled window).
  • Consistent with the documented runner-loss/OOM pattern — the job runs ~8–9 min then vanishes with no step-level failure.

Root cause (hypothesis)

Runner memory exhaustion under cargo/test parallelism during the coverage-instrumented validate --no-e2e build. Resource-shaped, not a logic failure.

Fix directions

  • Cap build/test concurrency (cargo jobs / nextest threads) to fit the runner's memory, or move the job to a larger runner.
  • Measure peak RSS to size it deliberately.

Why this matters before merge queue

A merge queue runs merge_group checks in addition to pull_request checks — added concurrent load. This OOM is exactly the mode that extra load worsens, and it has no test to mark flaky (retries can't help). So headroom here should precede queue adoption (see the merge-queue issue).

Acceptance

  • Validate (no e2e) completes reliably without runner loss across repeated runs.
  • Peak memory is measured and the job is sized (concurrency cap or larger runner) with headroom.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    Projects

    Status
    Todo

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions