ci(e2e): deduplicate the self-hosted e2e lanes into a composite action

[rominf]

Problem

The self-hosted e2e lane definitions in .github/workflows/e2e-selfhosted.yml and .github/workflows/nightly.yml are largely copy-paste. Measured across the 10 lanes in those two files, 62% of step-body lines (502 of 806) sit inside a block that appears verbatim in at least one other lane.

Repeated block	Copies	Lines
GPU preflight (bounded wait…) bash	5	130
GPU preflight — advisory + PowerShell variants	2 + 2	128
Upload E2E report (differs only in name:)	10	60
Reclaim GPU from stray E2E processes	8	66
WSL bootstrap / toolchain / checkout	various	106

The five copies of the GPU preflight script differ in exactly one integer (MIN_FREE_GIB, 16 vs 8); the other ~26 lines are byte-identical. The main Run E2E tests script only escapes exact matching because of path substitutions — by sequence similarity it is 97.1% identical between e2e-gpu and e2e-gpu-rad3, and 97.6% between their nightly equivalents.

The cost is drift, and it is already visible. Examples found while reviewing #278:

• e2e-gpu-rad3 omits E2E_MERGE_QUEUE, which all four sibling lanes set — so it silently runs the cheap PR canary during merge-queue.
• e2e-gpu-nightly is the only GPU lane with no preflight step at all.
• The per-PR e2e-gpu-strix-windows lane has no E2E_SERVE_TIMEOUT_SECS while its nightly twin sets "300".
• e2e-gpu-nightly-rad3 drops the pre-warm diagnostics its own comment says it mirrors.

Each of these is a one-line divergence in a block that is otherwise identical five ways over. There is no mechanism that would catch any of them.

Proposed approach

Extract the repeated blocks into a composite action (e.g. .github/actions/e2e-gpu-lane-linux/action.yml) parameterised on the axes that genuinely vary: min-free-gib, preflight strict-vs-advisory, shared uv cache dir, scenario name filter, artifact name.

A composite action cannot touch name:, if:, needs:, runs-on:, continue-on-error: or strategy:, so it is provably invisible to branch protection — job headers stay byte-identical. Eight Linux call sites (4 lanes × 2 files) collapse onto one definition; an individual lane goes from ~139 lines to ~25. Windows needs a second PowerShell-only action or stays inline.

Alternatives considered

Matrix — viable for check names (codeql.yml already proves the name: Analyze (${{ matrix.language }}) pattern here), but matrix is not in scope at jobs.<id>.if, so the per-lane inputs.platform == '…' gating would have to move into a step gate. That means waking a scale-to-zero runner just to no-op. It also needs strategy.fail-fast: false or one flaky self-hosted machine cancels the rest, and the PowerShell lane can't join. Reasonable as a second step, not the first.

workflow_call — should not be used to bridge ci.yml and e2e-selfhosted.yml. That split exists so a job queued on an offline self-hosted runner cannot stall a required check, and it is enforced by ci_yml_schedules_no_self_hosted_job and workflows_use_distinct_concurrency_groups in xtask/src/workflow_contract.rs. It also mangles check names to <caller> / <called>.

Non-goal: collapsing to one or two jobs

Four constraints set the floor, none of them stylistic:

1. ci.yml's E2E tests and E2E consolidated report are required checks and must stay distinct.
2. Self-hosted lanes must live in a separate workflow file for concurrency isolation — enforced by the two workflow_contract.rs tests above.
3. Windows vs Linux is a hard job split (PowerShell 5.1 vs bash throughout).
4. nightly.yml is a separate trigger and report tier.

Realistic floor is 8 jobs from the current 14, and only if the if:-to-step-gate refactor is accepted. Keeping all 14 jobs and applying composite actions alone removes ~500 duplicated lines at zero structural risk, which is the better trade and should be done first.

Also in scope: make the docs guard actually guard

hardware_testing_docs_cover_all_four_self_hosted_platforms in xtask/src/workflow_contract.rs compares docs/ci-hardware-testing.md against hardcoded expectations, never against e2e-selfhosted.yml / nightly.yml. It therefore cannot detect a newly added lane — it only fails when someone tries to fix the docs to match reality, which is backwards. Adding a lane in #278 left the doc table, the artifact list, two "the four self-hosted jobs" sentences and the dispatch-options list stale, with the test green throughout.

Derive the expected rows from the actual job list (runs_on_values() and friends already exist in that file) so adding a lane fails the build until the docs follow.

Acceptance

• Repeated preflight / reclaim / build-and-run / upload blocks exist in one place each.
• Every existing job name and required-check context is unchanged.
• Adding a new lane without updating docs/ci-hardware-testing.md fails CI.
• The four drift instances listed above are resolved or deliberately documented.

[volen-silo]

Picked this up. Before starting I checked the four drift instances against the tree — two hold, two don't, so the acceptance list is worth correcting.

Confirmed

e2e-gpu-nightly has no preflight step. Its full step list is checkout → Reclaim (nightly.yml:356) → setup-rust-toolchain (:368) → Run (:372) → Upload (:410). It is the only one of the ten self-hosted lanes across both files without a preflight; its per-PR twin has one (e2e-selfhosted.yml:200, MIN_FREE_GIB 16) and so does its sibling e2e-gpu-nightly-rad3 (nightly.yml:446).

Per-PR e2e-gpu-strix-windows has no E2E_SERVE_TIMEOUT_SECS. e2e-selfhosted.yml:505-516 is the complete env block: E2E_TUI_TIMEOUT_SECS, E2E_INCLUDE_NIGHTLY, E2E_MERGE_QUEUE, and nothing else. The other four per-PR lanes set "300", as does its own nightly twin (nightly.yml:613-616).

Not reproduced

e2e-gpu-rad3 does set E2E_MERGE_QUEUE — e2e-selfhosted.yml:844-848:

    env:
      E2E_SERVE_TIMEOUT_SECS: "300"
      E2E_INCLUDE_NIGHTLY: "${{ inputs.include_nightly && '1' || '' }}"
      # Full serve matrix at the merge-queue gate, like every other lane (see e2e-gpu).
      E2E_MERGE_QUEUE: "${{ github.event_name == 'merge_group' && '1' || '' }}"

All five per-PR lanes set it. The env key e2e-gpu-rad3 actually omits is E2E_TUI_TIMEOUT_SECS — but it shares that omission with e2e-gpu, and the key is a Strix-machine-contention mitigation, so it reads as deliberate rather than drift.

e2e-gpu-nightly-rad3 drops comment prose, not diagnostics. Its pre-warm invocation (nightly.yml:498-500) is the same three lines as the per-PR lane's (e2e-selfhosted.yml:930-932); a comments-stripped diff of the two run steps shows only the step name: and the absent NAME_FILTER conditional, which cannot exist here because nightly.yml has no name_filter dispatch input. There is no executable pre-warm diagnostic in either copy to drop. What the nightly copy does compress is ~12 lines of rationale down to ~6.

One thing the issue understates

The composite-action extraction has a blast radius worth knowing about before starting: six guard tests read run: script bodies straight out of the workflow files. Four break outright — the two *_prebuilt_e2e_lanes_export_rocmd / *_enable_test_hooks pairs (via multiline_run_blocks) and wsl_jobs_accept_every_canonical_wsl_detection_signal (exact shell-line match).

Worse, two would pass vacuously rather than fail: uploaded_e2e_artifacts (xtask/src/e2e_report.rs:191-215) globs only .github/workflows/*, so moving the upload step makes every_uploaded_e2e_artifact_has_a_name_the_report_can_label iterate an empty list, and the_nightly_lanes_publish_the_same_platforms_as_the_per_pr_lanes compare [] == []. Both go green with the coverage gone — including the guard docs/ci-hardware-testing.md:98-99 explicitly advertises. So read_workflow and workflow_files() need to widen to .github/actions/** before any run: body moves.

Also, a note on the docs guard

It is a bit worse than described. Besides never reading the workflows, it drops the Runner labels column entirely (.map(|row| (row[0], row[2])), workflow_contract.rs:752), so that column is unverified against anything. The dispatch-options list and both "the self-hosted jobs (…)" prose enumerations are unguarded too. (Minor: the test is now hardware_testing_docs_cover_all_self_hosted_platforms — the ..._four_... name was renamed in #278.)

Plan

Two focused PRs first, delivering acceptance criteria 3 and 4:

1. Derive the docs guard from the workflows — job rows, runner labels, artifact list, dispatch options, and both prose enumerations — so adding a lane fails CI until the docs follow.
2. Fix the two confirmed divergences, with contract tests over the derived job list so neither can re-drift.

The composite-action extraction follows separately, once the guard tests can survive it and #316 (which edits the same build lines) has landed.