fix(install): always keep the SDK's build of the torch an engine pins #661

Workflow file for this run

.github/workflows/e2e-selfhosted.yml at 74a157a

	name: E2E self-hosted

	# Self-hosted GPU E2E lanes, split out of ci.yml on purpose. A job queued on an
	# OFFLINE self-hosted runner cannot be cancelled by GitHub, so if it shared
	# ci.yml's concurrency group a superseded run would hold that group forever and
	# the newer run's merge-required (GitHub-hosted) checks would sit pending with
	# zero jobs (observed on PR #138). Isolating these lanes in their own workflow —
	# with their OWN concurrency group — means an offline runner can only ever stall
	# THIS workflow's supersession, never the required checks in ci.yml.
	#
	# These lanes are non-blocking (continue-on-error). NOTE: their check names
	# (`E2E tests (GPU)` etc.) are still in main's required-status-check list, so
	# while a runner is offline they report as missing and can still block a merge;
	# fully closing that requires removing them from the required list (a separate
	# branch-protection change, out of scope for this workflow).

	on:
	push:
	branches: [main]
	# Mirror ci.yml: run on PRs against ANY base branch so a stacked PR still gets
	# the GPU lanes. Affected-crate/heavy selection diffs against github.base_ref.
	pull_request:
	merge_group:
	branches: [main]
	types: [checks_requested]
	# On-demand GPU E2E without a push/PR. Dispatch against any ref
	# (`gh workflow run e2e-selfhosted.yml --ref <branch> -f platform=... `) — the
	# mock lane's dispatch stays on ci.yml; this one covers the self-hosted runners.
	workflow_dispatch:
	inputs:
	platform:
	description: Which self-hosted runner(s) to target
	type: choice
	default: all
	# Each value also selectable on its own; `all` covers every lane below.
	options: [all, app-dev-gpu, strix-ubuntu, strix-windows, strix-wsl, rad3]
	name_filter:
	description: "Scenario-name regex (cucumber --name); empty = full suite"
	type: string
	default: ""
	include_nightly:
	description: "Include @nightly scenarios (large-model serve, cold install)"
	type: boolean
	default: false

	concurrency:
	# Own group, namespaced by this workflow so it is DISTINCT from ci.yml's shared
	# group — the whole point of the split. Manual dispatches still get a unique
	# (run_id) group so a stuck dispatch never blocks later dispatches.
	group: >-
	${{ github.workflow }}-${{ github.ref }}-${{
	github.event_name == 'workflow_dispatch' && github.run_id \|\| 'shared' }}
	cancel-in-progress: true

	permissions:
	contents: read

	env:
	CARGO_TERM_COLOR: always

	jobs:
	# Trimmed copy of ci.yml's `changes` job: cross-workflow `needs` is impossible,
	# so this workflow computes its own `heavy` gate to avoid running the expensive
	# GPU lanes on a doc-only PR. Off pull_request (push/merge_group) heavy is
	# forced true, matching ci.yml.
	changes:
	runs-on: ubuntu-latest
	permissions:
	contents: read
	pull-requests: read
	outputs:
	heavy: ${{ steps.filter.outputs.heavy \|\| steps.all.outputs.forced }}
	# Narrow GPU-serve gate; forced true off-PR so the merge queue always runs
	# the full matrix and its required checks are never starved.
	serve: ${{ steps.filter.outputs.serve \|\| steps.all.outputs.forced }}
	steps:
	- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

	- name: Filter changed paths (pull requests only)
	id: filter
	if: github.event_name == 'pull_request'
	uses: dorny/paths-filter@ceb8a2b8f2d89434be7ff52d3de7ec3738c5cc9d # v4.0.3
	with:
	filters: \|
	# Same `heavy` set as ci.yml: anything that can affect the E2E suite.
	heavy:
	- '*/.rs'
	- '**/Cargo.toml'
	- 'Cargo.lock'
	- 'rust-toolchain*'
	- 'scripts/**'
	- 'xtask/**'
	- 'engines/**'
	- '*/.py'
	- '*/.sh'
	- '*/.ps1'
	- '*/.feature'
	- 'tests/e2e-cucumber/**'
	- 'install*'
	- 'docs/keys/**'
	- '.github/workflows/**'
	# The real-GPU serve matrix (e2e-gpu*). Narrower than `heavy`: only
	# paths that can change serve BEHAVIOUR or the GPU E2E harness — NOT a
	# blanket `*/.rs`. Compile coverage for every crate already runs on
	# ci.yml's always-on build/test lanes, so a dash-only or unrelated-
	# crate Rust PR need not fire the heavy serve matrix. Err toward
	# inclusion — Cargo.lock, the toolchain, and the workflow itself are
	# broad safety nets so a transitive-dep or CI change still runs the
	# matrix. (Excluded on purpose: crates/rocm-dash-* — they build into
	# `rocm` but cannot change serve behaviour.)
	serve:
	- 'engines/**'
	- 'crates/rocm-core/**'
	- 'crates/rocm-engine-protocol/**'
	- 'apps/rocm/**'
	- 'apps/rocmd/**'
	- 'tests/e2e-cucumber/**'
	- 'crates/e2e-report/**'
	- 'xtask/**'
	- '*/.feature'
	- 'scripts/**'
	# Root manifest only (NOT `**/Cargo.toml` — that would re-include
	# the excluded dash crates): `[workspace.dependencies]` edits here
	# can change a serving crate's deps without touching Cargo.lock.
	- 'Cargo.toml'
	- 'Cargo.lock'
	- 'rust-toolchain*'
	- '.github/workflows/**'

	- name: Force full run off pull requests
	id: all
	if: github.event_name != 'pull_request'
	run: echo "forced=true" >> "$GITHUB_OUTPUT"

	e2e-gpu:
	name: E2E tests (GPU)
	timeout-minutes: 90
	runs-on: [self-hosted, linux, amd-gpu]
	needs: [changes]
	# No build-and-test gate (cross-workflow needs is unavailable): the job builds
	# the rocm binary itself and is continue-on-error; ci.yml's required
	# build-and-test / mock e2e remain the authoritative pre-merge build gate.
	if: >-
	always()
	&& needs.changes.result == 'success'
	&& (
	(github.event_name != 'workflow_dispatch'
	&& needs.changes.outputs.serve == 'true')
	\|\| (github.event_name == 'workflow_dispatch'
	&& (inputs.platform == 'all' \|\| inputs.platform == 'app-dev-gpu'))
	)
	continue-on-error: true
	env:
	# Bound serve readiness below the 35-min job cap so a serve that never comes
	# ready fails the scenario with a real error instead of hanging until the
	# job is cancelled. 300s is ample for a real MI300X vLLM cold-start;
	# per-scenario overrides in expectations.toml / a `@serve-timeout` tag adjust
	# it (shorter for known bugs, longer for large models).
	E2E_SERVE_TIMEOUT_SECS: "300"
	# Opt-in @nightly scenarios on a manual dispatch (default off). The nightly
	# workflow sets this unconditionally; here it lets a scoped dispatch confirm
	# a single nightly scenario (e.g. the 27B serve) without the full nightly run.
	E2E_INCLUDE_NIGHTLY: "${{ inputs.include_nightly && '1' \|\| '' }}"
	# Opt-in @merge-queue serves: the heavy real serves (default-engine +
	# readiness) run only in the merge queue, where a cheaper per-engine canary
	# (scenarios 5 vLLM / 7 lemonade) has already guarded the PR. This keeps the
	# per-PR GPU run short while still exercising the full serve matrix before a
	# change lands. Only set on the `merge_group` event.
	E2E_MERGE_QUEUE: "${{ github.event_name == 'merge_group' && '1' \|\| '' }}"
	steps:
	- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

	# Reclaim the GPU before running: a serve leaked by a killed/timed-out prior
	# run (its Drop teardown never executed) can keep an engine process spinning
	# on the GPU, starving this job's serves until it hits the timeout. Kill only
	# e2e leftovers — scoped to /tmp/rocm-e2e-* and the e2e-target/e2e-shared
	# trees — never the runner or any /workload manual-testing processes.
	- name: Reclaim GPU from stray E2E processes
	run: \|
	pkill -f '/tmp/rocm-e2e.*llama-server' 2>/dev/null \|\| true
	pkill -f '/tmp/rocm-e2e.*vllm serve' 2>/dev/null \|\| true
	pkill -f 'e2e-shared.*llama-server' 2>/dev/null \|\| true
	# NOTE: no unanchored `pkill -f 'vulkan/llama-server'` — the lemonade
	# Vulkan assistant an e2e scenario spawns lives under /tmp/rocm-e2e-*
	# and is already caught by the first line; an unanchored pattern would
	# also kill a legitimate /workload manual-testing serve on this shared
	# self-hosted runner.
	pkill -f '__engine-serve-http.*rocm-e2e' 2>/dev/null \|\| true
	pkill -f 'e2e-target/release/rocm daemon' 2>/dev/null \|\| true
	rm -rf /tmp/rocm-e2e-* 2>/dev/null \|\| true
	echo "reclaimed"

	# GPU preflight: fail fast (~90s) instead of hanging to the job cap when the
	# GPU is missing, the driver is wedged, or VRAM is still saturated by a
	# leftover serve. A BOUNDED POLL, not a one-shot check: transient contention
	# (e.g. the reclaim step's kills still draining VRAM) self-heals within
	# seconds, so we retry up to a ceiling and succeed the moment the GPU is both
	# responsive AND has enough free VRAM. Only a genuinely absent/wedged/held
	# GPU reaches the ceiling and fails — with a per-reason message.
	- name: GPU preflight (bounded wait for an available GPU)
	run: \|
	# A serve here needs most of the card; require a generous free-VRAM
	# floor so a leftover serve (which the reclaim step should have killed)
	# is caught, while normal baseline (~300 MB used) passes immediately.
	MIN_FREE_GIB="${GPU_PREFLIGHT_MIN_FREE_GIB:-16}"
	CEILING_SECS="${GPU_PREFLIGHT_CEILING_SECS:-90}"
	min_free=$(( MIN_FREE_GIB * 1024 * 1024 * 1024 ))
	deadline=$(( SECONDS + CEILING_SECS ))
	reason="rocm-smi never returned within its timeout (driver wedged or GPU absent)"
	while [ "$SECONDS" -lt "$deadline" ]; do
	# rocm-smi itself can hang on a wedged driver — bound it with timeout.
	out=$(timeout 15 rocm-smi --showmeminfo vram 2>/dev/null) \|\| { sleep 5; continue; }
	# Parse the byte value AFTER the colon; the line prefix "GPU[0]" would
	# otherwise make a naive first-number match pick up the "0".
	total=$(printf '%s\n' "$out" \| grep -i 'VRAM Total Memory' \| sed 's/.: //' \| grep -oE '[0-9]+' \| tail -1)
	used=$(printf '%s\n' "$out" \| grep -i 'VRAM Total Used Memory' \| sed 's/.: //' \| grep -oE '[0-9]+' \| tail -1)
	if [ -z "$total" ] \|\| [ -z "$used" ]; then
	reason="rocm-smi returned no VRAM figures (no AMD GPU detected)"
	sleep 5; continue
	fi
	free=$(( total - used ))
	if [ "$free" -ge "$min_free" ]; then
	echo "GPU ready: $(( free / 1024 / 1024 / 1024 )) GiB free (>= ${MIN_FREE_GIB} GiB)."
	exit 0
	fi
	reason="VRAM never dropped below the floor: only $(( free / 1024 / 1024 / 1024 )) GiB free (< ${MIN_FREE_GIB} GiB) — a serve is likely still holding the GPU"
	echo "waiting: $(( free / 1024 / 1024 / 1024 )) GiB free (< ${MIN_FREE_GIB} GiB)…"
	sleep 5
	done
	echo "::error::GPU preflight failed after ${CEILING_SECS}s: ${reason}"
	exit 1

	# cache: false — on this self-hosted runner we persist the build cache
	# ourselves via CARGO_TARGET_DIR (below). The action's built-in
	# Swatinem/rust-cache otherwise tries to SAVE the large target dir to
	# GitHub's cache service in a post-step (slow/hangs) and its cleanup wipes
	# the local target.
	- uses: actions-rust-lang/setup-rust-toolchain@166cdcfd11aee3cb47222f9ddb555ce30ddb9659 # v1.17.0
	with:
	cache: false

	# `actions/checkout` runs `git clean -ffdx`, which deletes the gitignored
	# `target/` inside the repo every job → a full ~15min rebuild each run.
	# Point CARGO_TARGET_DIR at a sibling of the checkout ($RUNNER_WORKSPACE is
	# the checkout's parent — untouched by git clean and persistent between jobs
	# on a self-hosted runner), so cargo rebuilds incrementally.
	- name: Run E2E tests on GPU hardware
	run: \|
	export CARGO_TARGET_DIR="$RUNNER_WORKSPACE/e2e-target"
	# Share heavy immutable artifacts (TheRock runtimes ~3.3GB, HF weights,
	# vLLM venv) across scenarios so they download once per runner, not per
	# scenario. Persistent path; service state stays isolated per scenario.
	export E2E_SHARED_CACHE_DIR="$RUNNER_WORKSPACE/e2e-shared"
	# Share uv's wheel download/build cache so `rocm install sdk` is a warm
	# ~34s per scenario instead of a cold ~160s (measured on MI300X). Kept
	# OFF the RUNNER_WORKSPACE Longhorn PVC (near-full) and on the roomy `/`
	# overlay; ~23GB, one cold fill per pod-life. Not git-cleaned (outside
	# the checkout) and outside the reclaim step's /tmp/rocm-e2e-* glob.
	export E2E_SHARED_UV_CACHE_DIR="/var/tmp/rocm-e2e-uv-cache"
	# Share ONE installed managed runtime across the serve/chat scenarios so
	# `rocm install sdk` runs once per runner, not once per scenario (the
	# per-scenario install count — each a multi-GiB TheRock SDK whose probe
	# unpacks an ~8.8 GiB devel tarball — is what blew the time cap). The
	# "a managed runtime is active" precondition symlinks each scenario's
	# data/runtimes here (see use_shared_runtimes); clean-slate scenarios
	# stay isolated. Persisted across runs on RUNNER_WORKSPACE and refreshed
	# by `xtask e2e-prewarm` when the channel index publishes a newer version.
	#
	# CRITICAL: the shared dir IS the pre-warm's own `data/runtimes`, and we
	# NEVER move it. `install sdk` bakes ABSOLUTE paths (install_root,
	# python_executable) into the runtime manifest; a post-install `mv` would
	# leave those pointing at a deleted location and every serve would fail
	# instantly (observed on run 29320025393). Installing in place keeps the
	# baked paths valid, and each scenario's data/runtimes symlink resolves to
	# this same real tree.
	prewarm="$RUNNER_WORKSPACE/e2e-prewarm"
	export E2E_SHARED_RUNTIMES_DIR="$prewarm/data/runtimes"

	# Build the rocm and rocmd binaries ONCE and reuse them for both the
	# pre-warm and suite so xtask does not rebuild. Honors
	# CARGO_TARGET_DIR set above.
	#
	# `--features rocm/e2e-test-hooks` must match what `cargo xtask e2e`
	# builds when it builds for itself. The suite's deterministic failure
	# seams (e.g. the scripted Lemonade backend-install failure) are
	# compiled out without it, so a pre-built binary that omits the feature
	# leaves those scenarios unable to reach their premise — they then fail
	# as regressions on whichever lane happens to select them. Every lane
	# that pre-builds and exports ROCM_CLI_BINARY must pass it.
	cargo build --release -p rocm -p rocmd --features rocm/e2e-test-hooks
	export ROCM_CLI_BINARY="$CARGO_TARGET_DIR/release/rocm"
	export ROCM_CLI_ROCMD_BINARY="$CARGO_TARGET_DIR/release/rocmd"

	# Pre-warm the shared runtime ONCE, SERIALLY, before the suite — never
	# lazily inside a concurrent scenario (two multi-GiB installs racing the
	# same dir). Installs directly into the persistent pre-warm data dir (no
	# mv), so the manifest's absolute install_root stays valid for every
	# scenario. Uses the prebuilt binary (no cargo run --release) and the
	# shared uv + HF caches.
	#
	# This is a CACHE, not a one-shot: `xtask e2e-prewarm` installs when the
	# tree is empty, installs the newer runtime side-by-side and activates it
	# when the channel index has moved on, and otherwise reuses what is there.
	# The old guard here tested directory existence only, so it never
	# reinstalled and every lane froze on the first runtime it ever saw
	# (16 days stale on both MI300X runners when measured — EAI-8057).
	HF_HOME="$E2E_SHARED_CACHE_DIR/huggingface" \
	UV_CACHE_DIR="$E2E_SHARED_UV_CACHE_DIR" \
	cargo xtask e2e-prewarm --channel release --prewarm-dir "$prewarm"

	# Optional scenario-name filter for a scoped dispatch — lets a manual
	# run select only the large-model scenario instead of the whole suite.
	NAME_FILTER="${{ github.event.inputs.name_filter }}"
	if [ -n "$NAME_FILTER" ]; then
	echo "name filter active: $NAME_FILTER"
	cargo xtask e2e -- --name "$NAME_FILTER"
	else
	cargo xtask e2e
	fi

	- name: Upload E2E report
	if: always()
	uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
	with:
	name: e2e-gpu-report
	path: tests/e2e-cucumber/results/

	# Second AMD GPU architecture: the Strix Halo (gfx1151) Ubuntu runner, targeted
	# by the `strix-halo` label (app-dev-gpu carries `amd-gpu`, not `strix-halo`).
	# Non-blocking while this hardware is proven out.
	e2e-gpu-strix-ubuntu:
	name: E2E tests (Strix Halo, Ubuntu)
	# 35min: see e2e-gpu — one collapsed job runs all serves + per-scenario
	# install sdk; the cap must exceed the run so the job writes platform.json.
	timeout-minutes: 35
	# `native` disambiguates the two Linux Strix runners: the WSL host also
	# carries `strix-halo`, but the paths below exist only on the native one.
	runs-on: [self-hosted, linux, strix-halo, native]
	needs: [changes]
	# See `e2e-gpu`: no build-and-test gate (cross-workflow); strix-ubuntu.
	if: >-
	always()
	&& needs.changes.result == 'success'
	&& (
	(github.event_name != 'workflow_dispatch'
	&& needs.changes.outputs.serve == 'true')
	\|\| (github.event_name == 'workflow_dispatch'
	&& (inputs.platform == 'all' \|\| inputs.platform == 'strix-ubuntu'))
	)
	continue-on-error: true
	# On this runner `/`, `/home/ubuntu`, and `/tmp` are ALL on a full root
	# partition; only /home/ubuntu/actions-runner (a 1.7T nvme) has space. So
	# EVERYTHING the job writes must land on the nvme. Point HOME there (catches
	# ~/.cache/pip, ~/.config, and any other $HOME writer — pip's cache under the
	# real /home/ubuntu is what previously failed `install sdk` with ENOSPC),
	# plus the toolchain, temp, and pip cache. The rustup bootstrap still uses
	# --no-modify-path so it doesn't touch $HOME/.profile. These paths are
	# specific to that host, which is why `runs-on` pins `native` above.
	env:
	HOME: /home/ubuntu/actions-runner/e2e-home
	CARGO_HOME: /home/ubuntu/actions-runner/.cargo
	RUSTUP_HOME: /home/ubuntu/actions-runner/.rustup
	TMPDIR: /home/ubuntu/actions-runner/tmp
	PIP_CACHE_DIR: /home/ubuntu/actions-runner/pip-cache
	E2E_SERVE_TIMEOUT_SECS: "300"
	# The three Strix lanes share one physical machine, and this workflow now
	# runs a third of them, so a TUI frame that renders well inside the 30s
	# default on an idle runner can miss it while a sibling lane loads a model.
	# Raise the wait budget rather than let that read as a product failure; a
	# genuine hang still fails, just later.
	E2E_TUI_TIMEOUT_SECS: "90"
	# Match the MI300X dispatch path: opt into the platform-adaptive large-model
	# scenario only when the manual include_nightly input is enabled.
	E2E_INCLUDE_NIGHTLY: "${{ inputs.include_nightly && '1' \|\| '' }}"
	# Heavy @merge-queue serves run only in the merge queue; see e2e-gpu.
	E2E_MERGE_QUEUE: "${{ github.event_name == 'merge_group' && '1' \|\| '' }}"
	steps:
	- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

	- name: Prepare writable dirs on the nvme + reclaim GPU from stray E2E procs
	run: \|
	mkdir -p /home/ubuntu/actions-runner/e2e-home /home/ubuntu/actions-runner/tmp /home/ubuntu/actions-runner/pip-cache
	# Reclaim the GPU from any serve leaked by a killed/timed-out prior run
	# (see e2e-gpu). Scoped to e2e leftovers only.
	pkill -f '/tmp/rocm-e2e.*llama-server' 2>/dev/null \|\| true
	pkill -f '/tmp/rocm-e2e.*vllm serve' 2>/dev/null \|\| true
	pkill -f 'e2e-shared.*llama-server' 2>/dev/null \|\| true
	pkill -f '__engine-serve-http.*rocm-e2e' 2>/dev/null \|\| true
	rm -rf /tmp/rocm-e2e-* 2>/dev/null \|\| true
	echo "prepared + reclaimed"

	# GPU preflight: bounded wait so a missing/wedged/held GPU fails fast (~90s)
	# instead of hanging to the job cap. Transient contention (VRAM still
	# draining from the reclaim above) self-heals within the ceiling. See the
	# e2e-gpu job for the rationale. gfx1151 shares system memory, so the free
	# floor is smaller than the Instinct card but still catches a leftover serve.
	- name: GPU preflight (bounded wait for an available GPU)
	run: \|
	MIN_FREE_GIB="${GPU_PREFLIGHT_MIN_FREE_GIB:-8}"
	CEILING_SECS="${GPU_PREFLIGHT_CEILING_SECS:-90}"
	min_free=$(( MIN_FREE_GIB * 1024 * 1024 * 1024 ))
	deadline=$(( SECONDS + CEILING_SECS ))
	reason="rocm-smi never returned within its timeout (driver wedged or GPU absent)"
	while [ "$SECONDS" -lt "$deadline" ]; do
	out=$(timeout 15 rocm-smi --showmeminfo vram 2>/dev/null) \|\| { sleep 5; continue; }
	total=$(printf '%s\n' "$out" \| grep -i 'VRAM Total Memory' \| sed 's/.: //' \| grep -oE '[0-9]+' \| tail -1)
	used=$(printf '%s\n' "$out" \| grep -i 'VRAM Total Used Memory' \| sed 's/.: //' \| grep -oE '[0-9]+' \| tail -1)
	if [ -z "$total" ] \|\| [ -z "$used" ]; then
	reason="rocm-smi returned no VRAM figures (no AMD GPU detected)"
	sleep 5; continue
	fi
	free=$(( total - used ))
	if [ "$free" -ge "$min_free" ]; then
	echo "GPU ready: $(( free / 1024 / 1024 / 1024 )) GiB free (>= ${MIN_FREE_GIB} GiB)."
	exit 0
	fi
	reason="VRAM never dropped below the floor: only $(( free / 1024 / 1024 / 1024 )) GiB free (< ${MIN_FREE_GIB} GiB) — a serve is likely still holding the GPU"
	echo "waiting: $(( free / 1024 / 1024 / 1024 )) GiB free (< ${MIN_FREE_GIB} GiB)…"
	sleep 5
	done
	echo "::error::GPU preflight failed after ${CEILING_SECS}s: ${reason}"
	exit 1

	# Bootstrap rustup ourselves with --no-modify-path so it never writes to
	# $HOME/.profile (setup-rust-toolchain doesn't expose that flag).
	# rust-toolchain.toml pins the exact toolchain, installed on first cargo
	# use. Idempotent.
	- name: Ensure Rust toolchain
	run: \|
	if ! command -v cargo >/dev/null 2>&1 && [ ! -x "$CARGO_HOME/bin/cargo" ]; then
	curl --proto '=https' --tlsv1.2 -fsSL https://sh.rustup.rs \
	\| sh -s -- -y --no-modify-path --default-toolchain none
	fi
	echo "$CARGO_HOME/bin" >> "$GITHUB_PATH"

	- name: Run E2E tests on Strix Halo
	run: \|
	export CARGO_TARGET_DIR="$RUNNER_WORKSPACE/e2e-target"
	export E2E_SHARED_CACHE_DIR="$RUNNER_WORKSPACE/e2e-shared"
	# Share ONE installed managed runtime across serve/chat scenarios, and
	# PRE-WARM it in place before the suite (mirrors e2e-gpu). This is not an
	# optimization — it is REQUIRED for correctness. `install sdk` bakes
	# ABSOLUTE paths (install_root, python_executable, rocm_sdk.*) into the
	# runtime manifest. If the first scenario installs into its own isolated
	# /tmp/rocm-e2e-XXXX data dir and only the registry is shared onward, those
	# baked paths point at that scenario's temp dir — which is deleted when the
	# scenario ends. Every later serve then sees `status=unusable (install root
	# is missing)` and fails (diagnosed on the box 2026-07-15). Installing in
	# place, directly into the persistent shared dir, keeps the baked paths
	# valid for all scenarios and for the end-of-run version probe.
	prewarm="$RUNNER_WORKSPACE/e2e-prewarm"
	export E2E_SHARED_RUNTIMES_DIR="$prewarm/data/runtimes"

	# Build the rocm and rocmd binaries once; reuse them for pre-warm + suite.
	# See the e2e-gpu lane for why the e2e-test-hooks feature must match
	# what `cargo xtask e2e` would build.
	cargo build --release -p rocm -p rocmd --features rocm/e2e-test-hooks
	export ROCM_CLI_BINARY="$CARGO_TARGET_DIR/release/rocm"
	export ROCM_CLI_ROCMD_BINARY="$CARGO_TARGET_DIR/release/rocmd"

	# Pre-warm once, serially, in place (no mv/symlink), and refresh it when
	# the channel index has moved on — the tree is a cache, not a one-shot.
	# See the e2e-gpu lane and `xtask e2e-prewarm` for why the old
	# existence-only guard froze this lane on its first runtime (EAI-8057).
	HF_HOME="$E2E_SHARED_CACHE_DIR/huggingface" \
	cargo xtask e2e-prewarm --channel release --prewarm-dir "$prewarm"

	# Optional scenario-name filter for a scoped manual dispatch.
	NAME_FILTER="${{ github.event.inputs.name_filter }}"
	if [ -n "$NAME_FILTER" ]; then
	echo "name filter active: $NAME_FILTER"
	cargo xtask e2e -- --name "$NAME_FILTER"
	else
	cargo xtask e2e
	fi

	- name: Upload E2E report
	if: always()
	uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
	with:
	name: e2e-gpu-strix-ubuntu-report
	path: tests/e2e-cucumber/results/

	# First real Windows GPU coverage: the Strix Halo Windows 11 runner. The
	# existing windows-build-and-test uses GitHub-hosted windows-latest, which has
	# no GPU. Non-blocking so it never gates the PR.
	e2e-gpu-strix-windows:
	name: E2E tests (Strix Halo, Windows)
	# 35min: see e2e-gpu — one collapsed job runs all serves + per-scenario
	# install sdk; the cap must exceed the run so the job writes platform.json.
	timeout-minutes: 35
	runs-on: [self-hosted, windows, strix-halo, native]
	needs: [changes]
	# See `e2e-gpu`: no build-and-test gate (cross-workflow); strix-windows.
	if: >-
	always()
	&& needs.changes.result == 'success'
	&& (
	(github.event_name != 'workflow_dispatch'
	&& needs.changes.outputs.serve == 'true')
	\|\| (github.event_name == 'workflow_dispatch'
	&& (inputs.platform == 'all' \|\| inputs.platform == 'strix-windows'))
	)
	continue-on-error: true
	env:
	# The three Strix lanes share one physical machine, and this workflow now
	# runs a third of them, so a TUI frame that renders well inside the 30s
	# default on an idle runner can miss it while a sibling lane loads a model.
	# Raise the wait budget rather than let that read as a product failure; a
	# genuine hang still fails, just later.
	E2E_TUI_TIMEOUT_SECS: "90"
	# Match the Linux Strix dispatch path: opt into the platform-adaptive
	# large-model scenario only when the manual input is enabled.
	E2E_INCLUDE_NIGHTLY: "${{ inputs.include_nightly && '1' \|\| '' }}"
	# Heavy @merge-queue serves run only in the merge queue; see e2e-gpu.
	E2E_MERGE_QUEUE: "${{ github.event_name == 'merge_group' && '1' \|\| '' }}"
	steps:
	- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

	# setup-rust-toolchain runs an internal bash script, which this Windows
	# runner lacks (bash: command not found). Bootstrap rustup with the
	# PowerShell-native installer instead; idempotent, so it only downloads on
	# a runner that doesn't already have the toolchain. Use `powershell`
	# (Windows PowerShell 5.1, always present) rather than `pwsh` (PowerShell 7),
	# which this self-hosted runner does not have installed.
	# Reclaim the GPU from any E2E serve leaked by a killed/timed-out prior run
	# (see e2e-gpu). PowerShell (5.1) equivalent; best-effort.
	- name: Reclaim GPU from stray E2E processes
	shell: powershell
	run: \|
	Get-CimInstance Win32_Process -ErrorAction SilentlyContinue \|
	Where-Object { $_.CommandLine -match 'rocm-e2e\|__engine-serve-http\|e2e-target\|e2e-shared' -and $_.CommandLine -match 'llama-server\|vllm\|rocm ' } \|
	ForEach-Object { Stop-Process -Id $_.ProcessId -Force -ErrorAction SilentlyContinue }
	Write-Host "reclaimed"

	# GPU preflight (best-effort on Windows): bounded wait for the GPU to be
	# available so a wedged/held GPU fails fast instead of hanging to the cap.
	# BEST-EFFORT because the Windows ROCm GPU query tool isn't verified here —
	# if no rocm-smi is found we WARN and continue rather than false-fail a
	# working runner. When rocm-smi IS present we poll free VRAM the same way as
	# the Linux jobs and fail only after the ceiling.
	- name: GPU preflight (bounded wait for an available GPU)
	shell: powershell
	run: \|
	$minFreeGiB = if ($env:GPU_PREFLIGHT_MIN_FREE_GIB) { [int]$env:GPU_PREFLIGHT_MIN_FREE_GIB } else { 8 }
	$ceilingSecs = if ($env:GPU_PREFLIGHT_CEILING_SECS) { [int]$env:GPU_PREFLIGHT_CEILING_SECS } else { 90 }
	if (-not (Get-Command rocm-smi -ErrorAction SilentlyContinue)) {
	Write-Host "rocm-smi not found on this Windows runner; skipping GPU preflight (best-effort)."
	exit 0
	}
	$minFree = [int64]$minFreeGiB * 1GB
	$deadline = (Get-Date).AddSeconds($ceilingSecs)
	$reason = "rocm-smi never returned usable VRAM figures"
	while ((Get-Date) -lt $deadline) {
	$out = (rocm-smi --showmeminfo vram 2>$null \| Out-String)
	$total = ([regex]::Matches($out, 'VRAM Total Memory $B$:\s*(\d+)') \| Select-Object -First 1).Groups[1].Value
	$used = ([regex]::Matches($out, 'VRAM Total Used Memory $B$:\s*(\d+)') \| Select-Object -First 1).Groups[1].Value
	if (-not $total -or -not $used) {
	$reason = "rocm-smi returned no VRAM figures (no AMD GPU detected)"
	Start-Sleep -Seconds 5; continue
	}
	$free = [int64]$total - [int64]$used
	if ($free -ge $minFree) {
	Write-Host "GPU ready: $([math]::Floor($free/1GB)) GiB free (>= $minFreeGiB GiB)."
	exit 0
	}
	$reason = "VRAM never dropped below the floor: only $([math]::Floor($free/1GB)) GiB free (< $minFreeGiB GiB) - a serve is likely still holding the GPU"
	Write-Host "waiting: $([math]::Floor($free/1GB)) GiB free (< $minFreeGiB GiB)..."
	Start-Sleep -Seconds 5
	}
	Write-Host "::error::GPU preflight failed after ${ceilingSecs}s: $reason"
	exit 1

	- name: Ensure Rust toolchain (PowerShell)
	shell: powershell
	run: \|
	if (-not (Get-Command cargo -ErrorAction SilentlyContinue)) {
	Invoke-WebRequest https://win.rustup.rs/x86_64 -OutFile $env:TEMP\rustup-init.exe
	# --default-toolchain none: rust-toolchain.toml pins the exact
	# version (1.96.0 + components), auto-installed on first cargo use.
	& $env:TEMP\rustup-init.exe -y --default-toolchain none
	"$env:USERPROFILE\.cargo\bin" \| Out-File -FilePath $env:GITHUB_PATH -Append
	}

	- name: Run E2E tests on Strix Halo Windows
	shell: powershell
	run: \|
	# Share ONE installed managed runtime across serve/chat scenarios, and
	# PRE-WARM it in place before the suite (mirrors e2e-gpu / strix-ubuntu).
	# REQUIRED for correctness, not just speed: `install sdk` bakes ABSOLUTE
	# paths (install_root, python_executable, rocm_sdk.*, .rocm-cli-runtime.json)
	# into the runtime manifest. If the first scenario installs into its own
	# isolated temp data dir and only the registry is shared onward, those baked
	# paths point at that scenario's dir — deleted when the scenario ends — so
	# every later serve sees status=unusable and fails (diagnosed on the Linux
	# box 2026-07-15; the Windows scenario-8 cold-download failure is the same
	# class). Install in place so the baked paths stay valid for all scenarios.
	$prewarm = "$env:RUNNER_WORKSPACE\e2e-prewarm"
	$env:E2E_SHARED_RUNTIMES_DIR = "$prewarm\data\runtimes"

	# Build the rocm and rocmd binaries once; reuse them for pre-warm + suite.
	# This job does not set CARGO_TARGET_DIR, so the binaries land in
	# the default target\release (fall back to it when the env var is unset).
	# See the e2e-gpu lane for why the e2e-test-hooks feature must match
	# what `cargo xtask e2e` would build.
	cargo build --release -p rocm -p rocmd --features rocm/e2e-test-hooks
	if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE }
	$targetDir = if ($env:CARGO_TARGET_DIR) { $env:CARGO_TARGET_DIR } else { "target" }
	$env:ROCM_CLI_BINARY = "$targetDir\release\rocm.exe"
	$env:ROCM_CLI_ROCMD_BINARY = "$targetDir\release\rocmd.exe"

	# Pre-warm once, in place (no move/symlink), and refresh it when the
	# channel index has moved on — the tree is a cache, not a one-shot. The
	# decision lives in `xtask e2e-prewarm` rather than being reimplemented
	# here in PowerShell, so this lane and the Linux lanes cannot drift
	# (EAI-8057; the old existence-only guard never reinstalled).
	cargo xtask e2e-prewarm --channel release --prewarm-dir "$prewarm"
	if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE }

	# Optional scenario-name filter for a scoped manual dispatch.
	$nameFilter = "${{ github.event.inputs.name_filter }}"
	if ($nameFilter) {
	Write-Host "name filter active: $nameFilter"
	cargo xtask e2e -- --name "$nameFilter"
	} else {
	cargo xtask e2e
	}

	- name: Upload E2E report
	if: always()
	uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
	with:
	name: e2e-gpu-strix-windows-report
	path: tests/e2e-cucumber/results/

	# WSL2 coverage on real hardware: the runner is an Ubuntu distro running under
	# WSL2 on the Strix Halo Windows box, so this lane exercises the WSL host
	# boundary AND whatever GPU access WSL exposes. Same suite as every other
	# platform — scenarios the host cannot satisfy resolve to skip from the
	# capability probe, so nothing is filtered out here. `wsl` in `runs-on`
	# disambiguates it from the `native` Strix Linux runner. Non-blocking while
	# GPU-on-WSL is proven out.
	e2e-wsl:
	name: E2E tests (Strix Halo, WSL2)
	# A manual include_nightly dispatch runs the 2400s large-model readiness
	# scenario, so match the dedicated nightly lane's cap and leave room for
	# build, runtime pre-warm, the rest of the suite, and platform.json.
	timeout-minutes: 90
	runs-on: [self-hosted, linux, strix-halo, wsl]
	needs: [changes]
	# See `e2e-gpu`: no build-and-test gate (cross-workflow); strix-wsl. Gated on
	# `serve` like the sibling lanes rather than the broader `heavy`: the
	# consolidated report gates on `serve` too, so a heavy-but-not-serve change
	# would otherwise run this lane and then discard its artifact unreported.
	if: >-
	always()
	&& needs.changes.result == 'success'
	&& (
	(github.event_name != 'workflow_dispatch'
	&& needs.changes.outputs.serve == 'true')
	\|\| (github.event_name == 'workflow_dispatch'
	&& (inputs.platform == 'all' \|\| inputs.platform == 'strix-wsl'))
	)
	continue-on-error: true
	env:
	E2E_SERVE_TIMEOUT_SECS: "300"
	# The three Strix lanes share one physical machine, and this workflow now
	# runs a third of them, so a TUI frame that renders well inside the 30s
	# default on an idle runner can miss it while a sibling lane loads a model.
	# Raise the wait budget rather than let that read as a product failure; a
	# genuine hang still fails, just later.
	E2E_TUI_TIMEOUT_SECS: "90"
	# Match the other hardware lanes: opt into the platform-adaptive
	# large-model scenario only when the manual include_nightly input is on.
	E2E_INCLUDE_NIGHTLY: "${{ inputs.include_nightly && '1' \|\| '' }}"
	# Heavy @merge-queue serves run only in the merge queue; see e2e-gpu.
	E2E_MERGE_QUEUE: "${{ github.event_name == 'merge_group' && '1' \|\| '' }}"
	steps:
	- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

	# Fail fast and loudly if this job ever lands on a native Linux runner:
	# every WSL-tagged scenario would silently resolve to skip instead.
	- name: Verify the host really is WSL2
	run: \|
	proc_version=$(cat /proc/version 2>/dev/null \|\| true)
	if [ ! -e /dev/dxg ] \
	&& [ -z "${WSL_DISTRO_NAME+x}" ] \
	&& ! printf '%s\n' "$proc_version" \| grep -qiE 'microsoft\|wsl'; then
	echo "::error::runner is not a WSL host: $(uname -r)"
	exit 1
	fi
	printf 'WSL kernel: %s\n' "$(uname -r)"

	# The native Strix runners come pre-provisioned; a WSL distro often does
	# not. Install the native build deps only when something is missing, and
	# only if passwordless sudo is available — otherwise say what is missing
	# instead of hanging on a password prompt.
	- name: Ensure native build deps
	run: \|
	missing=""
	command -v pkg-config >/dev/null 2>&1 \|\| missing="$missing pkg-config"
	command -v cc >/dev/null 2>&1 \|\| missing="$missing build-essential"
	pkg-config --exists libcap 2>/dev/null \|\| missing="$missing libcap-dev"
	if [ -z "$missing" ]; then
	echo "native build deps present"
	elif sudo -n true 2>/dev/null; then
	echo "installing:$missing"
	sudo -n apt-get update
	# shellcheck disable=SC2086
	sudo -n apt-get install -y $missing
	else
	echo "::error::missing native build deps:$missing (no passwordless sudo to install them)"
	exit 1
	fi

	- name: Reclaim GPU from stray E2E processes
	run: \|
	# Reclaim from any serve leaked by a killed/timed-out prior run
	# (see e2e-gpu). Scoped to e2e leftovers only.
	pkill -f '/tmp/rocm-e2e.*llama-server' 2>/dev/null \|\| true
	pkill -f '/tmp/rocm-e2e.*vllm serve' 2>/dev/null \|\| true
	pkill -f 'e2e-shared.*llama-server' 2>/dev/null \|\| true
	pkill -f '__engine-serve-http.*rocm-e2e' 2>/dev/null \|\| true
	rm -rf /tmp/rocm-e2e-* 2>/dev/null \|\| true
	echo "reclaimed"

	# GPU preflight, bounded like the native lanes — but ADVISORY here. GPU
	# access under WSL is exactly what this lane is proving out, so an absent
	# rocm-smi is a reported condition, not a job failure: the capability probe
	# then resolves @requires-gpu scenarios to skip and the rest still runs. A
	# GPU that IS present but stays held by a leftover serve still fails, since
	# that would corrupt the serve scenarios' results.
	- name: GPU preflight (advisory bounded wait)
	run: \|
	MIN_FREE_GIB="${GPU_PREFLIGHT_MIN_FREE_GIB:-8}"
	CEILING_SECS="${GPU_PREFLIGHT_CEILING_SECS:-90}"
	if ! command -v rocm-smi >/dev/null 2>&1; then
	echo "::warning::rocm-smi not found in this WSL distro — GPU scenarios will resolve to skip"
	exit 0
	fi
	min_free=$(( MIN_FREE_GIB * 1024 * 1024 * 1024 ))
	deadline=$(( SECONDS + CEILING_SECS ))
	saw_vram=0
	while [ "$SECONDS" -lt "$deadline" ]; do
	out=$(timeout 15 rocm-smi --showmeminfo vram 2>/dev/null) \|\| { sleep 5; continue; }
	total=$(printf '%s\n' "$out" \| grep -i 'VRAM Total Memory' \| sed 's/.: //' \| grep -oE '[0-9]+' \| tail -1)
	used=$(printf '%s\n' "$out" \| grep -i 'VRAM Total Used Memory' \| sed 's/.: //' \| grep -oE '[0-9]+' \| tail -1)
	if [ -z "$total" ] \|\| [ -z "$used" ]; then
	sleep 5; continue
	fi
	saw_vram=1
	free=$(( total - used ))
	if [ "$free" -ge "$min_free" ]; then
	echo "GPU ready: $(( free / 1024 / 1024 / 1024 )) GiB free (>= ${MIN_FREE_GIB} GiB)."
	exit 0
	fi
	echo "waiting: $(( free / 1024 / 1024 / 1024 )) GiB free (< ${MIN_FREE_GIB} GiB)…"
	sleep 5
	done
	if [ "$saw_vram" -eq 0 ]; then
	echo "::warning::rocm-smi reported no VRAM figures under WSL — GPU scenarios will resolve to skip"
	exit 0
	fi
	echo "::error::GPU preflight failed after ${CEILING_SECS}s: VRAM never dropped below the floor — a serve is likely still holding the GPU"
	exit 1

	# Bootstrap rustup with --no-modify-path so it never writes $HOME/.profile
	# (setup-rust-toolchain doesn't expose that flag). rust-toolchain.toml pins
	# the exact toolchain, installed on first cargo use. Idempotent.
	- name: Ensure Rust toolchain
	run: \|
	if ! command -v cargo >/dev/null 2>&1 && [ ! -x "$HOME/.cargo/bin/cargo" ]; then
	curl --proto '=https' --tlsv1.2 -fsSL https://sh.rustup.rs \
	\| sh -s -- -y --no-modify-path --default-toolchain none
	fi
	echo "$HOME/.cargo/bin" >> "$GITHUB_PATH"

	- name: Run E2E tests on Strix Halo WSL2
	run: \|
	export CARGO_TARGET_DIR="$RUNNER_WORKSPACE/e2e-target"
	export E2E_SHARED_CACHE_DIR="$RUNNER_WORKSPACE/e2e-shared"
	# Share ONE installed managed runtime across serve/chat scenarios and
	# PRE-WARM it in place (mirrors e2e-gpu-strix-ubuntu). Required for
	# correctness, not speed: `install sdk` bakes ABSOLUTE paths into the
	# runtime manifest, so installing into a per-scenario temp dir leaves
	# every later serve pointing at a deleted install root.
	prewarm="$RUNNER_WORKSPACE/e2e-prewarm"
	export E2E_SHARED_RUNTIMES_DIR="$prewarm/data/runtimes"

	# Build the rocm and rocmd binaries once; reuse them for pre-warm + suite.
	# See the e2e-gpu lane for why the e2e-test-hooks feature must match
	# what `cargo xtask e2e` would build.
	cargo build --release -p rocm -p rocmd --features rocm/e2e-test-hooks
	export ROCM_CLI_BINARY="$CARGO_TARGET_DIR/release/rocm"
	export ROCM_CLI_ROCMD_BINARY="$CARGO_TARGET_DIR/release/rocmd"

	# Pre-warm once, serially, in place (no mv/symlink), and refresh it when
	# the channel index has moved on — the tree is a cache, not a one-shot.
	# See the e2e-gpu lane and `xtask e2e-prewarm` for why the old
	# existence-only guard froze this lane on its first runtime (EAI-8057).
	HF_HOME="$E2E_SHARED_CACHE_DIR/huggingface" \
	cargo xtask e2e-prewarm --channel release --prewarm-dir "$prewarm"

	# Optional scenario-name filter for a scoped manual dispatch.
	NAME_FILTER="${{ github.event.inputs.name_filter }}"
	if [ -n "$NAME_FILTER" ]; then
	echo "name filter active: $NAME_FILTER"
	cargo xtask e2e -- --name "$NAME_FILTER"
	else
	cargo xtask e2e
	fi

	- name: Upload E2E report
	if: always()
	uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
	with:
	name: e2e-gpu-strix-wsl-report
	path: tests/e2e-cucumber/results/

	# Third AMD GPU target: the rad3 cluster's Radeon AI PRO R9700S (gfx1201, 32GB),
	# targeted by the `r9700` label. This runner is a KEDA-scaled pod (0<->1 off the
	# job queue), so the card is only reserved while a job exists (dev-tooling EAI-8065).
	# Non-blocking while this hardware is proven out.
	e2e-gpu-rad3:
	name: E2E tests (rad3 R9700)
	# 90min: matches e2e-gpu — same collapsed suite, and the card is smaller, so
	# do not tighten this below the Instinct lane's cap.
	timeout-minutes: 90
	# `r9700` names the hardware, not a generic `amd-gpu`: GitHub matches a job to
	# any runner whose labels are a superset, so a shared label would let this card
	# pick up MI300X-targeted work.
	runs-on: [self-hosted, linux, r9700]
	needs: [changes]
	if: >-
	always()
	&& needs.changes.result == 'success'
	&& (
	(github.event_name != 'workflow_dispatch'
	&& needs.changes.outputs.serve == 'true')
	\|\| (github.event_name == 'workflow_dispatch'
	&& (inputs.platform == 'all' \|\| inputs.platform == 'rad3'))
	)
	continue-on-error: true
	env:
	E2E_SERVE_TIMEOUT_SECS: "300"
	E2E_INCLUDE_NIGHTLY: "${{ inputs.include_nightly && '1' \|\| '' }}"
	# Full serve matrix at the merge-queue gate, like every other lane (see e2e-gpu).
	E2E_MERGE_QUEUE: "${{ github.event_name == 'merge_group' && '1' \|\| '' }}"
	steps:
	- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

	# The pod is usually fresh (scale-to-zero recreates it per job), but the
	# work PVC is not, and two jobs can share one pod's life. Same reclaim as
	# e2e-gpu, scoped to e2e leftovers only.
	- name: Reclaim GPU from stray E2E processes
	run: \|
	pkill -f '/tmp/rocm-e2e.*llama-server' 2>/dev/null \|\| true
	pkill -f '/tmp/rocm-e2e.*vllm serve' 2>/dev/null \|\| true
	pkill -f 'e2e-shared.*llama-server' 2>/dev/null \|\| true
	pkill -f '__engine-serve-http.*rocm-e2e' 2>/dev/null \|\| true
	pkill -f 'e2e-target/release/rocm daemon' 2>/dev/null \|\| true
	rm -rf /tmp/rocm-e2e-* 2>/dev/null \|\| true
	echo "reclaimed"

	# Bounded wait, as in e2e-gpu. The pod requests `amd.com/gpu: 1`, so
	# rocm-smi sees exactly the one card Kueue admitted it for. The floor is
	# half of the R9700S's 32GB: high enough to catch a leftover serve still
	# holding the card, low enough that a clean pod passes on the first poll.
	- name: GPU preflight (bounded wait for an available GPU)
	run: \|
	MIN_FREE_GIB="${GPU_PREFLIGHT_MIN_FREE_GIB:-16}"
	CEILING_SECS="${GPU_PREFLIGHT_CEILING_SECS:-90}"
	min_free=$(( MIN_FREE_GIB * 1024 * 1024 * 1024 ))
	deadline=$(( SECONDS + CEILING_SECS ))
	reason="rocm-smi never returned within its timeout (driver wedged or GPU absent)"
	while [ "$SECONDS" -lt "$deadline" ]; do
	out=$(timeout 15 rocm-smi --showmeminfo vram 2>/dev/null) \|\| { sleep 5; continue; }
	total=$(printf '%s\n' "$out" \| grep -i 'VRAM Total Memory' \| sed 's/.: //' \| grep -oE '[0-9]+' \| tail -1)
	used=$(printf '%s\n' "$out" \| grep -i 'VRAM Total Used Memory' \| sed 's/.: //' \| grep -oE '[0-9]+' \| tail -1)
	if [ -z "$total" ] \|\| [ -z "$used" ]; then
	reason="rocm-smi returned no VRAM figures (no AMD GPU detected)"
	sleep 5; continue
	fi
	free=$(( total - used ))
	if [ "$free" -ge "$min_free" ]; then
	echo "GPU ready: $(( free / 1024 / 1024 / 1024 )) GiB free (>= ${MIN_FREE_GIB} GiB)."
	exit 0
	fi
	reason="VRAM never dropped below the floor: only $(( free / 1024 / 1024 / 1024 )) GiB free (< ${MIN_FREE_GIB} GiB) — a serve is likely still holding the GPU"
	echo "waiting: $(( free / 1024 / 1024 / 1024 )) GiB free (< ${MIN_FREE_GIB} GiB)…"
	sleep 5
	done
	echo "::error::GPU preflight failed after ${CEILING_SECS}s: ${reason}"
	exit 1

	# cache: false for the same reason as e2e-gpu — the build cache is kept in
	# CARGO_TARGET_DIR on the PVC, not in GitHub's cache service.
	- uses: actions-rust-lang/setup-rust-toolchain@166cdcfd11aee3cb47222f9ddb555ce30ddb9659 # v1.17.0
	with:
	cache: false

	- name: Run E2E tests on rad3
	run: \|
	# $RUNNER_WORKSPACE is on the runner-work PVC, which survives the pod
	# being scaled away, so the cargo target dir and the shared caches are
	# still warm on the next job even though the container is not.
	export CARGO_TARGET_DIR="$RUNNER_WORKSPACE/e2e-target"
	export E2E_SHARED_CACHE_DIR="$RUNNER_WORKSPACE/e2e-shared"
	# A SEPARATE PVC is mounted at exactly this path by the runner's cluster
	# overlay. Keeping the ~23GB uv cache off the work PVC is deliberate; if
	# you change this path, change the overlay that mounts it too.
	export E2E_SHARED_UV_CACHE_DIR="/var/tmp/rocm-e2e-uv-cache"
	# Pre-warm one shared runtime in place before the suite. Not an
	# optimization — `install sdk` bakes absolute paths into the runtime
	# manifest, so installing anywhere temporary breaks every later serve.
	# See e2e-gpu for the full rationale.
	prewarm="$RUNNER_WORKSPACE/e2e-prewarm"
	export E2E_SHARED_RUNTIMES_DIR="$prewarm/data/runtimes"

	# See the e2e-gpu lane for why the e2e-test-hooks feature must match
	# what `cargo xtask e2e` would build.
	cargo build --release -p rocm -p rocmd --features rocm/e2e-test-hooks
	export ROCM_CLI_BINARY="$CARGO_TARGET_DIR/release/rocm"
	export ROCM_CLI_ROCMD_BINARY="$CARGO_TARGET_DIR/release/rocmd"

	# Pre-warm once, serially, in place (no mv/symlink), and refresh it when
	# the channel index has moved on — the tree is a cache, not a one-shot.
	# See the e2e-gpu lane and `xtask e2e-prewarm` for why the old
	# existence-only guard froze this lane on its first runtime (EAI-8057).
	HF_HOME="$E2E_SHARED_CACHE_DIR/huggingface" \
	UV_CACHE_DIR="$E2E_SHARED_UV_CACHE_DIR" \
	cargo xtask e2e-prewarm --channel release --prewarm-dir "$prewarm"

	# Optional scenario-name filter for a scoped manual dispatch — a cheap
	# way to exercise this lane against a single scenario without the full suite.
	NAME_FILTER="${{ github.event.inputs.name_filter }}"
	if [ -n "$NAME_FILTER" ]; then
	echo "name filter active: $NAME_FILTER"
	cargo xtask e2e -- --name "$NAME_FILTER"
	else
	cargo xtask e2e
	fi

	- name: Upload E2E report
	if: always()
	uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
	with:
	name: e2e-gpu-rad3-report
	path: tests/e2e-cucumber/results/

	# Consolidate this workflow's self-hosted platform reports into one GPU-side
	# cross-platform grid (Summary + merged HTML). Distinct name from ci.yml's
	# required `E2E consolidated report` so it does NOT collide with that required
	# check; this one is advisory and lives with the lanes it summarizes. Runs on
	# GitHub-hosted ubuntu (no GPU needed — it only parses report.json). `always()`
	# so a failing/skipped self-hosted platform still appears in the grid.
	e2e-report:
	name: E2E consolidated report (self-hosted)
	runs-on: ubuntu-latest
	timeout-minutes: 15
	needs:
	- changes
	- e2e-gpu
	- e2e-gpu-strix-ubuntu
	- e2e-gpu-strix-windows
	- e2e-wsl
	- e2e-gpu-rad3
	# Gate on `serve`: every lane this report consolidates (the GPU jobs) is now
	# serve-gated, so a serve-only change runs them and their report must still be
	# produced. On dispatch `serve` is unset, so also run when the trigger was
	# manual; `always()` still lets it collect partial/failed tiers.
	if: >-
	always()
	&& (needs.changes.outputs.serve == 'true'
	\|\| github.event_name == 'workflow_dispatch')
	steps:
	- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

	- uses: actions-rust-lang/setup-rust-toolchain@166cdcfd11aee3cb47222f9ddb555ce30ddb9659 # v1.17.0

	# Pull every self-hosted e2e artifact matching the glob; `xtask e2e-report`
	# turns each into one labeled platform and the `*-report` glob picks up any
	# new self-hosted platform. Layout note: download-artifact@v8 uses a
	# per-artifact subdir (e2e-artifacts/<name>/) when several match, but
	# flattens into e2e-artifacts/ when EXACTLY ONE matches (its source picks
	# the root path when `artifacts.length === 1`, regardless of merge-multiple).
	# A single-platform dispatch hits the flattened case; `discover()` handles
	# both, labeling a root-level report from platform.json's slug.
	- name: Download all self-hosted E2E reports
	uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
	with:
	pattern: '*-report'
	path: e2e-artifacts

	- name: Build consolidated report + step summary
	run: \|
	mkdir -p consolidated
	cargo xtask e2e-report \
	--artifacts-dir e2e-artifacts \
	--html-out consolidated/index.html >> "$GITHUB_STEP_SUMMARY"

	- name: Upload consolidated report
	if: always()
	uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
	with:
	name: e2e-consolidated-report-selfhosted
	path: consolidated/

Navigation Menu

Uh oh!

fix(install): always keep the SDK's build of the torch an engine pins #661

Workflow file

fix(install): always keep the SDK's build of the torch an engine pins #661

Uh oh!

Workflow file for this run