test(e2e): add GPU end-to-end suite for dynamo, vllm, kaito

[surajssd]

Description

Adds a consolidated, GPU-cluster end-to-end test suite (test/e2e/gpu/) that deploys each inference provider — dynamo, vllm, and kaito — through a real ModelDeployment, drives it to Running, and asserts that inference actually serves through the inference gateway. The suite is a zero-dependency Go module driven by a thin Bash orchestrator (scripts/gpu-e2e.sh), with its cluster-free decision logic carved into unit-testable packages that run in CI on a plain runner. It supersedes the old single-provider TestDynamoProviderE2E, porting its deep assertions into the new table-driven matrix. The workflow is documented in docs/development.md under GPU End-to-End Testing.

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)
• ✨ New feature (non-breaking change that adds functionality)
• 💥 Breaking change (fix or feature that would cause existing functionality to change)
• 📚 Documentation update
• 🎨 UI/UX improvement
• ♻️ Refactoring (no functional changes)
• 🧪 Test update
• 🔧 Build/CI configuration

Related Issues

• Relates to kaito-project/airunway#334 — BBR builds its model registry only at startup, so the controller rolling-restarts the shared BBR Deployment once per new ModelDeployment (tracked by the airunway.ai/bbr-restarted annotation). The restart is not zero-downtime: during it, an in-flight request for an already-serving model can miss its X-Gateway-Model-Name header and mis-route. This is documented as a known gateway limitation in docs/gateway.md, and is why disaggregated Dynamo serving is excluded from the v1 matrix.

Changes Made

GPU end-to-end suite (test/e2e/gpu/)

• New zero-dependency Go module gated by the e2e build tag. A single table-driven TestGPUProviders runs every (provider × scenario) case as a parallel subtest through a uniform lifecycle: apply fixture → wait for the rendered upstream CR → scheduling classification → Running + provider-name check → GatewayReady → provider-specific assertions → inference via the gateway → teardown.
• v1 matrix covers three aggregated cases: vllm/agg (deployments.apps), kaito/agg (workspaces.kaito.sh), and dynamo/agg (dynamographdeployments.nvidia.com), each with a fixture under testdata/. Adding a new (provider × scenario) case is a data-only change to the cases table.
• TestMain enforces cheap preconditions (≥1 allocatable nvidia.com/gpu, gateway Programmed) and fails fast.
• Two-phase scheduling classifier produces three-state PASS/FAIL/SKIP outcomes: a static permanent-unschedulable check (per-pod GPU demand vs. largest node) plus a deadline-bounded poll that distinguishes "not scheduled" (PodScheduled=False) from "scheduled, pulling image."
• Teardown runs as t.Cleanup so each parallel case frees its GPU as soon as it finishes; a graceful ModelDeployment delete is followed by assertNoOrphans (upstream CR, Dynamo PVC, and download Job are garbage-collected), with a timeout-only force-cascade fallback. Per-case logs and a result marker are written under the results directory.

Orchestration & build

• scripts/gpu-e2e.sh builds and pushes the controller + provider images in parallel, gates setup-<provider> on operator health, deploys, then invokes the Go suite. It never creates or deletes the cluster. KAITO detection recognizes both the Helm chart and the AKS AI-toolchain add-on (kube-system), mirroring providers/kaito/upstream_health.go. HF_TOKEN is passed to kubectl create secret via stdin so it never lands in process argv.
• Root Makefile gains gpu-e2e (full run, flags via GPU_E2E_ARGS) and gpu-e2e-check (cluster-free gate).

CI

• New gpu-e2e-check job in .github/workflows/test.yml runs gofmt, go vet, an -tags=e2e compile-check, and the cluster-free unit tests on a plain ubuntu-latest runner, so the GPU-coupled suite cannot rot between out-of-band GPU runs.

Cluster-free logic extracted for unit testing

• sched package (UnschedulableReason, PodScheduledMessage, PodInfo, GPUResource) and e2eutil helpers (parseChatResponse, InjectStorageClass) are pure, tag-free functions with table-driven tests — exercising the classifier, the chat-response parser, and the storage-class injector without a cluster.

Inference reachability fix

• assertInference reaches the gateway through a kubectl port-forward to svc/inference-gateway-istio (e2eutil.PortForwardService) rather than the external LoadBalancer IP, so it works from machines whose egress to that IP is blocked by network policy. The port-forward uses a readiness poll instead of a fixed sleep and re-establishes itself via EnsureReady if the tunnel drops mid-window.

Cleanup of superseded test

• Removes TestDynamoProviderE2E and its exclusive helpers from providers/dynamo/test/e2e/; its PVC / download-Job / DGD-ownership assertions are ported into the new dynamo case. The Dynamo mocker, multinode, and storage-validation tests are retained, and a stale Makefile comment is corrected.

Docs

• docs/development.md gains a ## GPU End-to-End Testing section: the workflow, cluster preconditions (GPU nodes + NFD, an RWX-capable StorageClass, the inference gateway, image pull access), the run commands, the GPU_E2E_* environment knobs, and the PASS/FAIL/SKIP outcome semantics.
• docs/gateway.md documents the shared-BBR restart race (kaito-project/airunway#334).

Testing

The full suite requires a pre-provisioned GPU cluster and runs out-of-band via scripts/gpu-e2e.sh (see docs/development.md). CI runs only the cluster-free gpu-e2e-check gate.

# All three providers, building+pushing images to your registry:
make gpu-e2e GPU_E2E_ARGS="--provider all --registry <your-registry>"

# A single provider:
make gpu-e2e GPU_E2E_ARGS="--provider vllm --registry <your-registry>"

# Re-test without rebuilding (requires an explicit, already-pushed tag):
make gpu-e2e GPU_E2E_ARGS="--provider dynamo --skip-build \
    --registry <your-registry> --img-tag <tag>"

# Run the Go suite directly against an already-deployed cluster (no rebuild):
go test -C test/e2e/gpu -tags=e2e -v -run 'TestGPUProviders/vllm' ./

# Cluster-free gate (what CI runs): gofmt + go vet + -tags=e2e compile + unit tests:
make gpu-e2e-check

• Unit tests pass (bun run test) — N/A: this branch is a standalone Go module, not the web UI. The equivalent gate is make gpu-e2e-check, which passes locally (sched and e2eutil green, gofmt clean, -tags=e2e compile clean).
• Manual testing performed — iterate-to-green against a live cluster; all three aggregated cases reach Running, GatewayReady, and serve inference. A SKIP (insufficient GPU capacity) does not fail the run; only a broken deployment, failed inference, or orphaned resource after delete is a FAIL.
• Tested with a Kubernetes cluster — 4×A100 80GB AKS cluster (southcentralus).

Checklist

• My code follows the project's style guidelines (gofmt clean, go vet clean)
• I have run bun run lint — N/A for this Go module; go vet is wired into gpu-e2e-check instead.
• I have added tests that prove my fix/feature works
• New and existing unit tests pass locally (sched, e2eutil green)
• I have updated documentation if needed (docs/development.md, docs/gateway.md)
• My changes generate no new warnings

Additional Notes

Cluster preconditions (the harness installs none of these except a missing operator via setup-<p>):

• GPU nodes with the NVIDIA GPU Operator and NFD, advertising nvidia.com/gpu and the nvidia.com/gpu.present=true label.
• An RWX-capable StorageClass. The Dynamo model-cache PVC defaults to ReadWriteMany; Azure Disk classes are ReadWriteOnce and leave the PVC Pending. Default is azurefile-premium; override with --storage-class.
• The inference gateway (Gateway API CRDs + GAIE + Istio + BBR + a Gateway named inference-gateway), present and Programmed. make -C providers/dynamo setup-dynamo installs it on a fresh cluster.
• Pull access to the pushed images. The manager manifests carry no imagePullSecret, so images must be public or the nodes must have pull access — new registry repos often default to private.

Environment knobs (forwarded by the script; can also be set directly for go test):

Variable	Meaning
GPU_E2E_STORAGE_CLASS	RWX StorageClass injected into the Dynamo fixture and asserted on (default azurefile-premium). Set by --storage-class.
GPU_E2E_KEEP	When true, leave ModelDeployments running after the test for inspection. Set by --keep.
GPU_E2E_RESULTS_DIR	Override for where per-case result bundles are written (default test/e2e/gpu/gpu-e2e-results/<timestamp>/).
GPU_E2E_RUN_TS	Optional fixed timestamp for the results directory name.

• The new test/e2e/gpu module has zero external dependencies (stdlib only), so it has a go.mod but no go.sum; the CI cache key is test/e2e/gpu/go.mod.
• KubeRay is not yet covered by the suite.

[Copilot]

Pull request overview

This PR adds a consolidated, GPU-cluster end-to-end test harness for the airunway inference providers (Dynamo, vLLM, KAITO). It deploys each provider through a real ModelDeployment, drives it to Running, and asserts inference actually serves through the inference gateway — closing the coverage gap where Dynamo's GPU e2e test was unused and vLLM/KAITO had none. The suite is a standalone, dependency-free Go module (build tag e2e) orchestrated by scripts/gpu-e2e.sh/make gpu-e2e, and is table-driven so future scenarios are data-only additions.

Changes:

• New test/e2e/gpu/ Go module: table-driven TestGPUProviders running dynamo/agg, vllm/agg, kaito/agg through a uniform lifecycle (pre-delete → apply → upstream CR → schedule classification → Running → GatewayReady → provider checks → inference), with kubectl-shelling helpers, scheduling/teardown/results logic, and three fixtures.
• New scripts/gpu-e2e.sh orchestration (build+push controller/provider images, gate operator install on health, deploy, run the suite), a gpu-e2e Makefile target, and a .gitignore entry for result bundles.
• Removed the superseded TestDynamoProviderE2E and its exclusive helpers/constants from providers/dynamo/test/e2e/dynamo_e2e_test.go (its deep assertions were ported into the new dynamo/agg case).

Reviewed changes

Copilot reviewed 16 out of 17 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
test/e2e/gpu/gpu_e2e_test.go	Table-driven lifecycle orchestration for each provider case.
test/e2e/gpu/cases_test.go	Test matrix (provider, fixture, upstream CR, pod selector).
test/e2e/gpu/main_test.go	TestMain GPU + gateway preconditions.
test/e2e/gpu/scheduling_test.go	Phase-1 scheduling classification (PASS/FAIL/SKIP).
test/e2e/gpu/lifecycle_test.go	Fixture apply/patch, pre-delete, and cleanup helpers.
test/e2e/gpu/teardown_test.go	Owner-first force-cascade + debug collection.
test/e2e/gpu/results_test.go	Per-case PASS/FAIL/SKIP artifact bundles.
test/e2e/gpu/dynamo_test.go	Ported Dynamo deep assertions (PVC, Job, ownership, conditions).
test/e2e/gpu/e2eutil/e2eutil.go	Dependency-free kubectl/HTTP helpers.
test/e2e/gpu/go.mod	New module declaration (go 1.25.3, consistent with repo).
test/e2e/gpu/testdata/*.yaml	Dynamo/vLLM/KAITO ModelDeployment fixtures.
scripts/gpu-e2e.sh	Build/deploy/run orchestration harness.
providers/dynamo/test/e2e/dynamo_e2e_test.go	Removes superseded TestDynamoProviderE2E and exclusive helpers.
Makefile	Adds gpu-e2e target and help entry.
.gitignore	Ignores gpu-e2e-results/.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 18 out of 19 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 19 out of 20 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 24 out of 25 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 24 out of 25 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 24 out of 25 changed files in this pull request and generated no new comments.