feat(ci): onboard GB200 AWS UAT reservation#1774
Conversation
njhensley
added
the
theme/ci-dx
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Enterprise Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughThe UAT AWS workflow now accepts an accelerator, selects matching test configurations, derives the AWS account from cluster tenancy, and builds platform-specific images. The UAT runner forwards accelerator data from reservations. A new Estimated code review effort: 4 (Complex) | ~45 minutes Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
.github/workflows/uat-aws.yaml (1)
121-121: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick winExport tenancy before the config guard.
TEST_CONFIGis checked beforeAWS_ACCOUNT_IDis written toGITHUB_ENV, so a missing accelerator/intent config can stop the teardown-onlydaytime-downpath before it can assume the role. Derive and export tenancy first, and skip theTEST_CONFIGexistence check fordaytime-down.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/uat-aws.yaml at line 121, Update the workflow setup around TEST_CONFIG to derive and export AWS_ACCOUNT_ID before validating the configuration file. Bypass the TEST_CONFIG existence guard when inputs.intent is daytime-down, while retaining the existing guard for other intents so teardown can assume the role without a test config.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In @.github/workflows/uat-aws.yaml:
- Line 121: Update the workflow setup around TEST_CONFIG to derive and export
AWS_ACCOUNT_ID before validating the configuration file. Bypass the TEST_CONFIG
existence guard when inputs.intent is daytime-down, while retaining the existing
guard for other intents so teardown can assume the role without a test config.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 3f5f75a0-7418-4a0b-8019-0b91102bd884
📒 Files selected for processing (1)
.github/workflows/uat-aws.yaml
mchmarny
left a comment
mchmarny
left a comment
There was a problem hiding this comment.
The cross-account wiring is coherent, but two runtime paths remain blocked: main-cell images do not match the GB200 worker architecture, and teardown can fail before account discovery. Required CI is green, but it does not exercise either path.
| workers: | ||
| - name: gpu-worker | ||
| instanceType: p6e-gb200.36xlarge | ||
| architecture: arm64 # GB200 = Grace/arm64 (arm64 Ubuntu EKS AMI) |
There was a problem hiding this comment.
blocker: This makes the GPU workers arm64, but the AWS main cell builds the snapshot-agent image only for linux/amd64, and the new test config pins that agent to this gpu-worker pool. The pod cannot start, so the default main UAT path fails before recipe generation. Build and push arm64/multi-arch images here, and verify the validator images scheduled on GPU workers support arm64 too.
There was a problem hiding this comment.
Addressed. Main-cell image builds are now arch-aware via a new BUILD_PLATFORMS job env — linux/amd64,linux/arm64 for GB200, linux/amd64 for h100 (so the common nightly path skips the QEMU cross-build).
- Snapshot-agent:
ko build --platform="${BUILD_PLATFORMS}"(Go cross-compile, emits a manifest list). - Validators: converted to
docker buildx build --platform="${BUILD_PLATFORMS}" --push, withSet up QEMUgated oncontains(env.BUILD_PLATFORMS, 'arm64')andSet up Docker Buildx.
On the validator arm64 point: the release pipeline (on-push.yaml/on-tag.yaml) already builds these exact phases (deployment/performance/conformance/aiperf-bench) for linux/amd64 linux/arm64 with an EXPECTED_PLATFORMS assertion, so the Dockerfiles are proven arm64-capable — this change applies the same multi-arch build to main cells. Release cells are unaffected (they pull the already-multi-arch released images).
| @@ -154,7 +167,20 @@ jobs: | |||
| if [[ ! -f "${TEST_CONFIG}" ]]; then | |||
There was a problem hiding this comment.
blocker: This guard runs for daytime-down and before AWS_ACCOUNT_ID is exported below. If the accelerator/intent config is absent, the job fails before teardown re-authentication can construct the role ARN, potentially leaving the cluster and Capacity Block in use. Derive/export tenancy first, and skip the TEST_CONFIG existence check for daytime-down.
There was a problem hiding this comment.
Addressed in the "Validate inputs" step. AWS_ACCOUNT_ID is now derived from .deployment.tenancy and exported to $GITHUB_ENV first, before any lifecycle-specific gate can exit — so daytime-down teardown can always construct the role ARN and re-authenticate. The TEST_CONFIG existence check is now guarded with [[ "${LIFECYCLE}" != "daytime-down" && ! -f "${TEST_CONFIG}" ]], so a missing/absent config no longer fails the job before teardown and can no longer strand the cluster + Capacity Block.
Add the aws-gb200 reservation row (cross-account GB200 Capacity Block), its EKS cluster-config and Kubeflow training test-config, and generalize the AWS UAT pipeline's test-config selection from h100-<intent> to <accelerator>-<intent> so per-accelerator configs are picked up. Derive AWS_ACCOUNT_ID from the cluster-config's .deployment.tenancy rather than a caller-supplied input: the OIDC role can only be assumed into the account being provisioned, removing the two-source drift risk that a separate account input carried. Signed-off-by: Nathan Hensley <nhensley@nvidia.com>
Address review on NVIDIA#1774. - Derive+export AWS_ACCOUNT_ID (from .deployment.tenancy) before any lifecycle gate, and skip the TEST_CONFIG existence check for daytime-down, so a missing config can't block teardown re-auth and strand the Capacity Block. - Build main-cell images (snapshot-agent via ko, validators via buildx) for BUILD_PLATFORMS: multi-arch amd64+arm64 for GB200 (arm64 GPU pool), single-arch amd64 for h100. QEMU is set up only when cross-building arm64, so the common h100 nightly path is unaffected. Signed-off-by: Nathan Hensley <nhensley@nvidia.com>
ace4ff5 to
2536a45
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/uat/aws/cluster-config-gb200.yaml`:
- Around line 19-23: Update the cross-account comment in the deployment
configuration to remove the stale reference to the reservations.yaml account
field. Keep the guidance that the pipeline derives the account from
.deployment.tenancy and preserve the existing tenancy value and prerequisite
details.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 4cea57a8-c1ea-4d03-9621-f8d6391e6820
📒 Files selected for processing (6)
.github/workflows/uat-aws.yaml.github/workflows/uat-run.yamlinfra/uat/reservations.yamlpkg/uatbroker/registry_test.gotests/uat/aws/cluster-config-gb200.yamltests/uat/aws/tests/gb200-training-config.yaml
The aws-gb200 reservation carries no account field; the AWS pipeline derives AWS_ACCOUNT_ID from .deployment.tenancy, the single source of truth. Correct the cross-account header note accordingly (CodeRabbit). Signed-off-by: Nathan Hensley <nhensley@nvidia.com>
2 participants
Summary
Onboard the
aws-gb200UAT reservation (cross-account GB200 Capacity Block) and generalize the AWS UAT pipeline to select test configs by<accelerator>-<intent>.Motivation / Context
The AWS UAT pipeline hardcoded
h100-<intent>-config.yamland a fixed account, so it could only run the standard-tenancy H100 reservation. GB200 capacity lives in a separate tenancy (a time-bounded Capacity Block), so enabling it requires both per-accelerator test-config selection and cross-account role assumption.Fixes: N/A
Related: #1274
Type of Change
Component(s) Affected
.github/workflows/uat-*.yaml), reservation registry (infra/uat/reservations.yaml,pkg/uatbroker), UAT test configs (tests/uat/aws)Implementation Notes
aws-gb200ininfra/uat/reservations.yaml: cross-account GB200 Capacity Block, opted out of the nightly batch (nightly-intents: []) during bring-up — manual dispatch only viauat-run.yaml.tests/uat/aws/cluster-config-gb200.yaml: Local-Zone, arm64/Grace, Capacity-Block EKS cluster;deployment.tenancy: 897722687756.tests/uat/aws/tests/gb200-training-config.yaml: mirrors the H100 training config;platform: kubeflowis load-bearing so the NCCL validators launch via the tolerated Kubeflow Trainer on the all-tainted cluster.TEST_CONFIGas<accelerator>-<intent>-config.yaml;acceleratoris plumbed from the reservation throughuat-run.yaml.AWS_ACCOUNT_IDis now read from the cluster-config's.deployment.tenancy(the single source of truth) and exported to$GITHUB_ENV, instead of being a caller input. This makes it impossible to assume the OIDC role into a different account than the one being provisioned — the previous design carried both anaccountinput and a drift-check assertion; both are removed.Testing
TestCommittedRegistryValidcovers the newaws-gb200row (cloud +nightly-intents: []guard). The cross-account run itself validates end-to-end via a manualuat-run.yamldispatch once the AWS account bootstrap lands (GitHub OIDC provider + role in897722687756).Risk Assessment
acceleratordefaults toh100, tenancy derivation yields the same615299774277).aws-gb200is opted out of the nightly batch, so no nightly capacity is provisioned until explicitly enrolled.Rollout notes: Requires
infra/uat-aws-accountapplied in account897722687756(GitHub OIDC provider +github-actions-role-aicr) before the first GB200 dispatch. Runs must fall inside the Capacity Block's active window.Checklist
make testwith-race)make lint)git commit -S)