Skip to content

Commit 7e54fca

Browse files
authored
feat(slinky-slurm): add AKS H100 recipe, Enroot/Pyxis support, and Gres/GPU config (#1744)
Signed-off-by: Kayne Tu <kaynet@nvidia.com>
1 parent 51232cd commit 7e54fca

64 files changed

Lines changed: 1072 additions & 1753 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.github/workflows/uat-aws.yaml

Lines changed: 2 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -77,9 +77,8 @@ jobs:
7777
# counts the larger (serve). The teardown MUST fit in this budget: a job-level
7878
# timeout cancels pending always() steps, so an undersized cap would skip
7979
# teardown and leak the GPU node / capacity reservation. 190 UAT + ~30 setup +
80-
# ~8 failure-path debug collection (Collect cluster debug bundle) + ~40
81-
# teardown = ~268, so 280 leaves the teardown headroom on a worst-case run.
82-
timeout-minutes: 280
80+
# ~40 teardown = ~260, so 270 leaves the teardown headroom on a worst-case run.
81+
timeout-minutes: 270
8382
permissions:
8483
contents: read
8584
actions: read
@@ -625,35 +624,6 @@ jobs:
625624
retention-days: 90
626625
if-no-files-found: error
627626

628-
# Refresh the AWS session before collecting: a long tuning-race failure
629-
# (exactly what this bundle targets) can outlast the 1h session minted before
630-
# validate, which would leave every kubectl in the collector failing EKS
631-
# exec-auth and the bundle silently empty. id-token: write is job-wide, so
632-
# configure-aws-credentials can re-assume here.
633-
- name: Refresh AWS credentials before debug collection
634-
if: failure() && steps.prep.outcome != 'skipped'
635-
uses: aws-actions/configure-aws-credentials@517a711dbcd0e402f90c77e7e2f81e849156e31d # v6.2.2
636-
with:
637-
role-to-assume: "arn:aws:iam::${{ env.AWS_ACCOUNT_ID }}:role/${{ env.GITHUB_ACTIONS_ROLE_NAME }}"
638-
aws-region: ${{ env.AWS_REGION }}
639-
role-session-name: GitHubActions-UAT-AICR-Debug
640-
641-
# Snapshot live cluster state on failure BEFORE teardown, while the
642-
# cluster is still up. The runner-local artifacts above (report.json et al.)
643-
# say WHAT check failed; this captures WHY — node conditions/taints, cluster
644-
# events (reboots, pod evictions), the operator CRs (incl. Skyhook
645-
# status.status the readiness gate keys off of), and operator/check-Job pod
646-
# logs. Best-effort (the collector guards every kubectl call) and bounded so
647-
# it never eats the teardown budget or leaks the GPU node.
648-
- name: Collect cluster debug bundle
649-
if: failure() && steps.prep.outcome != 'skipped'
650-
timeout-minutes: 8
651-
shell: bash
652-
env:
653-
AICR_BIN: ${{ github.workspace }}/aicr
654-
RUN_ID: ${{ github.run_id }}
655-
run: ./tests/uat/aws/run debug "${TEST_CONFIG}" || true
656-
657627
- name: Upload failure debug
658628
if: failure() && steps.prep.outcome != 'skipped'
659629
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
@@ -668,7 +638,6 @@ jobs:
668638
evidence/pointer.yaml
669639
train-logs/**
670640
serve-logs/**
671-
cluster-debug/**
672641
retention-days: 30
673642
if-no-files-found: ignore
674643

.github/workflows/uat-azure.yaml

Lines changed: 3 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -77,10 +77,9 @@ jobs:
7777
# each = up to ~40m). Only one of train/serve runs per intent, so the budget
7878
# counts the larger (serve). The teardown MUST fit in this budget: a job-level
7979
# timeout cancels pending always() steps, so an undersized cap would skip
80-
# teardown and leak the GPU node. 190 UAT + ~30 setup + ~8 failure-path debug
81-
# collection (Collect cluster debug bundle) + ~40 teardown = ~268, so 280
82-
# leaves the teardown headroom even on a worst-case phase run.
83-
timeout-minutes: 280
80+
# teardown and leak the GPU node. 190 UAT + ~30 setup + ~40 teardown = ~260,
81+
# so 270 leaves the teardown headroom even on a worst-case phase run.
82+
timeout-minutes: 270
8483
permissions:
8584
contents: read
8685
actions: read
@@ -699,22 +698,6 @@ jobs:
699698
retention-days: 90
700699
if-no-files-found: error
701700

702-
# Snapshot live cluster state on failure BEFORE teardown, while the
703-
# cluster is still up. The runner-local artifacts above say WHAT check
704-
# failed; this captures WHY — node conditions/taints, cluster events
705-
# (reboots, pod evictions), the operator CRs (incl. Skyhook status.status
706-
# the readiness gate keys off of), and operator/check-Job pod logs.
707-
# Best-effort (the collector guards every kubectl call) and bounded so it
708-
# never eats the teardown budget or leaks the GPU node.
709-
- name: Collect cluster debug bundle
710-
if: failure() && steps.prep.outcome != 'skipped'
711-
timeout-minutes: 8
712-
shell: bash
713-
env:
714-
AICR_BIN: ${{ github.workspace }}/aicr
715-
RUN_ID: ${{ github.run_id }}
716-
run: ./tests/uat/azure/run debug "${TEST_CONFIG}" || true
717-
718701
- name: Upload failure debug
719702
if: failure() && steps.prep.outcome != 'skipped'
720703
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
@@ -729,7 +712,6 @@ jobs:
729712
evidence/pointer.yaml
730713
train-logs/**
731714
serve-logs/**
732-
cluster-debug/**
733715
retention-days: 30
734716
if-no-files-found: ignore
735717

.github/workflows/uat-gcp.yaml

Lines changed: 3 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -76,10 +76,9 @@ jobs:
7676
# each = up to ~40m). Only one of train/serve runs per intent, so the budget
7777
# counts the larger (serve). The teardown MUST fit in this budget: a job-level
7878
# timeout cancels pending always() steps, so an undersized cap would skip
79-
# teardown and leak the GPU node. 190 UAT + ~30 setup + ~8 failure-path debug
80-
# collection (Collect cluster debug bundle) + ~40 teardown = ~268, so 280
81-
# leaves the teardown headroom even on a worst-case phase run.
82-
timeout-minutes: 280
79+
# teardown and leak the GPU node. 190 UAT + ~30 setup + ~40 teardown = ~260,
80+
# so 270 leaves the teardown headroom even on a worst-case phase run.
81+
timeout-minutes: 270
8382
permissions:
8483
contents: read
8584
actions: read
@@ -541,34 +540,6 @@ jobs:
541540
retention-days: 90
542541
if-no-files-found: error
543542

544-
# Refresh the GCP session before collecting (mirrors the AWS rationale): a
545-
# long tuning-race failure can outlast the WIF token minted at setup, which
546-
# would leave every kubectl in the collector failing auth and the bundle
547-
# silently empty. The gke-gcloud-auth-plugin reads ADC live, so re-auth alone
548-
# refreshes kubectl — no get-credentials re-run needed.
549-
- name: Re-authenticate to GCP for debug collection
550-
if: failure() && steps.prep.outcome != 'skipped'
551-
uses: google-github-actions/auth@7c6bc770dae815cd3e89ee6cdf493a5fab2cc093 # v3.0.0
552-
with:
553-
workload_identity_provider: ${{ env.GCP_WIF_PROVIDER }}
554-
service_account: ${{ env.GCP_WIF_SERVICE_ACCOUNT }}
555-
556-
# Snapshot live cluster state on failure BEFORE teardown, while the
557-
# cluster is still up. The runner-local artifacts above say WHAT check
558-
# failed; this captures WHY — node conditions/taints, cluster events
559-
# (reboots, pod evictions), the operator CRs (incl. Skyhook status.status
560-
# the readiness gate keys off of), and operator/check-Job pod logs.
561-
# Best-effort (the collector guards every kubectl call) and bounded so it
562-
# never eats the teardown budget or leaks the GPU node.
563-
- name: Collect cluster debug bundle
564-
if: failure() && steps.prep.outcome != 'skipped'
565-
timeout-minutes: 8
566-
shell: bash
567-
env:
568-
AICR_BIN: ${{ github.workspace }}/aicr
569-
RUN_ID: ${{ github.run_id }}
570-
run: ./tests/uat/gcp/run debug "${TEST_CONFIG}" || true
571-
572543
- name: Upload failure debug
573544
if: failure() && steps.prep.outcome != 'skipped'
574545
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
@@ -583,7 +554,6 @@ jobs:
583554
evidence/pointer.yaml
584555
train-logs/**
585556
serve-logs/**
586-
cluster-debug/**
587557
retention-days: 30
588558
if-no-files-found: ignore
589559

api/aicr/v1/server.yaml

Lines changed: 0 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -928,22 +928,6 @@ paths:
928928
schema:
929929
type: boolean
930930
default: false
931-
- name: serial
932-
in: query
933-
required: false
934-
description: >
935-
Sequence components strictly one at a time in deployment order,
936-
disabling the parallel rollout of independent components. Affects
937-
the `argocd`, `argocd-helm`, `flux`, and `helmfile` deployers
938-
(`helm` is already serial): argocd falls back to a linear
939-
sync-wave per folder, flux chains each HelmRelease `dependsOn` to
940-
the previous component, and helmfile chains every release via
941-
`needs:` into one linear apply order. An escape hatch for
942-
reproducing the pre-parallelism ordering or bisecting a
943-
misbehaving rollout.
944-
schema:
945-
type: boolean
946-
default: false
947931
- name: app-name
948932
in: query
949933
required: false

demos/cuj1-slinky-slurm.md

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@ Pick the row that matches your cluster. Each resolves to a slurm leaf with at le
3131

3232
| Cloud | Command | Leaf overlay |
3333
| -------- | ------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------- |
34+
| **AKS** | `aicr recipe --service aks --accelerator h100 --intent training --os ubuntu --platform slurm -o recipe.yaml` | `h100-aks-ubuntu-training-slurm` |
3435
| **EKS** | `aicr recipe --service eks --accelerator h100 --intent training --os ubuntu --platform slurm -o recipe.yaml` | `h100-eks-ubuntu-training-slurm` |
3536
| **GKE** | `aicr recipe --service gke --accelerator h100 --intent training --os cos --platform slurm -o recipe.yaml` | `h100-gke-cos-training-slurm` |
3637
| **Kind** | `aicr recipe --service kind --accelerator h100 --intent training --platform slurm -o recipe.yaml` | `h100-kind-training-slurm` (CPU-only NodeSet; no GPU GRES) |
@@ -112,6 +113,27 @@ GKE system nodes should **not** carry custom taints (konnectivity and other mana
112113

113114
Optional: `--accelerated-node-toleration nvidia.com/gpu=present:NoSchedule` (harmless if that taint is absent).
114115

116+
### AKS (system + cpu + gpu pools; CriticalAddonsOnly + GPU taint)
117+
118+
Example layout: 3× `system` (`CriticalAddonsOnly=true:NoSchedule`), 1× `cpuworker1` (untainted), 2× `gpuworker1` (`nvidia.com/gpu=present:NoSchedule`). Operator on **system**; controller / login / restapi on **cpuworker1**; slurmd on **gpuworker1**.
119+
120+
```shell
121+
aicr bundle \
122+
--recipe recipe.yaml \
123+
--deployer helm \
124+
--system-node-selector agentpool=system \
125+
--system-node-toleration CriticalAddonsOnly=true:NoSchedule \
126+
--accelerated-node-selector agentpool=gpuworker1 \
127+
--accelerated-node-toleration nvidia.com/gpu=present:NoSchedule \
128+
--set slinkyslurm:nodesets.slinky.replicas=2 \
129+
--set-json 'slinkyslurm:controller.podSpec={"nodeSelector":{"agentpool":"cpuworker1"}}' \
130+
--set-json 'slinkyslurm:restapi.podSpec={"nodeSelector":{"agentpool":"cpuworker1"}}' \
131+
--set-json 'slinkyslurm:loginsets.slinky.podSpec={"nodeSelector":{"agentpool":"cpuworker1"}}' \
132+
--output bundle
133+
```
134+
135+
AKS ships `managed-csi` as the default StorageClass; omit `--storage-class` unless you need a non-default class.
136+
115137
### Kind (CPU-only smoke / CI)
116138

117139
No GPU pools or taints; omit accelerated flags unless your Kind config adds them.

docs/contributor/component.md

Lines changed: 0 additions & 69 deletions
Original file line numberDiff line numberDiff line change
@@ -293,75 +293,6 @@ AICR ships five output adapters in
293293
`HelmRelease`s). **Components do not need to be deployer-aware** —
294294
the bundler renders per-deployer from one component definition.
295295

296-
### Deployment ordering
297-
298-
**One model, three projections.** Every deployer derives ordering from
299-
the same source: each component's declared `dependencyRefs`. What
300-
differs is how faithfully a deployer's native mechanism can express
301-
that dependency graph. For the concurrent deployers, components with no
302-
path between them are independent and roll out **concurrently**, while a
303-
real dependency **gates** (the dependent waits for its dependency to be
304-
healthy). The `NNN-<name>/` folder numbers reflect `DeploymentOrder`
305-
(a topological serialization) for readability, and drive the two
306-
deliberately linear paths — the helm `deploy.sh` and `--serial` mode —
307-
which install strictly in that order regardless of tiers.
308-
309-
`pkg/recipe` exposes the graph two ways, and deployers pick whichever
310-
fits their mechanism:
311-
312-
- `TopologicalSort` → a flat `DeploymentOrder` (one valid serialization).
313-
- `ComponentRefsTopologicalLevels` → dependency-depth **tiers**, where a
314-
tier holds exactly the components whose dependencies are all satisfied
315-
by earlier tiers (so a tier's members are mutually independent).
316-
317-
**flux — the exact DAG.** Flux's `dependsOn` is a native dependency
318-
graph, so the flux deployer projects each component's declared
319-
`dependencyRefs` directly onto it: a `HelmRelease`'s `dependsOn` names
320-
exactly its dependencies' terminal releases, nothing more. This is the
321-
most faithful and most parallel rendering — a component waits only for
322-
what it actually needs — and it reads naturally to flux users because
323-
`dependsOn` mirrors `dependencyRefs` one-for-one. See `declaredDependsOn`.
324-
325-
**argocd / argocd-helm — tiers as sync-wave bands.** Argo CD's
326-
`sync-wave` is a single integer per Application. Applications sharing a
327-
wave sync together, and Argo advances to a higher wave only once the
328-
current one is healthy — ordered *bands*, a partial order rather than a
329-
total one. Because a component gets one integer, it cannot express "wait
330-
for A but not B" when A and B sit at the same depth, so the DAG is
331-
**approximated** by tiers: `wave = tier*4 + phase`, where the
332-
per-component phase orders `-pre` → primary → `-post` → `-readiness`.
333-
The stride-4 band width keeps consecutive tiers disjoint, so a tier's
334-
readiness gate still blocks the next tier. The cost of the coarser
335-
banding is mild over-constraint (a component waits for its whole prior
336-
tier, not just its own dependencies). See `waveForFolder`.
337-
338-
**helmfile — tiers as sequential sub-helmfiles.** Emits one
339-
`level-N.yaml` sub-helmfile per tier, processed in sequence so each
340-
tier's CRDs land in the cluster's REST mapper before the next tier's
341-
plan is rendered (issue #914). Within a sub-helmfile, `needs:` edges
342-
chain only a component's own `-pre` → primary → `-post` releases;
343-
independent components in the tier carry no edge, so helmfile applies
344-
them concurrently. Cross-tier ordering is the sub-helmfile sequence, not
345-
per-release `needs:` — the same tier approximation as argocd. See
346-
`buildHelmfile`.
347-
348-
**helm — deliberately serial.** The `deploy.sh` installs one component
349-
at a time in `DeploymentOrder`, trading parallelism for the simplest
350-
possible shell. The readiness gates already sequence it correctly.
351-
352-
#### Disabling parallelism (`--serial`)
353-
354-
`aicr bundle --serial` forces every deployer to install components
355-
strictly one at a time in `DeploymentOrder`, an escape hatch for
356-
reproducing the pre-parallelism ordering or bisecting a misbehaving
357-
rollout. It affects the four concurrent deployers: argocd and
358-
argocd-helm fall back to a linear sync-wave per folder, flux chains each
359-
`HelmRelease` `dependsOn` to the previous component instead of
360-
projecting the DAG, and helmfile chains every release to its predecessor
361-
via `needs:` (one linear apply chain) instead of only within a
362-
component. The helm `deploy.sh` is already serial, so the flag is a
363-
no-op for it. Off by default.
364-
365296
See [index.md](index.md#community-standard-deployment-targets) for
366297
the deployer matrix.
367298

docs/integrator/components/nodewright.md

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -75,15 +75,15 @@ The table below is generated from the recipes by `make tuning-docs` — **do not
7575

7676
| Service | Accelerator | Profile | Setup | Tuning |
7777
|---------|--------------|---------|--------------------|-------------------------|
78-
| aks | a100 | h100 | nvidia-setup 0.5.0 | nvidia-tuned 0.3.2 |
79-
| aks | h100 | - | nvidia-setup 0.5.0 | nvidia-tuned 0.3.2 |
78+
| aks | a100 | h100 | nvidia-setup 0.5.0 | nvidia-tuned 0.3.1 |
79+
| aks | h100 | - | nvidia-setup 0.5.0 | nvidia-tuned 0.3.1 |
8080
| bcm | * | h100 | nvidia-setup 0.3.0 | - |
8181
| bcm | h100 | - | nvidia-setup 0.3.0 | - |
82-
| eks | a100 | h100 | nvidia-setup 0.5.0 | nvidia-tuned 0.3.2 |
83-
| eks | gb200 | - | nvidia-setup 0.5.0 | nvidia-tuned 0.3.2 |
84-
| eks | h100 | - | nvidia-setup 0.5.0 | nvidia-tuned 0.3.2 |
85-
| eks | h200 | h100 | nvidia-setup 0.5.0 | nvidia-tuned 0.3.2 |
86-
| eks | rtx-pro-6000 | generic | - | nvidia-tuned 0.3.2 |
82+
| eks | a100 | h100 | nvidia-setup 0.5.0 | nvidia-tuned 0.3.1 |
83+
| eks | gb200 | - | nvidia-setup 0.5.0 | nvidia-tuned 0.3.1 |
84+
| eks | h100 | - | nvidia-setup 0.5.0 | nvidia-tuned 0.3.1 |
85+
| eks | h200 | h100 | nvidia-setup 0.5.0 | nvidia-tuned 0.3.1 |
86+
| eks | rtx-pro-6000 | generic | - | nvidia-tuned 0.3.1 |
8787
| gke | a100 | h100 | - | nvidia-tuning-gke 0.1.2 |
8888
| gke | b200 | - | - | nvidia-tuning-gke 0.1.2 |
8989
| gke | h100 | - | - | nvidia-tuning-gke 0.1.2 |

0 commit comments

Comments
 (0)