ci(uat): gate nightly intents by minimum release; enroll azure inference

Add nightly-intent-min-versions to the UAT reservation registry: a
per-intent minimum AICR release that gates RELEASE cells of the nightly
version matrix. The tip-of-main cell always runs every listed intent; a
release cell drops any intent whose minimum version is newer than the tag
(semver). This generalizes the informal 'a released version that predates
a platform is a premature cell' guidance into checked config, so a release
that predates an intent's support is skipped instead of running a
permanently-red cell.

The gate is resolved in the schedule — uat-broker attaches each cell's
eligible intents — so the nightly controller dispatches a cell's own
intents with no per-version semver logic in workflow bash. Validate
rejects a gate on an unrun/unknown intent or a non-semver value.

Enroll azure-h100 for [training, inference] with inference gated to
v0.18.0: the AKS ND H100 v5 perf fix (nvidia-tuned 0.3.2, #1767) is on
main but not yet in a release tag (v0.17.0 pins 0.3.1). Today only main
runs Azure inference nightly (green); older releases are skipped and the
gated release enrolls automatically once it ships.

Signed-off-by: Nathan Hensley <nhensley@nvidia.com>

Author: njhensley

.github/workflows/uat-nightly-batch.yaml

@@ -174,8 +174,10 @@ jobs:
             echo "::notice::Reservation ${RESERVATION} opts out of the nightly batch (nightly-intents: []); skipping this leg."
             exit 0
           fi
-          IFS=',' read -r -a intents <<< "$NIGHTLY_INTENTS_CSV"
-          echo "Nightly intents for ${RESERVATION}: ${intents[*]}"
+          # The actual per-cell intents come from the schedule below, which
+          # applies the reservation's nightly-intent-min-versions gate per
+          # version; this line is just the leg-level summary of what's enrolled.
+          echo "Nightly intents for ${RESERVATION}: ${NIGHTLY_INTENTS_CSV} (release cells further gated by nightly-intent-min-versions)"
 
           # Validate previous_n before the broker consumes it. `--previous-n` is
           # an int flag so it rejects non-integers, but it would accept a
@@ -213,8 +215,19 @@ jobs:
           # release cells first — never main's inference. Each (version × intent)
           # is a full provision→CUJ→teardown dispatched through the shared lease,
           # so the intents serialize automatically (no contention).
+          #
+          # Intents are per-CELL: the schedule already applied each
+          # reservation's nightly-intent-min-versions gate, so a release that
+          # predates an intent's minimum version carries a shorter (or empty)
+          # .intents list and that (version × intent) never dispatches; `main`
+          # always carries every listed intent.
           for ver in "${versions[@]}"; do
-            for intent in "${intents[@]}"; do
+            mapfile -t cell_intents < <(jq -r --arg r "$RESERVATION" --arg v "$ver" \
+              '.[$r][] | select(.aicr_version == $v) | .intents[]' <<<"$schedule")
+            # A fully-gated release cell (no eligible intents) contributes
+            # nothing — skip it without consuming the time-box or cell counter.
+            (( ${#cell_intents[@]} == 0 )) && continue
+            for intent in "${cell_intents[@]}"; do
               if (( $(date +%s) >= deadline )); then
                 echo "::notice::Time-box reached — dropping remaining cells for ${RESERVATION} (oldest release first)."
                 break 2

docs/contributor/uat.md

@@ -56,20 +56,31 @@ In both cases the signed evidence bundle is emitted by the earlier conformance s
 
 An unrecognized `intent` (or a missing sibling config) fails closed in the pipeline's `Validate inputs` step before any provisioning.
 
-### Nightly intent cadence (both intents, both clouds)
+### Nightly intent cadence (both intents, all three clouds)
 
-The single nightly cron (`uat-nightly-batch.yaml`, `0 4 * * *`) runs **both intents on every nightly-enrolled reservation**, so training *and* inference are exercised nightly on AWS and GCP, and training on Azure (`azure-h100` enrolls `[training]` only until a manual inference run goes green — see the table below). (Note: an inference cell currently provisions and validates the inference platform; the `phase_serve` serving-CUJ step itself remains disabled in both cloud workflows pending #1644, so nightly runs do not yet execute the serving request path.) The set of intents per reservation is data — the `nightly-intents` list in `infra/uat/reservations.yaml` (absent defaults to `[training]`; an explicit empty list `[]` opts the reservation out of the nightly batch entirely — bring-up mode, manual dispatch only):
+The single nightly cron (`uat-nightly-batch.yaml`, `0 4 * * *`) runs **both intents on every nightly-enrolled reservation**, so training *and* inference are exercised nightly on AWS, GCP, and Azure (see the table below). (Note: an inference cell currently provisions and validates the inference platform; the `phase_serve` serving-CUJ step itself remains disabled in both cloud workflows pending #1644, so nightly runs do not yet execute the serving request path.) The set of intents per reservation is data — the `nightly-intents` list in `infra/uat/reservations.yaml` (absent defaults to `[training]`; an explicit empty list `[]` opts the reservation out of the nightly batch entirely — bring-up mode, manual dispatch only):
 
 | Reservation | Cloud | `nightly-intents` | Nightly CUJs |
 |-------------|-------|-------------------|--------------|
 | `aws-h100` | AWS | `[training, inference]` | `phase_train` + `phase_serve` (serve step disabled pending #1644) |
 | `gcp-h100` | GCP | `[training, inference]` | `phase_train` + `phase_serve` (serve step disabled pending #1644) |
-| `azure-h100` | Azure | `[training]` | `phase_train` (inference joins after a green manual `intent=inference` run) |
+| `azure-h100` | Azure | `[training, inference]` | `phase_train` + `phase_serve` (serve step disabled pending #1644); inference gated to `>= v0.18.0` via `nightly-intent-min-versions` (see **Cost / tuning** below) |
 
 **How it stays contention-free — serialize, don't add a second cron.** The intents are folded into the existing [version matrix](#the-version-matrix) as extra cells rather than a second scheduled job. The controller's drive loop is **version outer / intent inner**: for each version it dispatches one intent's full provision→CUJ→teardown cell (inference cells currently run provision→validate→teardown; the serve CUJ is disabled pending #1644), waits for it (`gh run watch`), then dispatches the next — all through the *same* per-reservation lease. So the intents serialize naturally, and because `main` runs every intent before any release cell, a time-box drop only ever sheds the oldest *release* cells (never `main`'s inference). This is the deliberate DC3 cadence decision: **never schedule two daily crons against one reservation** — the lease is a single-slot queue (one in-progress + one pending), so a second cron plus an occasional human dispatch on the same reservation is a routine three-contender case whose loser is silently [superseded](#how-queuing-works-the-reservation-lease). One cron dispatching serialized cells sidesteps that entirely.
 
 **Cost / tuning.** Listing both intents roughly **doubles a reservation's nightly cell count** (each version now runs two full cluster lifecycles). If the batch [time-box](#the-version-matrix) is exceeded the oldest cells are dropped first, so `main`+freshest always land; tune `previous_n` (fewer release versions) or `deadline_offset_hours` to fit the window. A released version that predates a platform (e.g. `dynamo`) fails its inference cell's recipe resolution as a genuine regression signal — drop `previous_n` if that coverage is premature. Changing which intents a reservation runs is a registry edit — no workflow change; the `uatbroker` committed-registry test pins the launch set.
 
+**Gating an intent to a minimum release — `nightly-intent-min-versions`.** When an intent only became *supported* on a reservation at a particular release — a fix or platform that older releases lack — running it on the pre-support releases produces a permanently-red cell, not a regression signal. Express the floor per intent in the registry row:
+
+```yaml
+- name: azure-h100
+  nightly-intents: [training, inference]
+  nightly-intent-min-versions:
+    inference: v0.18.0   # first release that carries the AKS perf fix (#1767)
+```
+
+Semantics: **`main` is never gated** (it is built from source and carries the newest fixes, so it always runs every listed intent); a **release** cell drops any intent whose minimum version is newer than the tag (semver; a tag `>=` the minimum runs). The gate lives in the schedule (`uat-broker schedule` attaches each cell's eligible `intents`), so the controller simply never dispatches a gated `(version × intent)` — no per-version workflow logic. Pointing the floor at a **not-yet-tagged** release is intentional and self-resolving: until that release ships, the intent runs on **`main` only** (green, continuous coverage of the fix), and the release enrolls automatically once it exists. `Validate` rejects a floor for an intent the row does not run, or a non-semver value. Bump the floor if the real first-fixed tag differs — an over-low floor surfaces as a visible red (safe), an over-high floor silently skips a good release (bump down).
+
 ## Cluster lifecycles
 
 The `lifecycle` input selects one of three cluster lifecycles, all sharing the reservation lease:

infra/uat/reservations.yaml

@@ -105,10 +105,18 @@ reservations:
     test-config-dir: tests/uat/azure/tests
     # Enrolled with [training] after the green manual acceptance run
     # (29125390442: full provision → CUJ → evidence → teardown). Inference
-    # joins after a green manual intent=inference dispatch — the AWS/GCP
-    # rows run both, so add it here rather than removing the key (a bare
-    # `nightly-intents:` decodes as absent and defaults to [training]).
-    nightly-intents: [training]
+    # joined after a green manual intent=inference dispatch, matching the
+    # AWS/GCP rows which run both.
+    nightly-intents: [training, inference]
+    # The AKS ND H100 v5 inference perf fix (nvidia-tuned 0.3.2, #1767) is on
+    # main but not in any release tag yet — v0.17.0 and older pin 0.3.1 and
+    # fail inference-perf. Gate release inference cells to the first release
+    # that will carry the fix, so today only `main` runs Azure inference
+    # nightly (green); older releases are skipped instead of running red, and
+    # the gated release enrolls automatically once it ships. Bump this when the
+    # actual first-fixed tag differs.
+    nightly-intent-min-versions:
+      inference: v0.18.0
   - name: aws-gb200
     cloud: aws
     # CROSS-ACCOUNT: the GB200 Capacity Block lives in a SEPARATE tenancy

pkg/uatbroker/doc.go

@@ -30,6 +30,10 @@
 // drops the OLDEST releases first when its time-box closes — it simply stops
 // at the cursor. Release cells carry their tag in AICRVersion for DC5's
 // version-parameterized install; until DC5 lands they install from source.
+// Each cell also carries the nightly intents ELIGIBLE at its version: the
+// main cell runs every listed intent, while a release cell drops any intent
+// whose nightly-intent-min-versions floor is newer than the tag, so a release
+// that predates an intent's support never contributes a permanently-red cell.
 //
 // The package performs no network or git I/O and holds no credentials: the
 // CLI feeds it the registry bytes and the raw `git tag` list.

pkg/uatbroker/model.go

@@ -72,6 +72,25 @@ type Reservation struct {
 	// GPU capacity. KnownFields cannot catch this (the key is valid);
 	// TestParseRegistryBareNullNightlyIntents locks the behavior.
 	NightlyIntents []string `yaml:"nightly-intents"`
+	// NightlyIntentMinVersions gates RELEASE cells of the nightly version matrix
+	// by a per-intent minimum AICR release: intent -> minimum version (a semver
+	// tag like "v0.18.0"). It expresses "the first released version known to
+	// support this intent on this reservation", so a released version that
+	// predates a fix or platform is not run for that intent and does not
+	// contribute a predictably-red cell.
+	//
+	// Only RELEASE cells are gated. The tip-of-main cell (Cell.IsMain) always
+	// runs every listed intent — it is built from source and carries the newest
+	// fixes — so a min-version pointing at a not-yet-tagged release (the fix is
+	// on main but unreleased) correctly runs the intent on main-only until that
+	// release ships, then enrolls it automatically. A release tag >= the min
+	// runs; a tag below it is dropped for that intent only.
+	//
+	// OPTIONAL: absent or empty means no gate (every listed intent runs on every
+	// cell — the pre-#1789 behavior). Validate requires each key to be an intent
+	// this reservation actually lists in NightlyIntents (a min-version for an
+	// unrun intent is dead config / a typo) and each value to parse as semver.
+	NightlyIntentMinVersions map[string]string `yaml:"nightly-intent-min-versions"`
 	// DaytimeIntent opts this reservation into the daytime human-access
 	// rotation (#1281, DC8) and picks the flavor stood up on it during the
 	// working day: "training" or "inference". Empty means the reservation is
@@ -121,4 +140,10 @@ type Cell struct {
 	Reservation string `json:"reservation"`
 	AICRVersion string `json:"aicr_version"`
 	IsMain      bool   `json:"is_main"`
+	// Intents are the nightly intents eligible at this cell's version, in
+	// registry order (see EligibleNightlyIntents). The main cell carries every
+	// listed intent; a release cell drops any intent gated off by
+	// nightly-intent-min-versions. The controller dispatches one run per entry,
+	// so an empty list means the cell dispatches nothing.
+	Intents []string `json:"intents"`
 }

pkg/uatbroker/registry.go

@@ -22,6 +22,7 @@ import (
 	"os"
 	"strings"
 
+	"github.com/Masterminds/semver/v3"
 	"github.com/NVIDIA/aicr/pkg/errors"
 	"gopkg.in/yaml.v3"
 )
@@ -133,6 +134,33 @@ func (r *Registry) Validate() error {
 			}
 			seenIntent[intent] = true
 		}
+		// nightly-intent-min-versions gates RELEASE cells per intent. Each key
+		// must be an intent this reservation actually runs (a gate on an unrun
+		// intent — including any gate on an explicitly opted-out row — is dead
+		// config / a typo that would silently never apply) and each value must
+		// parse as semver (else the gate is inert and every release cell runs
+		// the intent, defeating the gate). Fail closed on both.
+		runsIntent := make(map[string]bool, len(res.NightlyIntents))
+		for _, intent := range res.NightlyIntentsOrDefault() {
+			runsIntent[intent] = true
+		}
+		for intent, minVer := range res.NightlyIntentMinVersions {
+			if !validIntents[intent] {
+				return errors.New(errors.ErrCodeInvalidRequest,
+					fmt.Sprintf("reservation %s has a nightly-intent-min-version for unknown intent %q (want %s or %s)",
+						res.Name, intent, IntentTraining, IntentInference))
+			}
+			if !runsIntent[intent] {
+				return errors.New(errors.ErrCodeInvalidRequest,
+					fmt.Sprintf("reservation %s gates intent %q via nightly-intent-min-versions but does not run it nightly (add %q to nightly-intents or drop the gate)",
+						res.Name, intent, intent))
+			}
+			if _, err := semver.NewVersion(minVer); err != nil {
+				return errors.New(errors.ErrCodeInvalidRequest,
+					fmt.Sprintf("reservation %s has an invalid nightly-intent-min-version %q for intent %q (want a semver tag like v0.18.0)",
+						res.Name, minVer, intent))
+			}
+		}
 		// daytime-intent is optional (empty = not in the daytime rotation), but
 		// when set it must be a recognized intent — a typo would otherwise
 		// silently drop the reservation from the daytime rotation or dispatch a

pkg/uatbroker/registry_test.go

@@ -314,6 +314,75 @@ reservations:
 			wantErr: true,
 			code:    errors.ErrCodeInvalidRequest,
 		},
+		{
+			// A min-version gate for an intent the reservation does not run is
+			// dead config / a typo — reject it.
+			name: "min-version for an unrun intent",
+			yaml: `
+reservations:
+  - name: azure-h100
+    cloud: azure
+    accelerator: h100
+    gpu-count: 8
+    cluster-config-path: c.yaml
+    test-config-dir: t
+    nightly-intents: [training]
+    nightly-intent-min-versions:
+      inference: v0.18.0
+`,
+			wantErr: true,
+			code:    errors.ErrCodeInvalidRequest,
+		},
+		{
+			name: "min-version for an unknown intent",
+			yaml: `
+reservations:
+  - name: azure-h100
+    cloud: azure
+    accelerator: h100
+    gpu-count: 8
+    cluster-config-path: c.yaml
+    test-config-dir: t
+    nightly-intents: [training, inference]
+    nightly-intent-min-versions:
+      serving: v0.18.0
+`,
+			wantErr: true,
+			code:    errors.ErrCodeInvalidRequest,
+		},
+		{
+			name: "min-version that is not semver",
+			yaml: `
+reservations:
+  - name: azure-h100
+    cloud: azure
+    accelerator: h100
+    gpu-count: 8
+    cluster-config-path: c.yaml
+    test-config-dir: t
+    nightly-intents: [training, inference]
+    nightly-intent-min-versions:
+      inference: not-a-tag
+`,
+			wantErr: true,
+			code:    errors.ErrCodeInvalidRequest,
+		},
+		{
+			// A valid gate on a listed intent parses cleanly.
+			name: "valid nightly-intent-min-versions",
+			yaml: `
+reservations:
+  - name: azure-h100
+    cloud: azure
+    accelerator: h100
+    gpu-count: 8
+    cluster-config-path: c.yaml
+    test-config-dir: t
+    nightly-intents: [training, inference]
+    nightly-intent-min-versions:
+      inference: v0.18.0
+`,
+		},
 		{
 			// Two daytime reservations on one cloud would contend for the same
 			// reservation (a cloud cannot hold both a held daytime cluster and the
@@ -567,9 +636,9 @@ func TestCommittedRegistryValid(t *testing.T) {
 		"aws-h100": {IntentTraining, IntentInference},
 		"gcp-h100": {IntentTraining, IntentInference},
 		// azure-h100 enrolled with [training] after the green manual
-		// acceptance run (29125390442); inference joins after a green
+		// acceptance run (29125390442); inference joined after a green
 		// manual intent=inference dispatch.
-		"azure-h100": {IntentTraining},
+		"azure-h100": {IntentTraining, IntentInference},
 		// aws-gb200 is OPTED OUT of the nightly batch during bring-up (explicit
 		// empty list). Locked here so an accidental edit to a bare
 		// `nightly-intents:` (which defaults to [training]) — provisioning real