feat(validator): data-driven NCCL benchmark runtime for --data recipes

Add the nccl-benchmark-runtime-ref performance constraint: a recipe ships a
Kubeflow TrainingRuntime as a file in its --data tree (at the same
validators/performance/testdata/{accelerator}/{service}/runtime.yaml layout
the embedded templates use) and references it. This closes the gap the
nccl-benchmark-profile escape hatch (#1719) left open — a private
service+accelerator introduced only via --data, whose fabric matches no
embedded template, can now be NCCL-performance-certified keyed on its own
criteria with no compiled applicability entry.

The orchestrator (resolveBenchmarkRuntimeRef, called from ValidatePhases/
ValidatePhase) reads the referenced file through the recipe DataProvider and
lowers its content into an nccl-benchmark-runtime carrier constraint that
rides to the pod via the existing ValidationInput ConfigMap. The in-pod
validator renders it in place of a baked-in template, bypasses the compiled
supportedNCCLCombinations gate, and skips all service-specific fabric
plumbing (EFA/TCPXO/RDMA discovery, GB200-NVreg/GKE-TCPXO preflights, RoCE
claim, NVLS/IMEX) — the supplied runtime owns its fabric.

Trust-bounded to a TrainingRuntime GVR with forced name/namespace; fails
closed on a malformed/traversal ref, missing/empty file, unresolved ref
reaching the pod, or a non-TrainingRuntime carrier. Mutually exclusive with
nccl-benchmark-profile.

Fixes #1792

Signed-off-by: Nathan Hensley <nhensley@nvidia.com>

Author: njhensley

docs/contributor/validator.md

@@ -433,6 +433,49 @@ already present in the compiled matrix, so the
 (advertised tuple ⇒ parseable template) covers every profile a recipe can
 name. See `validators/performance/nccl_benchmark_profile.go`.
 
+#### NCCL: recipe-supplied runtime (`nccl-benchmark-runtime-ref`)
+
+A profile still requires an embedded template to borrow, so it cannot cover a
+private service+accelerator whose fabric matches none of the shipped templates
+([#1792](https://github.com/NVIDIA/aicr/issues/1792)). The optional
+`nccl-benchmark-runtime-ref` performance constraint closes that gap: its value is
+a bare `{accelerator}/{service}` naming a Kubeflow `TrainingRuntime` the recipe
+ships in its `--data` tree at
+`validators/performance/testdata/{accelerator}/{service}/runtime.yaml` — the same
+layout the embedded templates use, so an external runtime is a drop-in for
+upstreaming.
+
+Resolution is split across the two-stage design:
+
+- **Orchestrator** (`pkg/validator/benchmark_runtime_ref.go`,
+  `resolveBenchmarkRuntimeRef`, called at the top of `ValidatePhases` /
+  `ValidatePhase`): reads the referenced file through the recipe `DataProvider`
+  and lowers its content into the `nccl-benchmark-runtime` **carrier** constraint
+  before the `/data/validation` ConfigMap is written. Fails closed
+  (`ErrCodeInvalidRequest`) on a malformed/traversal ref, a missing/empty file,
+  an absent `DataProvider`, or a ref set alongside an inline carrier. The two
+  constraint names are defined once in `pkg/validator/v1` (`PerfConstraintNCCLBenchmarkRuntime*`)
+  so the write side and the pod read side cannot drift.
+- **Pod** (`validators/performance/nccl_benchmark_runtime.go`): reads the
+  `nccl-benchmark-runtime` carrier and renders it — via the same `${VAR}`
+  substitution as the baked-in templates, through the shared `renderYAMLTemplate`
+  core — in place of `testdata/{accelerator}/{service}`.
+  `validateNcclAllReduceBw` bypasses the `ncclCombinationSupported` applicability
+  gate entirely (applicability is granted by the recipe's explicit opt-in, keyed
+  on its own criteria) and `applyNCCLResources` skips every service-specific step
+  — EFA/TCPXO/RDMA discovery, the GB200-NVreg / GKE-TCPXO preflights, RoCE claim
+  application, and NVLS/IMEX provisioning — because the supplied runtime owns its
+  fabric wiring. Transport verification still runs (the `-net` / `-nvls` markers
+  are transport-internal and fabric-agnostic), so a runtime paired with a named
+  variant must still prove its transport rather than pass on bandwidth alone; the
+  worker cohort is also sized against the runtime's own `nodeSelector` so
+  `WorkerCount` matches placement. It fails
+  closed on a carrier that is not a `trainer.kubeflow.org/v1alpha1`
+  `TrainingRuntime` with a `node` replicatedJob. The runtime is applied only
+  through `trainingRuntimeGVR` with a force-set name/namespace, so a recipe can
+  supply a `TrainingRuntime` and nothing else — not an arbitrary resource kind.
+  It is mutually exclusive with `nccl-benchmark-profile`.
+
 #### `inference-perf`: model, concurrency, and weights cache
 
 The `inference-perf` check warms vLLM before measuring, so the one-time

docs/integrator/data-extension.md

@@ -130,6 +130,15 @@ to opt into one of the embedded benchmarks. The profile selects the benchmark
 template and fabric handling; node identification still follows the recipe's
 own accelerator. See
 [Opting external recipes into a benchmark profile](../user/validation.md#opting-external-recipes-into-a-benchmark-profile).
+If the private service's fabric matches none of the embedded templates, ship the
+benchmark itself: put a Kubeflow `TrainingRuntime` in your `--data` tree at
+`validators/performance/testdata/{accelerator}/{service}/runtime.yaml` and point
+the `nccl-benchmark-runtime-ref` constraint at it (`{accelerator}/{service}`).
+Pass the same `--data <dir>` to `aicr validate` so it can resolve the referenced
+file; validation then gates it keyed on the recipe's own criteria with no
+compiled entry required, and the file is a drop-in should the runtime ever be
+upstreamed.
+See [Supplying a benchmark runtime for a private service](../user/validation.md#supplying-a-benchmark-runtime-for-a-private-service).
 
 ## Adding a component
 

docs/integrator/recipe-development.md

@@ -290,6 +290,22 @@ discovery, and preflights. See
 [Opting external recipes into a benchmark profile](../user/validation.md#opting-external-recipes-into-a-benchmark-profile)
 for the valid pairs and skip/fail semantics.
 
+When no embedded pair matches — a genuinely private service+accelerator with a
+fabric none of the shipped templates cover — supply the benchmark yourself:
+ship a Kubeflow `TrainingRuntime` in your `--data` tree at
+`validators/performance/testdata/{accelerator}/{service}/runtime.yaml` and
+reference it with the `nccl-benchmark-runtime-ref` constraint (a bare
+`{accelerator}/{service}` value). Run `aicr validate --data <dir> ...` so the
+referenced file is resolvable; it is read and rendered in place of a baked-in
+template, keyed on the recipe's own criteria with no compiled applicability
+entry. The runtime owns its fabric wiring (the validator skips service-specific
+fabric setup — discovery, preflights, and NVLS/IMEX provisioning — but still
+asserts transport for the `-net`/`-nvls` variants), must
+declare a `node` replicatedJob, and is mutually exclusive with
+`nccl-benchmark-profile`. Laying the file at the embedded testdata path makes it
+a drop-in for upstreaming. See
+[Supplying a benchmark runtime for a private service](../user/validation.md#supplying-a-benchmark-runtime-for-a-private-service).
+
 ### Component Types
 
 **Helm components** (most common):

docs/user/validation.md

@@ -52,9 +52,12 @@ ones) that match the target fabric:
 | `nccl-all-reduce-bw-nvls` | NVLS (MNNVL across an NVL72 IMEX domain) | GB200 + EKS, and GB200 + OKE. Asserts the NVLS communicator actually initialized — catches silent fallback to EFA (EKS) or Socket (OKE) when the IMEX domain is misconfigured. |
 
 The applicability column is the *default*, derived from the recipe's
-`criteria`. A recipe whose criteria fall outside it can opt into one of these
-benchmarks explicitly — see
-[Opting external recipes into a benchmark profile](#opting-external-recipes-into-a-benchmark-profile).
+`criteria`. A recipe whose criteria fall outside it can still run these
+benchmarks explicitly — either by
+[borrowing an embedded benchmark profile](#opting-external-recipes-into-a-benchmark-profile)
+whose fabric matches its hardware, or, for a private service whose fabric
+matches no embedded template, by
+[supplying its own benchmark runtime](#supplying-a-benchmark-runtime-for-a-private-service).
 
 The `-net` check defaults to the AWS EFA fabric. On a ConnectX **RoCE** cluster
 (e.g. DGXC GB300 `p6e-gb300r`), set `AICR_NCCL_FABRIC=roce` in the `aicr
@@ -180,6 +183,124 @@ check with a message naming the profile. When a profile is set it wins over
 criteria-derived applicability, and the pass/fail thresholds stay in the
 same-named check constraints as always.
 
+### Supplying a benchmark runtime for a private service
+
+A profile only helps when an **embedded** template already matches the
+cluster's hardware and fabric. A private service introduced entirely through
+`--data` — a `criteria.service` (and possibly `criteria.accelerator`) the
+validator ships no template for, with a fabric that reuses none of the
+embedded ones — has no compiled pair to point a profile at. Such a recipe
+supplies the benchmark itself: it ships a Kubeflow `TrainingRuntime` as a file
+in its `--data` tree and references it with the `nccl-benchmark-runtime-ref`
+performance constraint (a bare `{accelerator}/{service}` value). `aicr validate`
+reads that file and gates the benchmark keyed on the recipe's **own** criteria,
+with no compiled applicability entry required.
+
+Lay the runtime out **exactly where the embedded templates live**, so it is a
+drop-in for upstreaming later:
+
+```text
+mydata/                                   # your --data directory
+├── overlays/
+│   └── mycloud-training.yaml             # the recipe overlay (below)
+└── validators/performance/testdata/
+    └── gb200/mycloud/runtime.yaml        # the TrainingRuntime, real file
+```
+
+```yaml
+# overlays/mycloud-training.yaml
+validation:
+  performance:
+    checks:
+      - nccl-all-reduce-bw
+    constraints:
+      - name: nccl-all-reduce-bw
+        value: ">= 450"
+      - name: nccl-benchmark-runtime-ref
+        value: gb200/mycloud   # -> validators/performance/testdata/gb200/mycloud/runtime.yaml
+```
+
+```yaml
+# validators/performance/testdata/gb200/mycloud/runtime.yaml
+apiVersion: trainer.kubeflow.org/v1alpha1
+kind: TrainingRuntime
+metadata:
+  name: nccl-all-reduce-runtime
+spec:
+  template:
+    spec:
+      replicatedJobs:
+        - name: node          # required: the worker cohort
+          template:
+            spec:
+              template:
+                spec:
+                  containers:
+                    - name: node
+                      image: <your NCCL benchmark image>
+                      resources:
+                        limits:
+                          nvidia.com/gpu: "${GPU_COUNT_PER_NODE}"
+                      # ...command, fabric resources, sidecars...
+```
+
+```bash
+aicr validate -r recipe.yaml -s snapshot.yaml \
+              --data ./mydata --phase performance
+```
+
+The supplied runtime **owns its fabric wiring end to end**: because the recipe
+opted in explicitly, the validator bypasses the compiled applicability gate and
+skips every service-specific *setup* step — EFA/TCPXO/RDMA NIC discovery, the
+GB200-NVreg / GKE-TCPXO preflights, and NVLS/IMEX auto-provisioning. It renders
+the runtime, sizes it (against the runtime's own `nodeSelector` when it pins
+one, so `WorkerCount` matches placement), applies it alongside the shared
+`TrainJob`, and evaluates the bandwidth threshold from the launcher logs.
+Transport verification is **not** skipped: pairing the runtime with the `-net`
+or `-nvls` check still asserts that transport actually carried traffic (the NCCL
+markers are fabric-agnostic), so a named variant can't pass on bandwidth alone.
+What it must honor:
+
+- It must be a Kubeflow `TrainingRuntime` (`apiVersion:
+  trainer.kubeflow.org/v1alpha1`); any other kind/apiVersion is rejected. The
+  validator force-sets its name and namespace, so a recipe can supply a
+  `TrainingRuntime` and nothing else — never an arbitrary resource.
+- It must declare a `node` replicatedJob (the worker cohort the shared
+  `TrainJob` sizes and the validator injects scheduling into and reads logs
+  from), mirroring the embedded templates.
+- The generic template variables `${NAMESPACE}`, `${WORKER_COUNT}`,
+  `${GPU_COUNT_PER_NODE}`, `${GPU_COUNT}`, `${TEST_TYPE}`,
+  `${MIN_MESSAGE_SIZE}`, and `${MAX_MESSAGE_SIZE}` are substituted; the
+  service-specific ones (EFA/RDMA/GKE) are not, so the runtime must pin its own
+  fabric resources.
+- The runtime should pin its own worker `nodeSelector`/`tolerations`, or pass
+  `--node-selector` to `aicr validate`, so workers land on the intended GPU
+  nodes.
+
+`nccl-benchmark-runtime-ref` and `nccl-benchmark-profile` are mutually
+exclusive — a recipe supplies its own runtime **or** borrows an embedded one,
+never both. An absent or blank ref falls back to criteria/profile-derived
+applicability. A malformed ref, or a file missing from `--data`, is rejected by
+`aicr validate` **before any validator Job is deployed** (a resolution error on
+stderr). A resolved file that is not a `TrainingRuntime` with a `node`
+replicatedJob **fails the check itself** (in the pod). Either way it fails
+rather than silently skipping.
+
+> **Sizing note.** The worker cohort is sized against the runtime's own
+> `nodeSelector` when it pins one, but richer scheduling constraints
+> (`nodeAffinity`, un-tolerated node taints) are the Kubernetes scheduler's
+> job, not the validator's — a runtime that restricts placement below the sized
+> cohort surfaces as a launcher timeout, not a false pass. Pass an explicit
+> `--node-selector` when you want the sized and scheduled cohorts to match
+> exactly.
+
+**Upstreaming.** Because the file already sits at
+`validators/performance/testdata/{accelerator}/{service}/runtime.yaml`, adopting
+it into AICR is a copy: move the file into the repo, add the accelerator/service
+tuple to the compiled `supportedNCCLCombinations` matrix, and drop the
+`nccl-benchmark-runtime-ref` constraint — the benchmark then runs from the
+recipe's plain criteria like any other embedded one.
+
 ## Inference performance validation
 
 Inference performance runs the `inference-perf` check — deploys a
@@ -776,8 +897,11 @@ Common reasons and their cause:
 | `no inference-throughput or inference-ttft-p99 constraint in recipe` | `stdout` | Check was invoked but recipe is missing the matching constraints | Re-generate the recipe or add the constraints |
 | `dynamo-platform not in recipe components` | `stdout` | Inference check selected but `dynamo-platform` absent from `componentRefs` | Use `--platform dynamo` when generating the recipe |
 | `DynamoGraphDeployment CRD not installed` | `stdout` | Recipe declares `dynamo-platform` but the operator is not deployed | Run `aicr bundle` + `./deploy.sh` first, or wait for bootstrap to complete |
-| `requires Service + Accelerator to be implemented` | `stdout` | The recipe's `criteria` are outside the NCCL benchmark's default applicability (e.g. a service registered via `--data`) | Add an `nccl-benchmark-profile` constraint — see [Opting external recipes into a benchmark profile](#opting-external-recipes-into-a-benchmark-profile) |
+| `requires Service + Accelerator to be implemented` | `stdout` | The recipe's `criteria` are outside the NCCL benchmark's default applicability (e.g. a service registered via `--data`) | Add an `nccl-benchmark-profile` constraint — see [Opting external recipes into a benchmark profile](#opting-external-recipes-into-a-benchmark-profile) — or, if no embedded template fits, an `nccl-benchmark-runtime-ref` constraint — see [Supplying a benchmark runtime for a private service](#supplying-a-benchmark-runtime-for-a-private-service) |
 | `benchmark profile … does not implement the … NCCL variant` | `stdout` | The declared `nccl-benchmark-profile` is valid but has no template for this check variant | Drop the non-applicable check from `validation.performance.checks`, or pick a profile that implements it |
+| `failed to resolve nccl-benchmark-runtime-ref=… expected … in the --data tree` | `stderr` | The referenced runtime file is missing from `--data` (or `--data` wasn't passed) | Place the `TrainingRuntime` at `validators/performance/testdata/{accelerator}/{service}/runtime.yaml` in your `--data` dir — see [Supplying a benchmark runtime for a private service](#supplying-a-benchmark-runtime-for-a-private-service) |
+| `nccl-benchmark-runtime … must be a … TrainingRuntime … must declare a "node" replicatedJob` | `stdout` | The referenced file is not a Kubeflow `trainer.kubeflow.org/v1alpha1` `TrainingRuntime`, or lacks the `node` replicatedJob | Fix the runtime file to be a valid `TrainingRuntime` with a `node` replicatedJob |
+| `nccl-benchmark-runtime and nccl-benchmark-profile are mutually exclusive` | `stdout` | The recipe declares both escape hatches at once | Keep only one: borrow an embedded profile **or** supply your own runtime |
 | `skipped - no-cluster mode` | `message` | `--no-cluster` was passed — the runner short-circuits every phase before dispatching any Job | Remove the flag to run behavioral checks |
 | `skipped due to previous phase failure` | `message` | `--fail-fast` was set and an earlier phase failed, so subsequent phases were skipped | Fix the earlier phase first, or drop `--fail-fast` to run all phases regardless |