diff --git a/demos/cuj2-demo.md b/demos/cuj2-demo.md index 70871cf4a..b68938cbc 100644 --- a/demos/cuj2-demo.md +++ b/demos/cuj2-demo.md @@ -210,7 +210,7 @@ publication revalidate a private snapshot and publish only that inventory. │ │ │ Discovery: Kubernetes-native (no etcd) │ │ Requests: Dynamo request plane (default TCP) │ -│ Events: NATS event plane for worker KV-cache events │ +│ Events: ZMQ-based KV-cache event plane (direct, no NATS) │ │ │ │ CRDs (6): │ │ ├── DynamoGraphDeployment (inference serving graph) │ @@ -249,8 +249,8 @@ publication revalidate a private snapshot and publish only that inventory. │ 1. Client → /v1/chat/completions → Frontend :8000 │ │ 2. Frontend → Dynamo request plane (TCP) → VllmDecodeWorker │ │ 3. VllmDecodeWorker runs Qwen3-0.6B on H100 │ -│ 4. Worker relays local vLLM ZMQ KV events to NATS │ -│ 5. KV router consumes NATS events; response returns over TCP │ +│ 4. Worker publishes ZMQ KV events directly to KV router │ +│ 5. KV router consumes ZMQ events; response returns over TCP │ └─────────────────────────────────────────────────────────────────┘ ``` ### ChatBot diff --git a/demos/workloads/inference/vllm-agg.yaml b/demos/workloads/inference/vllm-agg.yaml index 93c9da04d..b10c5f54f 100644 --- a/demos/workloads/inference/vllm-agg.yaml +++ b/demos/workloads/inference/vllm-agg.yaml @@ -75,7 +75,7 @@ spec: effect: NoExecute containers: - name: main - image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.2.1 + image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.4.1 env: - name: SERVED_MODEL_NAME value: Qwen/Qwen3-0.6B @@ -103,7 +103,7 @@ spec: effect: NoSchedule containers: - name: main - image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.2.1 + image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.4.1 workingDir: /workspace/examples/backends/vllm command: ["python3", "-m", "dynamo.vllm"] args: diff --git a/docs/contributor/validator.md b/docs/contributor/validator.md index 78d1a2973..ae418d371 100644 --- a/docs/contributor/validator.md +++ b/docs/contributor/validator.md @@ -870,14 +870,14 @@ run-to-run TTFT fluctuation (see NVIDIA/aicr#1192): worker's active in-flight load so a transiently-slow worker stops receiving its full share — mitigating the stochastic EKS H100 worker-stall / throughput degradation at the saturation knee (issue #1197). Frontend-to-worker - requests use Dynamo's request plane (Dynamo 1.2 defaults to TCP; AICR does - not set `DYN_REQUEST_PLANE=nats`). Workers still publish local vLLM KV-cache - events through their ZMQ publisher (relayed onto the NATS event plane), but - least-loaded routing does not consume them. + requests use Dynamo's request plane (Dynamo 1.4+ defaults to TCP; AICR does + not set `DYN_REQUEST_PLANE=nats`). Workers publish KV-cache events directly + over ZMQ; the KV router consumes them end-to-end with no NATS relay. + Least-loaded routing does not consume those events. The `inference-routing-mode` recipe input defaults to `dynamo-router`; set `gateway-epp` to validate the GAIE/EPP path through agentgateway with worker frontend sidecars in direct mode. The direct-mode sidecars honor EPP routing - headers; they do not perform the ZMQ-to-NATS KV-event relay. + headers; they do not relay KV events. - **The AIPerf load generator co-locates with the GPU workers, but that is not resource contention.** It is CPU-only and the GPU node has ample CPU headroom (measured node CPU pressure ≈ 0 across runs); co-location does not starve the diff --git a/docs/integrator/eks-dynamo-networking.md b/docs/integrator/eks-dynamo-networking.md index 6c8fb3531..d10fdca7e 100644 --- a/docs/integrator/eks-dynamo-networking.md +++ b/docs/integrator/eks-dynamo-networking.md @@ -1,30 +1,44 @@ # EKS Dynamo Networking Prerequisites For `*-eks-ubuntu-inference-dynamo` recipes, AICR configures -`dynamo-platform` with Kubernetes-native discovery and the standard NATS -event plane for KV-cache and runtime events: -- `nats` on TCP `4222` - -This NATS dependency is new as of the Dynamo 1.2 bump, which switched discovery -to the NATS event plane. A cluster whose system-node security group only -allowlisted the pre-1.2 control-plane ports will not have `4222` open, so a -bundle that worked on Dynamo 1.0.x can start failing purely from the version -bump — add the `4222` rule below. - -Frontend-to-worker inference request/response traffic is separate: Dynamo 1.2 -defaults `DYN_REQUEST_PLANE` to TCP, and AICR does not override it to NATS. The -worker runtime relays local vLLM ZMQ KV-cache events onto the NATS-backed event -plane so the KV router or EPP can consume live cache state. - -If system components and GPU workloads are on different node groups/security groups, these ports may be blocked from GPU nodes to system nodes. Typical symptoms: -- `JetStream not available` (NATS unreachable) -- Dynamo frontend and vLLM worker pods stuck in `CrashLoopBackOff`, with - `Exception: Failed to connect to NATS: timed out` in the frontend log -- Worker startup probes failing with `connection refused` because the process - exits before serving -- The `inference-perf` performance validator failing after its workload-readiness - (10 min) and health (5 min) gates lapse — roughly 15 min — while `deployment` - and `conformance` pass; the workload never reaches a ready state +`dynamo-platform` with Kubernetes-native discovery. As of the Dynamo 1.4+ +bump, AICR no longer installs bundled NATS by default: the request plane +defaults to TCP and the KV event plane defaults to ZMQ +(`ai-dynamo/dynamo#11951`). This removes the old `4222` NATS requirement, +but it does **not** remove the underlying cross-nodegroup networking +requirement — the request plane and KV events are now **direct +frontend↔worker pod-to-pod connections** instead of both sides talking to a +`dynamo-platform-nats` StatefulSet on the system nodegroup, and Frontend +pods still run on the system nodegroup while workers run on the GPU +nodegroup, so traffic still crosses the same GPU↔system nodegroup SG +boundary as before. + +> **TODO before merging (tracked in NVIDIA/aicr#1836):** the port(s) below +> are not yet confirmed against a real Dynamo 1.4+ EKS deployment. What's +> known from the AICR recipes: the ZMQ KV-event endpoint is set explicitly +> per worker via `--kv-events-config`, e.g. +> `{"enable_kv_cache_events":true,"publisher":"zmq","endpoint":"tcp://*:5557"}` +> (see `tests/manifests/dynamo-vllm-smoke-test.yaml`), offset by `+dp_rank` +> for dp_rank > 0. The TCP request plane does not have one fixed, +> documented port the way NATS had `4222` — confirm the actual listening +> port(s) on a live cluster before finalizing the SG rule below: +> ```shell +> kubectl exec -n dynamo-system -- ss -tlnp +> kubectl exec -n dynamo-system -- ss -tlnp +> ``` + +If the GPU and system node groups sit in different security groups, these +ports may be blocked from GPU nodes to the frontend's node (and vice versa). +Typical symptoms: +- Dynamo frontend and vLLM worker pods stuck in `CrashLoopBackOff`, or a + frontend that starts cleanly but never successfully routes a request + through to a worker +- Worker startup probes failing with `connection refused` because the + process exits before serving +- The `inference-perf` performance validator failing after its + workload-readiness (10 min) and health (5 min) gates lapse — roughly + 15 min — while `deployment` and `conformance` pass; the workload never + reaches a ready state You can confirm reachability directly from a GPU node before re-running. The toleration is required because the GPU node groups on these clusters are @@ -32,9 +46,9 @@ tainted (`NoSchedule`/`NoExecute`); without it the probe pod stays `Pending` and never runs: ```shell -kubectl run nats-probe --rm -i --restart=Never --image=busybox:1.36 \ +kubectl run tcp-probe --rm -i --restart=Never --image=busybox:1.36 \ --overrides='{"spec":{"nodeSelector":{"":""},"tolerations":[{"operator":"Exists"}]}}' \ - -- sh -c 'nc -zv -w 5 dynamo-platform-nats.dynamo-system.svc.cluster.local 4222' + -- sh -c 'nc -zv -w 5 ' ``` The conformance validator's `ai-service-metrics` check adds a third requirement: @@ -64,8 +78,9 @@ SG rule below remains the reliable cluster-side guarantee. ## Required Security Group Rules -Allow ingress from the GPU node security group to the system node security group on: -- TCP `4222` - NATS event plane (dynamo-platform) +Allow ingress from the GPU node security group to the system node security +group on: +- TCP `` - Dynamo request plane + KV events (dynamo-platform) — confirm exact port(s) on-cluster, see TODO above - TCP `9090` - Prometheus (required for the `ai-service-metrics` conformance check) The `9090` rule is required as a fallback guarantee: the orchestrator *prefers* @@ -94,9 +109,9 @@ aws ec2 describe-instances \ --query "Reservations[0].Instances[0].SecurityGroups[*].GroupId" \ --output text -# 2) Allow NATS + Prometheus from GPU SG -> system SG +# 2) Allow Dynamo request/event-plane + Prometheus from GPU SG -> system SG aws ec2 authorize-security-group-ingress --group-id \ - --protocol tcp --port 4222 --source-group + --protocol tcp --port --source-group aws ec2 authorize-security-group-ingress --group-id \ --protocol tcp --port 9090 --source-group diff --git a/docs/user/component-catalog.md b/docs/user/component-catalog.md index e62bce2a2..005ad4f1c 100644 --- a/docs/user/component-catalog.md +++ b/docs/user/component-catalog.md @@ -32,7 +32,7 @@ The source of truth is [`recipes/registry.yaml`](https://github.com/NVIDIA/aicr/ | **k8s-aibom** | Optional runtime AI workload inventory. Produces namespace-scoped CycloneDX 1.6 ML-BOM resources for explicitly opted-in namespaces. Installed by one stock recipe, `h100-gke-cos-inference`; every other stock recipe leaves it out. Decline it with `aicr recipe --runtime-inventory disabled`. CLI aliases: `k8saibom`, `aibom`. See [k8s-aibom Runtime Inventory](#k8s-aibom-runtime-inventory). | [k8s-aibom](https://github.com/GoogleCloudPlatform/k8s-aibom) | | **kai-scheduler** | Gang scheduler with hierarchical queues and topology-aware placement; works with device-plugin (`nvidia.com/gpu`) and DRA GPU allocation alike. Ensures distributed training jobs land on nodes with optimal interconnect topology. AICR pins `defaultQueue.createDefaultQueue: true`, so the chart creates the `default-parent-queue`/`default-queue` hierarchy on install. The `gang-scheduling` conformance check submits its synthetic test PodGroup to `default-queue` by name, so that queue is a hard dependency of validation, not an optional extra. Note the chart creates the queues only on first install and annotates them `helm.sh/resource-policy: keep` — a `helm upgrade` will not recreate them if they are deleted, so restore them manually (or reinstall the release) if that happens. Workloads are not restricted to this queue: Dynamo submits to its own `dynamo`/`dynamo-default` hierarchy, which its chart creates via post-install and post-upgrade hooks. | [KAI Scheduler](https://github.com/kai-scheduler/KAI-Scheduler) | | **grove** | Pod lifecycle management for Dynamo inference platform. Installed as a standalone component. | [Grove](https://github.com/ai-dynamo/grove) | -| **dynamo-platform** | NVIDIA Dynamo inference serving platform with bundled CRDs. Distributed inference with KV-cache-aware routing, Dynamo request-plane traffic, a NATS-backed Kubernetes event plane for KV-cache events, and disaggregated prefill/decode. | [Dynamo](https://github.com/ai-dynamo/dynamo) | +| **dynamo-platform** | NVIDIA Dynamo inference serving platform with bundled CRDs. Distributed inference with KV-cache-aware routing, Dynamo request-plane traffic, a ZMQ-based KV-cache event plane, and disaggregated prefill/decode. | [Dynamo](https://github.com/ai-dynamo/dynamo) | | **agentgateway-crds** | Custom Resource Definitions for agentgateway (Kubernetes Gateway API implementation for AI/ML inference). | [agentgateway](https://github.com/agentgateway/agentgateway) | | **agentgateway** | Kubernetes Gateway API implementation for AI/ML inference. Implements the Gateway API Inference Extension for model-aware ingress routing to InferencePool backends. | [agentgateway](https://github.com/agentgateway/agentgateway) | | **k8s-nim-operator** | NVIDIA NIM Operator for managing NIM (NVIDIA Inference Microservices) deployments on Kubernetes. AICR installs the operator only — it creates no `NIMService` and no credentials; see [NIM workload credentials](#nim-workload-credentials). | [K8s NIM Operator](https://github.com/NVIDIA/k8s-nim-operator) | diff --git a/docs/user/container-images.md b/docs/user/container-images.md index 112201971..0d27377a6 100644 --- a/docs/user/container-images.md +++ b/docs/user/container-images.md @@ -19,11 +19,11 @@ A machine-readable **CycloneDX 1.6 JSON** companion to this page is produced by ## Summary -- Components: **44** -- Unique images: **100** -- Distinct registries: **11** +- Components: **37** +- Unique images: **88** +- Distinct registries: **10** -Registries: `602401143452.dkr.ecr.us-west-2.amazonaws.com`, `cr.agentgateway.dev`, `docker.io`, `gcr.io`, `ghcr.io`, `gke.gcr.io`, `nvcr.io`, `public.ecr.aws`, `quay.io`, `registry.k8s.io`, `us-docker.pkg.dev` +Registries: `602401143452.dkr.ecr.us-west-2.amazonaws.com`, `cr.agentgateway.dev`, `docker.io`, `gcr.io`, `ghcr.io`, `gke.gcr.io`, `nvcr.io`, `quay.io`, `registry.k8s.io`, `us-docker.pkg.dev` _Rendering fidelity:_ `catalog-parity: charts are rendered with the shared recipes/components//values.yaml; per-recipe overlay overrides are not applied` @@ -33,12 +33,10 @@ _Rendering fidelity:_ `catalog-parity: charts are rendered with the shared recip |-----------|------|-------|----------------|--------| | agentgateway | helm | agentgateway | v1.3.1 | 1 | | agentgateway-crds | helm | agentgateway-crds | v1.3.1 | 0 | -| aws-ebs-csi-driver | helm | aws-ebs-csi-driver/aws-ebs-csi-driver | 2.59.0 | 6 | +| aws-ebs-csi-driver | helm | aws-ebs-csi-driver/aws-ebs-csi-driver | 2.59.0 | 0 | | aws-efa | helm | aws-efa-k8s-device-plugin | v0.5.29 | 1 | | cert-manager | helm | jetstack/cert-manager | v1.20.2 | 4 | -| cert-manager-ocp | manifest | — | — | 0 | -| cert-manager-ocp-olm | manifest | — | — | 0 | -| dynamo-platform | helm | dynamo-platform | 1.2.1 | 3 | +| dynamo-platform | helm | dynamo-platform | 1.4.1 | 1 | | gatekeeper | helm | gatekeeper/gatekeeper | 3.22.2 | 3 | | gcp-driver-installer | manifest | — | — | 3 | | gke-nccl-tcpxo | manifest | — | — | 4 | @@ -65,10 +63,8 @@ _Rendering fidelity:_ `catalog-parity: charts are rendered with the shared recip | nodewright-customizations | manifest | — | — | 5 | | nodewright-operator | helm | nodewright | v0.17.1 | 3 | | nvidia-dra-driver-gpu | helm | dra-driver-nvidia-gpu | 0.4.1 | 1 | -| nvidia-dra-driver-gpu-ocp | helm | dra-driver-nvidia-gpu | 0.4.1 | 1 | -| nvsentinel | helm | nvsentinel | v1.20.0 | 6 | -| prometheus-adapter | helm | prometheus-community/prometheus-adapter | 5.3.0 | 1 | -| prometheus-adapter-ocp | helm | prometheus-community/prometheus-adapter | 5.3.0 | 1 | +| nvsentinel | helm | nvsentinel | v1.9.0 | 6 | +| prometheus-adapter | helm | prometheus-community/prometheus-adapter | 5.3.0 | 0 | | prometheus-operator-crds | helm | prometheus-community/prometheus-operator-crds | 28.0.1 | 0 | | slinky-slurm | helm | slurm | 1.2.0 | 5 | | slinky-slurm-operator | helm | slurm-operator | 1.2.0 | 2 | @@ -97,12 +93,9 @@ _No images extracted._ ### aws-ebs-csi-driver -- `public.ecr.aws/csi-components/csi-attacher:v4.11.0-eksbuild.4` -- `public.ecr.aws/csi-components/csi-node-driver-registrar:v2.16.0-eksbuild.4` -- `public.ecr.aws/csi-components/csi-provisioner:v6.2.0-eksbuild.3` -- `public.ecr.aws/csi-components/csi-resizer:v2.1.0-eksbuild.4` -- `public.ecr.aws/csi-components/livenessprobe:v2.18.0-eksbuild.4` -- `public.ecr.aws/ebs-csi-driver/aws-ebs-csi-driver:v1.59.0` +> Warning: [INTERNAL] helm template failed: signal: killed + +_No images extracted._ ### aws-efa @@ -125,9 +118,7 @@ _No images extracted._ ### dynamo-platform -- `nats:2.10.21-alpine` -- `natsio/nats-server-config-reloader:0.16.0` -- `nvcr.io/nvidia/ai-dynamo/kubernetes-operator:1.2.1` +- `nvcr.io/nvidia/ai-dynamo/kubernetes-operator:1.4.1` ### gatekeeper @@ -298,7 +289,9 @@ _No images extracted._ ### prometheus-adapter -- `registry.k8s.io/prometheus-adapter/prometheus-adapter:v0.12.0` +> Warning: [INTERNAL] helm template failed: signal: killed + +_No images extracted._ ### prometheus-adapter-ocp diff --git a/docs/user/validation.md b/docs/user/validation.md index 936497bd2..702acac0e 100644 --- a/docs/user/validation.md +++ b/docs/user/validation.md @@ -382,14 +382,13 @@ default `dynamo-router` mode deploys a Dynamo frontend with load-aware least-loaded routing (`DYN_ROUTER_MODE=least-loaded`), which balances by each worker's active in-flight load so a transiently-slow worker stops receiving its full share (see issue #1197). Normal frontend-to-worker request/response traffic -uses Dynamo's request plane (Dynamo 1.2 defaults to TCP); AICR does not set -`DYN_REQUEST_PLANE=nats`. Workers still run the vLLM ZMQ KV-cache event -publisher relayed onto the NATS event plane, but least-loaded routing does not -consume those events. Set it to `gateway-epp` +uses Dynamo's request plane (Dynamo 1.4+ defaults to TCP); AICR does not set +`DYN_REQUEST_PLANE=nats`. Workers publish KV-cache events directly over ZMQ; +the KV router consumes them end-to-end with no NATS relay. Set it to `gateway-epp` to exercise GAIE/EPP: the validator deploys an EPP component, worker frontend sidecars in direct mode, and an HTTPRoute through the AICR-managed inference -gateway. The direct-mode sidecars honor EPP routing headers; they are not the -ZMQ-to-NATS relay. +gateway. The direct-mode sidecars honor EPP routing headers; they do not +relay KV events. **Model-weights cache and `AICR_INFERENCE_PERF_MODEL_CACHE_STORAGE_CLASS`.** The benchmark downloads the model **once** into a PVC and serves all workers from it (on by default; diff --git a/pkg/bundler/bundler_test.go b/pkg/bundler/bundler_test.go index 04fcec2cb..cb15459a5 100644 --- a/pkg/bundler/bundler_test.go +++ b/pkg/bundler/bundler_test.go @@ -2365,74 +2365,6 @@ func TestApplyNodeSchedulingOverrides_StorageClass(t *testing.T) { } } -func TestApplyNodeSchedulingOverrides_DynamoPlatformStorageClass(t *testing.T) { - const scPath = "nats.config.jetstream.fileStore.pvc.storageClassName" - - registry, err := recipe.GetComponentRegistry() - if err != nil { - t.Fatalf("GetComponentRegistry() error = %v", err) - } - comp := registry.Get("dynamo-platform") - if comp == nil { - t.Fatal("registry missing dynamo-platform") - } - if !slices.Contains(comp.GetStorageClassPaths(), scPath) { - t.Fatalf("registry storageClassPaths for dynamo-platform = %v, want %q", - comp.GetStorageClassPaths(), scPath) - } - - tests := []struct { - name string - cfgOpts []config.Option - initialValues map[string]string - wantValue string - }{ - { - name: "global storageClass injected into bundled NATS PVC", - cfgOpts: []config.Option{config.WithStorageClass("my-storage-class")}, - wantValue: "my-storage-class", - }, - { - name: "explicit dynamo --set wins over global --storage-class", - cfgOpts: []config.Option{ - config.WithStorageClass("my-storage-class"), - config.WithValueOverrides(map[string]map[string]string{ - "dynamo": {scPath: "explicit-gp2"}, - }), - }, - initialValues: map[string]string{scPath: "explicit-gp2"}, - wantValue: "explicit-gp2", - }, - } - - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - cfg := config.NewConfig(tt.cfgOpts...) - b, err := New(WithConfig(cfg)) - if err != nil { - t.Fatalf("New() error = %v", err) - } - - values := map[string]any{} - if len(tt.initialValues) > 0 { - if applyErr := component.ApplyMapOverrides(values, tt.initialValues); applyErr != nil { - t.Fatalf("ApplyMapOverrides() setup error = %v", applyErr) - } - } - - b.applyNodeSchedulingOverrides("dynamo-platform", values, nil, schedulingPathPolicy{}) - - got, ok := component.GetValueByPath(values, scPath) - if !ok { - t.Fatal("storageClassName not injected") - } - if got != tt.wantValue { - t.Errorf("storageClassName = %v, want %q", got, tt.wantValue) - } - }) - } -} - // TestApplyNodeSchedulingOverrides_RespectsRecipeSetPaths verifies the // precedence rule that paths the user explicitly populated via the recipe // overlay's inline overrides or CLI --set are NOT overwritten by CLI/config @@ -3029,76 +2961,6 @@ func TestWarnMissingStorageClassForPVCs(t *testing.T) { } } -func TestWarnMissingStorageClassForPVCs_DynamoPlatformNATS(t *testing.T) { - const scPath = "nats.config.jetstream.fileStore.pvc.storageClassName" - - recipeResult := &recipe.RecipeResult{ - ComponentRefs: []recipe.ComponentRef{{ - Name: "dynamo-platform", - ValuesFile: "components/dynamo-platform/values.yaml", - }}, - } - - tests := []struct { - name string - setupValues func(map[string]any) - wantWarning bool - }{ - { - name: "warns when bundled NATS PVC omits storageClassName", - wantWarning: true, - }, - { - name: "does not warn when bundled NATS PVC has storageClassName", - setupValues: func(values map[string]any) { - component.SetValueByPath(values, scPath, "gp3") - }, - }, - } - - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - values, err := recipeResult.GetValuesForComponent("dynamo-platform") - if err != nil { - t.Fatalf("GetValuesForComponent(dynamo-platform): %v", err) - } - if tt.setupValues != nil { - tt.setupValues(values) - } - - b, err := New() - if err != nil { - t.Fatalf("New() error = %v", err) - } - - err = b.warnMissingStorageClassForPVCs(context.Background(), recipeResult, map[string]map[string]any{ - "dynamo-platform": values, - }) - if err != nil { - t.Fatalf("warnMissingStorageClassForPVCs() error = %v", err) - } - - if gotWarning := len(b.warnings) > 0; gotWarning != tt.wantWarning { - t.Fatalf("warning present = %v, want %v; warnings = %v", gotWarning, tt.wantWarning, b.warnings) - } - - if tt.wantWarning { - warning := b.warnings[0] - for _, want := range []string{ - "Warning: dynamo-platform renders a PVC without storageClassName", - scPath, - "--storage-class ", - "--set dynamo-platform:" + scPath + "=", - } { - if !strings.Contains(warning, want) { - t.Errorf("warning = %q, want substring %q", warning, want) - } - } - } - }) - } -} - // TestAgentgatewayComponentExistsInRegistry locks agentgatewayComponentName to a // real registry entry. resolveAgentgatewayExposure keys into componentValues by // this name; if the "agentgateway" component were renamed in recipes/registry.yaml, diff --git a/pkg/evidence/cncf/scripts/collect-evidence.sh b/pkg/evidence/cncf/scripts/collect-evidence.sh index fa128fd86..e82c0ba17 100755 --- a/pkg/evidence/cncf/scripts/collect-evidence.sh +++ b/pkg/evidence/cncf/scripts/collect-evidence.sh @@ -2860,9 +2860,8 @@ webhooks operational, and custom resources reconciled. 2. **Custom Resource Definitions** — 6 Dynamo CRDs registered (DynamoGraphDeployment, DynamoComponentDeployment, etc.) 3. **Webhooks Operational** — a webhook-invalid DynamoGraphDeployment must be rejected with a webhook-attributed message 4. **Custom Resource Reconciled** — `DynamoGraphDeployment/vllm-agg` reconciled into running workload pods via PodCliques -5. **Supporting Services** — NATS running for Dynamo's Kubernetes event plane - -The result is stated by the verdict line at the end of this section, after the checks run. +5. **Supporting Services** — ZMQ-based KV-cache event plane (no NATS; Dynamo 1.4+ default) +6. **Result: PASS** --- diff --git a/pkg/evidence/cncf/scripts/manifests/dynamo-vllm-agg.yaml b/pkg/evidence/cncf/scripts/manifests/dynamo-vllm-agg.yaml index d1f487815..0cc3abfc9 100644 --- a/pkg/evidence/cncf/scripts/manifests/dynamo-vllm-agg.yaml +++ b/pkg/evidence/cncf/scripts/manifests/dynamo-vllm-agg.yaml @@ -82,7 +82,7 @@ spec: effect: NoExecute containers: - name: main - image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.2.1 + image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.4.1 env: - name: SERVED_MODEL_NAME value: Qwen/Qwen3-0.6B @@ -110,7 +110,7 @@ spec: effect: NoSchedule containers: - name: main - image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.2.1 + image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.4.1 workingDir: /workspace/examples/backends/vllm command: ["python3", "-m", "dynamo.vllm"] args: diff --git a/recipes/components/dynamo-platform/values.yaml b/recipes/components/dynamo-platform/values.yaml index e6fbeece9..268bf7f25 100644 --- a/recipes/components/dynamo-platform/values.yaml +++ b/recipes/components/dynamo-platform/values.yaml @@ -12,7 +12,7 @@ # See the License for the specific language governing permissions and # limitations under the License. -# Dynamo Platform Helm values (v1.2.1) +# Dynamo Platform Helm values (v1.4.1) # NVIDIA Dynamo inference serving platform operator. # Provides OpenAI-compatible endpoints, KV-cache-aware routing, # disaggregated prefill/decode, and SLA-driven autoscaling. @@ -23,13 +23,18 @@ global: # Kubernetes-native discovery replaces etcd for AICR Dynamo workloads. etcd: install: false - # Dynamo 1.2 defaults the runtime event plane to NATS for distributed - # Kubernetes backends. Install the bundled NATS chart so workloads can use - # the runtime default instead of per-workload DYN_EVENT_PLANE overrides. - # Request/response traffic stays on Dynamo's request plane default (TCP) - # unless a workload explicitly sets DYN_REQUEST_PLANE. - nats: - install: true + # Dynamo 1.4+ defaults the request plane to TCP and the KV event plane to + # ZMQ, so the bundled NATS chart is no longer needed for AICR's defaults. + # global.nats.install is intentionally left unset (chart default: false). + # Re-enable only if a workload explicitly opts into a NATS-based transport + # (legacy DYN_REQUEST_PLANE=nats, DYN_EVENT_PLANE=nats, or a NATS + # request-trace sink), via global.nats.install: true. + # + # Upgrade note: on an in-place `helm upgrade` of a standing cluster, + # accepting this default removes bundled NATS and drops NATS_SERVER from + # operator-managed workload specs, which triggers rolling restarts. Fresh + # bundle installs are unaffected. Pass --set global.nats.install=true + # during the upgrade to opt out and avoid the restart. # Disable kai-scheduler sub-chart install — managed as a separate AICR component. # Keep enabled: true so the Dynamo operator detects and uses the external scheduler. kai-scheduler: @@ -41,17 +46,6 @@ global: install: false enabled: true -nats: - config: - jetstream: - fileStore: - pvc: - # Mirror the chart default so AICR can warn when no StorageClass is - # selected and inject --storage-class into the bundled NATS PVC. - enabled: true - size: 10Gi - storageClassName: "" - dynamo-operator: # Upgrade CRDs via the platform chart (no separate dynamo-crds 1.0 chart) upgradeCRD: true @@ -63,4 +57,4 @@ dynamo-operator: # PodMonitor CRs are auto-created by the operator for metric discovery. dynamo: metrics: - prometheusEndpoint: "http://kube-prometheus-prometheus.monitoring.svc.cluster.local:9090" + prometheusEndpoint: "http://kube-prometheus-prometheus.monitoring.svc.cluster.local:9090" \ No newline at end of file diff --git a/recipes/registry.yaml b/recipes/registry.yaml index abbf297a2..486f32097 100644 --- a/recipes/registry.yaml +++ b/recipes/registry.yaml @@ -613,18 +613,14 @@ components: helm: defaultRepository: https://helm.ngc.nvidia.com/nvidia/ai-dynamo defaultChart: dynamo-platform - defaultVersion: "1.2.1" + defaultVersion: "1.4.1" defaultNamespace: dynamo-system - storageClassPaths: - - nats.config.jetstream.fileStore.pvc.storageClassName nodeScheduling: system: nodeSelectorPaths: - dynamo-operator.controllerManager.nodeSelector - - nats.podTemplate.merge.spec.nodeSelector tolerationPaths: - dynamo-operator.controllerManager.tolerations - - nats.podTemplate.merge.spec.tolerations - name: agentgateway-crds displayName: agentgateway-crds diff --git a/tests/chainsaw/ai-conformance/README.md b/tests/chainsaw/ai-conformance/README.md index 017597b82..c133f0d80 100644 --- a/tests/chainsaw/ai-conformance/README.md +++ b/tests/chainsaw/ai-conformance/README.md @@ -163,7 +163,7 @@ chainsaw test \ | Namespaces, CRDs | 2m | Should exist immediately after deployment | | cert-manager, agentgateway, skyhook, monitoring, kai-scheduler | 5m | Standard Deployment rollout | | gpu-operator, nvidia-dra-driver-gpu | 10m | GPU driver compilation on nodes is slow | -| dynamo-platform | 5m | Operator + Grove + NATS event-plane startup | +| dynamo-platform | 5m | Operator + Grove startup | ## Assertion Patterns diff --git a/tests/chainsaw/ai-conformance/cluster/assert-dynamo.yaml b/tests/chainsaw/ai-conformance/cluster/assert-dynamo.yaml index a5037510c..a6e6a9d8f 100644 --- a/tests/chainsaw/ai-conformance/cluster/assert-dynamo.yaml +++ b/tests/chainsaw/ai-conformance/cluster/assert-dynamo.yaml @@ -13,12 +13,13 @@ # limitations under the License. # Assert Dynamo platform components are healthy. -# Chart: dynamo-platform 1.2.1 +# Chart: dynamo-platform 1.4.1 # Provides NVIDIA Dynamo inference serving: OpenAI-compatible endpoints, # KV-cache-aware routing, disaggregated prefill/decode, SLA-driven autoscaling. # -# AICR uses Kubernetes-native discovery, so etcd remains disabled. NATS is -# enabled for Dynamo's Kubernetes event plane and worker KV-cache events. +# AICR uses Kubernetes-native discovery, so etcd remains disabled. As of +# Dynamo 1.4+, the request plane defaults to TCP and the KV event plane to +# ZMQ, so bundled NATS is no longer installed by default (NVIDIA/aicr#1836). # Dynamo Operator — manages DynamoComponent and DynamoGraphDeployment CRs apiVersion: apps/v1 @@ -30,15 +31,6 @@ status: (conditions[?type == 'Available']): - status: "True" --- -# NATS — event plane used by Dynamo KV-cache-aware routing on Kubernetes -apiVersion: apps/v1 -kind: StatefulSet -metadata: - name: dynamo-platform-nats - namespace: dynamo-system -status: - (readyReplicas > `0`): true ---- # Grove Operator — pod lifecycle management for DynamoGraphDeployments apiVersion: apps/v1 kind: Deployment diff --git a/tests/chainsaw/ai-conformance/cluster/chainsaw-test.yaml b/tests/chainsaw/ai-conformance/cluster/chainsaw-test.yaml index d858f14b1..c19893381 100644 --- a/tests/chainsaw/ai-conformance/cluster/chainsaw-test.yaml +++ b/tests/chainsaw/ai-conformance/cluster/chainsaw-test.yaml @@ -116,7 +116,7 @@ spec: # ── Dynamo Platform ──────────────────────────────────────────────── - name: assert-dynamo - description: Verify Dynamo operator, Grove, and NATS event plane are healthy. + description: Verify Dynamo operator and Grove are healthy. try: - assert: file: assert-dynamo.yaml diff --git a/tests/chainsaw/ai-conformance/kind-inference-dynamo/assert-dynamo.yaml b/tests/chainsaw/ai-conformance/kind-inference-dynamo/assert-dynamo.yaml index 9401afa0a..04c9153a4 100644 --- a/tests/chainsaw/ai-conformance/kind-inference-dynamo/assert-dynamo.yaml +++ b/tests/chainsaw/ai-conformance/kind-inference-dynamo/assert-dynamo.yaml @@ -13,8 +13,9 @@ # limitations under the License. # Assert Dynamo platform components are healthy on the kind inference stack. -# AICR uses Kubernetes-native discovery plus NATS for Dynamo's Kubernetes -# event plane and worker KV-cache events. +# AICR uses Kubernetes-native discovery. As of Dynamo 1.4+, the request +# plane defaults to TCP and the KV event plane to ZMQ, so bundled NATS is +# no longer installed by default (NVIDIA/aicr#1836). apiVersion: apps/v1 kind: Deployment metadata: @@ -25,14 +26,6 @@ status: - status: "True" --- apiVersion: apps/v1 -kind: StatefulSet -metadata: - name: dynamo-platform-nats - namespace: dynamo-system -status: - (readyReplicas > `0`): true ---- -apiVersion: apps/v1 kind: Deployment metadata: name: grove-operator diff --git a/tests/chainsaw/ai-conformance/kind-inference-dynamo/chainsaw-test.yaml b/tests/chainsaw/ai-conformance/kind-inference-dynamo/chainsaw-test.yaml index eefb665ad..d8e0b87b0 100644 --- a/tests/chainsaw/ai-conformance/kind-inference-dynamo/chainsaw-test.yaml +++ b/tests/chainsaw/ai-conformance/kind-inference-dynamo/chainsaw-test.yaml @@ -118,7 +118,7 @@ spec: # ── Dynamo Platform ──────────────────────────────────────────────── - name: assert-dynamo - description: Verify Dynamo operator, Grove, and NATS event plane are healthy. + description: Verify Dynamo operator and Grove are healthy. try: - assert: file: assert-dynamo.yaml diff --git a/tests/manifests/dynamo-vllm-smoke-test.yaml b/tests/manifests/dynamo-vllm-smoke-test.yaml index 6497a8bcb..da1ff09bd 100644 --- a/tests/manifests/dynamo-vllm-smoke-test.yaml +++ b/tests/manifests/dynamo-vllm-smoke-test.yaml @@ -20,11 +20,11 @@ # serving Qwen/Qwen3-0.6B (public, ungated, ~1.2GB). # No HuggingFace token required. # -# Uses the Dynamo 1.2 Kubernetes path: Kubernetes-native discovery plus the -# NATS-backed event plane from the dynamo-platform chart. Request/response -# traffic uses Dynamo's request plane default (TCP). The frontend runs -# KV-cache-aware routing; the local vLLM engine publishes KV events through -# ZMQ and the Dynamo worker runtime relays them onto the NATS event plane. +# Uses the Dynamo 1.4+ Kubernetes path: Kubernetes-native discovery, with +# the request plane on TCP (default) and the KV event plane on ZMQ +# (default) — no bundled NATS relay in between. The frontend runs +# KV-cache-aware routing; the local vLLM engine publishes KV events over +# ZMQ and the KV router consumes them directly, end-to-end. # # Usage: # kubectl apply -f tests/manifests/dynamo-vllm-smoke-test.yaml -n dynamo-system @@ -49,12 +49,16 @@ spec: spec: containers: - name: main - image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.2.1 + image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.4.1 env: - name: SERVED_MODEL_NAME value: Qwen/Qwen3-0.6B - name: DYN_ROUTER_MODE value: kv + - name: DYN_REQUEST_PLANE + value: tcp + - name: DYN_EVENT_PLANE + value: zmq - name: VllmDecodeWorker type: worker replicas: 1 @@ -67,9 +71,14 @@ spec: effect: NoSchedule containers: - name: main - image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.2.1 + image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.4.1 workingDir: /workspace/examples/backends/vllm command: ["python3", "-m", "dynamo.vllm"] + env: + - name: DYN_REQUEST_PLANE + value: tcp + - name: DYN_EVENT_PLANE + value: zmq args: - --model - Qwen/Qwen3-0.6B diff --git a/tests/uat/lib/phases.sh b/tests/uat/lib/phases.sh index 2229a9fad..327e6cec1 100644 --- a/tests/uat/lib/phases.sh +++ b/tests/uat/lib/phases.sh @@ -162,12 +162,10 @@ SERVE_NAMESPACE="${SERVE_NAMESPACE:-dynamo-workload}" SERVE_NAME="${SERVE_NAME:-vllm-agg}" SERVE_QUEUE="${SERVE_QUEUE:-dynamo}" SERVE_MODEL="${SERVE_MODEL:-Qwen/Qwen3-0.6B}" -SERVE_RUNTIME_IMAGE="${SERVE_RUNTIME_IMAGE:-nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.2.1}" -# GPU pool placement, applied to BOTH graph components (#1644 — it keeps the -# Frontend off the small CPU nodes, where its ~12GB image pull blew the -# readiness budget). The demo pins nodeGroup=gpu-worker; every UAT cluster -# labels its GPU pool the same way (tests/uat/*/cluster-config.yaml), so the -# default lands the graph on the GPU pool on any of the clouds. +SERVE_RUNTIME_IMAGE="${SERVE_RUNTIME_IMAGE:-nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.4.1}" +# GPU worker placement. The demo pins nodeGroup=gpu-worker; both UAT clusters +# label their GPU pool the same way (tests/uat/*/cluster-config.yaml), so the +# default lands the decode worker on the GPU node on either cloud. SERVE_GPU_NODE_SELECTOR_KEY="${SERVE_GPU_NODE_SELECTOR_KEY:-nodeGroup}" SERVE_GPU_NODE_SELECTOR_VALUE="${SERVE_GPU_NODE_SELECTOR_VALUE:-gpu-worker}" SERVE_FRONTEND_PORT="${SERVE_FRONTEND_PORT:-8000}" diff --git a/validators/performance/inference_perf_constraint.go b/validators/performance/inference_perf_constraint.go index 9da3fab94..592954765 100644 --- a/validators/performance/inference_perf_constraint.go +++ b/validators/performance/inference_perf_constraint.go @@ -2746,7 +2746,7 @@ func resolveModel(ctx *validators.Context) string { // `gateway-epp` switches to Gateway API Inference Extension: EPP // performs KV-aware endpoint selection and worker frontend sidecars run in // direct mode so they honor EPP's routing headers. The sidecars do not relay -// local vLLM ZMQ KV events onto NATS; that relay is handled by the worker +// KV events; workers publish ZMQ KV events directly to the KV router. // runtime. func resolveRoutingMode(ctx *validators.Context) (inferenceRoutingMode, error) { if c, ok := findPerformanceConstraint(ctx, perfConstraintRoutingMode); ok { diff --git a/validators/performance/model_cache.go b/validators/performance/model_cache.go index 229b46987..e6da370a0 100644 --- a/validators/performance/model_cache.go +++ b/validators/performance/model_cache.go @@ -101,7 +101,7 @@ const ( // ResolveImage for registry-override parity is tracked in #1159. Note that // registry parity alone is not air-gap support: the populate Job's // snapshot_download still reaches huggingface.co for the weights. - cacheWorkerImage = "nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.2.1" + cacheWorkerImage = "nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.4.1" // Resource requests for the populate container. snapshot_download is // network/IO-bound, not compute-bound, so requests stay small; they exist so diff --git a/validators/performance/testdata/inference/dynamo-deployment-gateway-epp.yaml b/validators/performance/testdata/inference/dynamo-deployment-gateway-epp.yaml index 88c11ab44..a7be0bf98 100644 --- a/validators/performance/testdata/inference/dynamo-deployment-gateway-epp.yaml +++ b/validators/performance/testdata/inference/dynamo-deployment-gateway-epp.yaml @@ -15,8 +15,8 @@ # Dynamo inference workload for performance benchmarking through GAIE/EPP. # EPP performs KV-cache-aware endpoint selection and worker frontend sidecars # run in direct mode so they honor EPP's routing headers. Direct mode is not -# the local ZMQ-to-NATS relay; the Dynamo worker runtime relays vLLM KV-cache -# events from the local ZMQ publisher onto the NATS-backed event plane. +# workers publish KV-cache events over ZMQ directly to the KV router; +# no NATS relay in the 1.4+ default path. # # Worker GPU wiring is MODE-DISPATCHED (capability-driven, issue #1327): DRA # ResourceClaimTemplate on nodes with usable full-GPU DRA, device-plugin @@ -41,7 +41,7 @@ spec: spec: containers: - name: main - image: nvcr.io/nvidia/ai-dynamo/epp-image:1.2.1 + image: nvcr.io/nvidia/ai-dynamo/epp-image:1.4.1 env: - name: DYN_KV_CACHE_BLOCK_SIZE value: "128" @@ -96,7 +96,7 @@ spec: spec: containers: - name: main - image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.2.1 + image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.4.1 workingDir: /workspace/examples/backends/vllm # Shell wrapper appends GKE's managed-device-plugin driver-lib # mount (/usr/local/nvidia/lib64) to LD_LIBRARY_PATH — the @@ -134,7 +134,7 @@ spec: key: token optional: true - name: sidecar-frontend - image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.2.1 + image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.4.1 args: - -m - dynamo.frontend diff --git a/validators/performance/testdata/inference/dynamo-deployment.yaml b/validators/performance/testdata/inference/dynamo-deployment.yaml index 07e12d4ca..e7ea81e59 100644 --- a/validators/performance/testdata/inference/dynamo-deployment.yaml +++ b/validators/performance/testdata/inference/dynamo-deployment.yaml @@ -16,8 +16,8 @@ # Deploys a vLLM aggregated backend with a small model for smoke testing. # Worker replicas scale to match available GPUs (1 GPU per replica). # Frontend-to-worker traffic uses Dynamo's request plane default (TCP). The -# local vLLM ZMQ KV-event publisher feeds the worker runtime, which relays -# KV-cache events onto the NATS-backed Dynamo event plane. +# local vLLM ZMQ KV-event publisher feeds the KV router directly; +# no NATS relay in the 1.4+ default path. # # Worker GPU wiring is MODE-DISPATCHED (capability-driven, issue #1327): on # nodes with usable full-GPU DRA the workers bind a DRA ResourceClaimTemplate @@ -45,7 +45,7 @@ spec: spec: containers: - name: main - image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.2.1 + image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.4.1 env: - name: SERVED_MODEL_NAME # Quoted so a scalar-looking model ID (e.g. a pure-numeric or @@ -84,7 +84,7 @@ spec: spec: containers: - name: main - image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.2.1 + image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.4.1 workingDir: /workspace/examples/backends/vllm # GKE's managed device plugin (the gke-default gpuStack value) # mounts the node's driver tree at /usr/local/nvidia via its