diff --git a/deploy/helm/llm-request-router/Makefile b/deploy/helm/llm-request-router/Makefile index 605e20cb5..84763bebc 100644 --- a/deploy/helm/llm-request-router/Makefile +++ b/deploy/helm/llm-request-router/Makefile @@ -71,10 +71,10 @@ check-pki-render: sh ./scripts/check-pki-render.sh check-multi-replica-render: - sh ./scripts/check-multi-replica-render.sh + bash ./scripts/check-multi-replica-render.sh check-backend-router-render: - sh ./scripts/check-backend-router-render.sh + bash ./scripts/check-backend-router-render.sh clean: rm -rf ./packaged-charts bin diff --git a/deploy/helm/llm-request-router/README.md b/deploy/helm/llm-request-router/README.md index 572ad5370..8b0f3ccd8 100644 --- a/deploy/helm/llm-request-router/README.md +++ b/deploy/helm/llm-request-router/README.md @@ -4,12 +4,12 @@ This repository contains the Helm chart for deploying the NVCF LLM Request Route ## Overview -The chart packages the LLM Request Router StatefulSet with HTTP and gRPC -services, a metrics endpoint, and a headless service for multi-instance DNS -discovery. It can also deploy the Stargate Kubernetes backend router for -worker gRPC registration and reverse QUIC tunnels through a shared Gateway or -load balancer. The backend router selects the correct Stargate pod from gRPC -authority and QUIC SNI. +The chart packages the LLM Request Router as either a Deployment or a +StatefulSet, with Deployment as the default. It includes HTTP and gRPC +services, a metrics endpoint, and a headless service. It can also deploy the +Stargate Kubernetes backend router for worker gRPC registration and reverse +QUIC tunnels through a shared Gateway or load balancer. The backend router +selects the correct Stargate pod from gRPC authority and QUIC SNI. A Vault Agent sidecar is configured to fetch a service token from a Vault or OpenBao backend. The application reads `nvcfApiToken` from @@ -28,11 +28,37 @@ llmRequestRouter: tag: ``` -Single-replica deployments may use self-only discovery with `llmRequestRouter.discovery.disableDnsDiscovery=true`. Multi-replica deployments require DNS discovery and stable per-pod identity, so the chart fails rendering if DNS discovery is disabled while `llmRequestRouter.replicaCount > 1`. For multi-replica deployments, the default advertised hostname template is `{pod_name}...svc.cluster.local`; the StatefulSet and headless service provide the stable pod DNS names required for router replicas to discover each other and share backend registrations. - -`llmRequestRouter.kubernetes.advertisedHostnameTemplate` supports the Stargate placeholders `{pod_name}` and `{namespace}`. Stargate resolves both placeholders at runtime. For certificate validation, the chart substitutes the deployment namespace and a representative StatefulSet pod name. `{pod_name}` must stay within the leftmost DNS label when certificate coverage relies on a wildcard. When `llmRequestRouter.certificate.enabled=true`, `certificate.dnsNames` must cover the advertised hostname with either a case-insensitive exact name or a valid leftmost `*.` wildcard. A wildcard covers exactly one label and requires at least two suffix labels. For example, `*.nvcf.example.internal` covers `{pod_name}.nvcf.example.internal`, but `*.example.internal` does not cover `{pod_name}.nvcf.example.internal`. - -Upgrading from a chart version that rendered a Deployment can briefly run both the old Deployment and new StatefulSet during `helm upgrade` while Helm replaces the workload kind. +Single-replica Deployments automatically use self-only discovery so their +headless Service cannot introduce dashed-IP SRV aliases. A multi-replica +Deployment requires the backend router. The default `null` value for +`llmRequestRouter.backendRouter.enabled` enables it automatically in that +topology; explicitly setting `false` is rejected. The backend router builds +Watch responses and forwarding routes from the same +EndpointSlice snapshot. Each ready endpoint is keyed by its Pod +`targetRef.name`, so one pod produces one canonical identity even when DNS also +exposes a dashed-IP SRV alias. A multi-replica StatefulSet can instead run +without the backend router and retain direct headless Service SRV discovery. +`llmRequestRouter.discovery.watchHeartbeatMs` controls the maximum interval +between unchanged Watch snapshots from both Stargate and the backend router. + +`llmRequestRouter.kubernetes.advertisedHostnameTemplate` supports the Stargate +placeholders `{pod_name}` and `{namespace}`. Stargate resolves both placeholders +at runtime. For certificate validation, the chart substitutes the deployment +namespace and a representative pod name. `{pod_name}` must stay within the +leftmost DNS label when certificate coverage relies on a wildcard. When +`llmRequestRouter.certificate.enabled=true`, `certificate.dnsNames` must cover +the advertised hostname with either a case-insensitive exact name or a valid +leftmost `*.` wildcard. A wildcard covers exactly one label and requires at +least two suffix labels. For example, `*.nvcf.example.internal` covers +`{pod_name}.nvcf.example.internal`, but `*.example.internal` does not cover +`{pod_name}.nvcf.example.internal`. + +Existing installations that currently run the StatefulSet must set +`llmRequestRouter.workload.kind=StatefulSet` before upgrading to this chart. +Changing `workload.kind` is a controlled migration, not an in-place Kubernetes +mutation. Plan a maintenance window, remove or rename the old workload, and +verify that only the selected kind owns the request-router Pods before scaling +it. A plain Helm upgrade across workload kinds can briefly run both workloads. ## Prerequisites @@ -80,6 +106,8 @@ Important settings to review before deployment: - `llmRequestRouter.image.*` for the router container image - `llmRequestRouter.imagePullSecrets` for private registry access +- `llmRequestRouter.workload.kind` to select `Deployment` (default) or `StatefulSet` +- `llmRequestRouter.workload.deployment.strategy` and `llmRequestRouter.workload.statefulSet.*` for workload-specific rollout settings - `llmRequestRouter.replicaCount`, resource requests, and limits for your environment - `llmRequestRouter.service.*` for HTTP, gRPC, metrics, and headless service ports - `llmRequestRouter.backendRouter.*` for multi-replica worker gRPC and reverse-tunnel routing @@ -96,9 +124,10 @@ The default values include development-oriented placeholders. Override them befo ## Backend Worker Routing -Enable `llmRequestRouter.backendRouter.enabled` when workers reach a -multi-replica request router through a shared endpoint. Set both pylon dial -addresses to the external endpoints that workers can resolve: +The backend router is enabled automatically for a multi-replica Deployment. +Set `llmRequestRouter.backendRouter.enabled=true` explicitly when workers reach +another supported workload topology through a shared endpoint. Set both pylon +dial addresses to the external endpoints that workers can resolve: ```yaml llmRequestRouter: @@ -114,8 +143,11 @@ chart appVersion. That image must contain `/usr/local/bin/stargate-k8s-router`; override `backendRouter.image.*` only to validate a different Stargate build. -The backend router watches EndpointSlices. The chart creates a dedicated -ServiceAccount by default and binds a namespaced Role to it when +The backend router watches EndpointSlices and publishes those ready targets +directly through `WatchStargates`. It uses the same snapshot for gRPC and QUIC +forwarding, so a removed or replaced Pod cannot remain as a discovery-only +target. The chart creates a dedicated ServiceAccount by default and binds a +namespaced Role to it when `llmRequestRouter.rbac.create=true`. When `llmRequestRouter.backendRouter.serviceAccount.create=false`, set `llmRequestRouter.backendRouter.serviceAccount.name` to an existing account. diff --git a/deploy/helm/llm-request-router/llm-request-router/templates/_helpers.tpl b/deploy/helm/llm-request-router/llm-request-router/templates/_helpers.tpl index 3b1600530..d3d2da83d 100644 --- a/deploy/helm/llm-request-router/llm-request-router/templates/_helpers.tpl +++ b/deploy/helm/llm-request-router/llm-request-router/templates/_helpers.tpl @@ -76,6 +76,30 @@ app.kubernetes.io/managed-by: {{ .Release.Service }} {{- default .Release.Namespace .Values.llmRequestRouter.namespace -}} {{- end -}} +{{- define "llm-request-router.workloadKind" -}} +{{- dig "workload" "kind" "Deployment" .Values.llmRequestRouter | toString -}} +{{- end -}} + +{{/* +An unset backendRouter.enabled follows the workload contract: a multi-replica +Deployment needs the EndpointSlice router, while StatefulSet and single-replica +direct modes retain their previous behavior. An explicit boolean always wins; +deployment.yaml rejects the unsafe explicit-false combination. +*/}} +{{- define "llm-request-router.backendRouterEnabled" -}} +{{- $backendRouter := .Values.llmRequestRouter.backendRouter | default dict -}} +{{- $configured := get $backendRouter "enabled" -}} +{{- if kindIs "bool" $configured -}} +{{- $configured -}} +{{- else if and + (eq (include "llm-request-router.workloadKind" .) "Deployment") + (gt (.Values.llmRequestRouter.replicaCount | int) 1) -}} +true +{{- else -}} +false +{{- end -}} +{{- end -}} + {{- define "llm-request-router.isValidDnsName" -}} {{- $name := .name | toString | lower -}} {{- $labels := splitList "." $name -}} @@ -103,7 +127,7 @@ rendering must agree on that list, so both read it from here. {{- define "llm-request-router.effectiveCertificateDnsNames" -}} {{- $certificate := .Values.llmRequestRouter.certificate | default dict -}} {{- $dnsNames := dig "dnsNames" (list) $certificate -}} -{{- if dig "backendRouter" "enabled" false .Values.llmRequestRouter -}} +{{- if eq (include "llm-request-router.backendRouterEnabled" .) "true" -}} {{- $wildcard := replace "{pod_name}" "*" (include "llm-request-router.advertisedHostnameTemplate" .) -}} {{- if not (has $wildcard $dnsNames) -}} {{- $dnsNames = append $dnsNames $wildcard -}} @@ -250,7 +274,7 @@ Coverage, matching the role flags: {{- define "llm-request-router.advertisedHostnameTemplate" -}} {{- $configured := .Values.llmRequestRouter.kubernetes.advertisedHostnameTemplate -}} -{{- $backendRouterEnabled := dig "backendRouter" "enabled" false .Values.llmRequestRouter -}} +{{- $backendRouterEnabled := eq (include "llm-request-router.backendRouterEnabled" .) "true" -}} {{- if and $backendRouterEnabled $configured (ne (len (splitList "{pod_name}" $configured)) 2) -}} {{- fail "llmRequestRouter.kubernetes.advertisedHostnameTemplate must contain exactly one {pod_name} when backendRouter.enabled is true" -}} {{- end -}} diff --git a/deploy/helm/llm-request-router/llm-request-router/templates/backend-router-poddisruptionbudget.yaml b/deploy/helm/llm-request-router/llm-request-router/templates/backend-router-poddisruptionbudget.yaml index bfb03dc2d..3fa7df15d 100644 --- a/deploy/helm/llm-request-router/llm-request-router/templates/backend-router-poddisruptionbudget.yaml +++ b/deploy/helm/llm-request-router/llm-request-router/templates/backend-router-poddisruptionbudget.yaml @@ -3,7 +3,7 @@ {{- $backendRouter := .Values.llmRequestRouter.backendRouter -}} {{- $pdb := $backendRouter.podDisruptionBudget | default dict -}} -{{- if and $backendRouter.enabled $pdb.enabled }} +{{- if and (eq (include "llm-request-router.backendRouterEnabled" .) "true") $pdb.enabled }} {{- $pdbMinAvailable := $pdb.minAvailable | toString -}} {{- $pdbMaxUnavailable := $pdb.maxUnavailable | toString -}} {{- if and (ne $pdbMinAvailable "") (ne $pdbMaxUnavailable "") }} diff --git a/deploy/helm/llm-request-router/llm-request-router/templates/backend-router-rbac.yaml b/deploy/helm/llm-request-router/llm-request-router/templates/backend-router-rbac.yaml index 6e8dae34a..6fbae63f6 100644 --- a/deploy/helm/llm-request-router/llm-request-router/templates/backend-router-rbac.yaml +++ b/deploy/helm/llm-request-router/llm-request-router/templates/backend-router-rbac.yaml @@ -1,7 +1,7 @@ # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 -{{- if and .Values.llmRequestRouter.backendRouter.enabled .Values.llmRequestRouter.rbac.create }} +{{- if and (eq (include "llm-request-router.backendRouterEnabled" .) "true") .Values.llmRequestRouter.rbac.create }} apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: diff --git a/deploy/helm/llm-request-router/llm-request-router/templates/backend-router-serviceaccount.yaml b/deploy/helm/llm-request-router/llm-request-router/templates/backend-router-serviceaccount.yaml index e92bcdb85..750dd9042 100644 --- a/deploy/helm/llm-request-router/llm-request-router/templates/backend-router-serviceaccount.yaml +++ b/deploy/helm/llm-request-router/llm-request-router/templates/backend-router-serviceaccount.yaml @@ -1,7 +1,7 @@ # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 -{{- if and .Values.llmRequestRouter.backendRouter.enabled .Values.llmRequestRouter.backendRouter.serviceAccount.create }} +{{- if and (eq (include "llm-request-router.backendRouterEnabled" .) "true") .Values.llmRequestRouter.backendRouter.serviceAccount.create }} apiVersion: v1 kind: ServiceAccount metadata: diff --git a/deploy/helm/llm-request-router/llm-request-router/templates/backend-router-servicemonitor.yaml b/deploy/helm/llm-request-router/llm-request-router/templates/backend-router-servicemonitor.yaml index b3b48854a..7569c2ae4 100644 --- a/deploy/helm/llm-request-router/llm-request-router/templates/backend-router-servicemonitor.yaml +++ b/deploy/helm/llm-request-router/llm-request-router/templates/backend-router-servicemonitor.yaml @@ -3,7 +3,7 @@ {{- $metricsEnabled := dig "metrics" "enabled" false .Values.llmRequestRouter }} {{- $serviceMonitorEnabled := dig "metrics" "serviceMonitor" "enabled" false .Values.llmRequestRouter }} -{{- if and .Values.llmRequestRouter.backendRouter.enabled $metricsEnabled $serviceMonitorEnabled }} +{{- if and (eq (include "llm-request-router.backendRouterEnabled" .) "true") $metricsEnabled $serviceMonitorEnabled }} apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: diff --git a/deploy/helm/llm-request-router/llm-request-router/templates/backend-router.yaml b/deploy/helm/llm-request-router/llm-request-router/templates/backend-router.yaml index e9e5efed1..8f838a32c 100644 --- a/deploy/helm/llm-request-router/llm-request-router/templates/backend-router.yaml +++ b/deploy/helm/llm-request-router/llm-request-router/templates/backend-router.yaml @@ -1,7 +1,7 @@ # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 -{{- if .Values.llmRequestRouter.backendRouter.enabled }} +{{- if eq (include "llm-request-router.backendRouterEnabled" .) "true" }} {{- include "llm-request-router.validateBackendRouterTls" . }} {{- include "llm-request-router.validateBackendRouterServiceAccount" . }} {{- $advertisedHostnameTemplate := include "llm-request-router.advertisedHostnameTemplate" . }} @@ -81,6 +81,9 @@ spec: - --target-namespace={{ include "llm-request-router.namespace" . }} - --target-service-name={{ include "llm-request-router.fullname" . }} - --advertised-hostname-template={{ $advertisedHostnameTemplate }} + - --advertised-grpc-port={{ .Values.llmRequestRouter.service.grpcPort }} + - --grpc-pylon-dial-addr={{ include "llm-request-router.backendRouterGrpcDialAddress" . }} + - --watch-heartbeat-ms={{ .Values.llmRequestRouter.discovery.watchHeartbeatMs }} - --grpc-port-name=grpc - --quic-port-name=quic - --tunnel-protocol=raw-quic diff --git a/deploy/helm/llm-request-router/llm-request-router/templates/deployment.yaml b/deploy/helm/llm-request-router/llm-request-router/templates/deployment.yaml index 7cde249c2..1d52f0dc9 100644 --- a/deploy/helm/llm-request-router/llm-request-router/templates/deployment.yaml +++ b/deploy/helm/llm-request-router/llm-request-router/templates/deployment.yaml @@ -13,22 +13,37 @@ # See the License for the specific language governing permissions and # limitations under the License. +{{- $workloadKind := include "llm-request-router.workloadKind" . -}} +{{- if not (has $workloadKind (list "Deployment" "StatefulSet")) -}} +{{- fail (printf "llmRequestRouter.workload.kind must be Deployment or StatefulSet, got %q" $workloadKind) -}} +{{- end }} apiVersion: apps/v1 -kind: StatefulSet +kind: {{ $workloadKind }} metadata: name: {{ include "llm-request-router.fullname" . }} namespace: {{ include "llm-request-router.namespace" . }} labels: {{- include "llm-request-router.labels" . | nindent 4 }} {{- /* -Single-replica deployments can advertise the stable service DNS because it maps -to one router pod. Multi-replica deployments need per-pod advertised hostnames -so reverse-tunnel clients connect back to the router instance that issued the -tunnel target. +Single-replica Deployments use only their local identity because Kubernetes +may publish both pod-name and dashed-IP SRV records for Deployment endpoints. +StatefulSets retain stable DNS identities. Multi-replica Deployments need the +EndpointSlice backend router so reverse-tunnel clients connect back to the +router instance that issued the tunnel target. */}} {{- $replicaCount := .Values.llmRequestRouter.replicaCount | int }} -{{- $disableDnsDiscovery := dig "discovery" "disableDnsDiscovery" false .Values.llmRequestRouter }} -{{- if and $disableDnsDiscovery (gt $replicaCount 1) }} +{{- $backendRouterEnabled := eq (include "llm-request-router.backendRouterEnabled" .) "true" }} +{{- $configuredDisableDnsDiscovery := dig "discovery" "disableDnsDiscovery" false .Values.llmRequestRouter }} +{{- $directDeployment := and (eq $workloadKind "Deployment") (not $backendRouterEnabled) }} +{{- $disableDnsDiscovery := or $configuredDisableDnsDiscovery $directDeployment }} +{{- $watchHeartbeatMs := .Values.llmRequestRouter.discovery.watchHeartbeatMs | int }} +{{- if le $watchHeartbeatMs 0 }} +{{- fail "llmRequestRouter.discovery.watchHeartbeatMs must be greater than 0" }} +{{- end }} +{{- if and (eq $workloadKind "Deployment") (gt $replicaCount 1) (not $backendRouterEnabled) }} +{{- fail "llmRequestRouter.backendRouter.enabled must be true when llmRequestRouter.workload.kind is Deployment and replicaCount is greater than 1" }} +{{- end }} +{{- if and $configuredDisableDnsDiscovery (gt $replicaCount 1) (not $backendRouterEnabled) }} {{- fail "llmRequestRouter.discovery.disableDnsDiscovery cannot be true when llmRequestRouter.replicaCount is greater than 1; multi-replica routers require DNS discovery" }} {{- end }} {{- /* @@ -37,7 +52,6 @@ renders no Certificate, so a guard in that template would never run. */}} {{- include "llm-request-router.validateTlsIdentity" . }} {{- $advertisedHostnameTemplate := include "llm-request-router.advertisedHostnameTemplate" . }} -{{- $backendRouterEnabled := dig "backendRouter" "enabled" false .Values.llmRequestRouter }} {{- if and $backendRouterEnabled (not .Values.llmRequestRouter.transport.reverseTunnelListenAddr) }} {{- fail "llmRequestRouter.backendRouter.enabled requires llmRequestRouter.transport.reverseTunnelListenAddr" }} {{- end }} @@ -54,9 +68,16 @@ renders no Certificate, so a guard in that template would never run. {{- end }} {{- end }} spec: + {{- if eq $workloadKind "Deployment" }} + strategy: + {{- toYaml .Values.llmRequestRouter.workload.deployment.strategy | nindent 4 }} + {{- else }} serviceName: {{ .Values.llmRequestRouter.service.headlessName }} + podManagementPolicy: {{ .Values.llmRequestRouter.workload.statefulSet.podManagementPolicy }} + updateStrategy: + {{- toYaml .Values.llmRequestRouter.workload.statefulSet.updateStrategy | nindent 4 }} + {{- end }} replicas: {{ $replicaCount }} - podManagementPolicy: Parallel selector: matchLabels: {{- include "llm-request-router.selectorLabels" . | nindent 6 }} diff --git a/deploy/helm/llm-request-router/llm-request-router/values.yaml b/deploy/helm/llm-request-router/llm-request-router/values.yaml index d693813fd..1f3941693 100644 --- a/deploy/helm/llm-request-router/llm-request-router/values.yaml +++ b/deploy/helm/llm-request-router/llm-request-router/values.yaml @@ -15,10 +15,24 @@ llmRequestRouter: enabled: true - # Multi-replica requires DNS discovery and stable per-pod hostnames from - # the StatefulSet and headless service. replicaCount: 3 + workload: + # Deployment matches managed NVCF. Existing installations can pin + # StatefulSet and migrate workload kinds in a controlled maintenance + # window; Kubernetes cannot mutate one workload kind into the other. + kind: Deployment + deployment: + strategy: + type: RollingUpdate + rollingUpdate: + maxSurge: 1 + maxUnavailable: 0 + statefulSet: + podManagementPolicy: Parallel + updateStrategy: + type: RollingUpdate + podDisruptionBudget: enabled: false # minAvailable and maxUnavailable are mutually exclusive; set exactly one. @@ -72,11 +86,14 @@ llmRequestRouter: headlessName: llm-request-router-headless # Routes backend registration and reverse-tunnel traffic to the Stargate pod - # named by gRPC authority or QUIC SNI. Enable this when pylons connect through - # a shared Gateway or load balancer and request-router replicaCount is greater - # than one. + # named by gRPC authority or QUIC SNI. Multi-replica Deployments enable this + # automatically. Other workload topologies can enable it when pylons connect + # through a shared Gateway or load balancer. backendRouter: - enabled: false + # null enables the backend router automatically for a multi-replica + # Deployment and disables it otherwise. Set a boolean to override this; + # false is rejected for a multi-replica Deployment. + enabled: null # The router is the single front door for worker registration and reverse # tunnels in its region, so one replica makes it a single point of failure # for every worker in that region. The router keeps no state beyond the QUIC @@ -218,6 +235,9 @@ llmRequestRouter: discovery: disableDnsDiscovery: false + # Maximum interval between unchanged WatchStargates snapshots. This is + # shared by Stargate and the EndpointSlice backend router. + watchHeartbeatMs: 5000 transport: quicConnectTimeoutMs: 2000 diff --git a/deploy/helm/llm-request-router/scripts/check-backend-router-render.sh b/deploy/helm/llm-request-router/scripts/check-backend-router-render.sh index 22236c513..366041f33 100755 --- a/deploy/helm/llm-request-router/scripts/check-backend-router-render.sh +++ b/deploy/helm/llm-request-router/scripts/check-backend-router-render.sh @@ -197,6 +197,10 @@ assert_contains "--target-service-name=llm-request-router" \ "backend router must watch the readiness-respecting request-router Service" assert_contains "--advertised-hostname-template={pod_name}.llm-request-router-headless.nvcf.svc.cluster.local" \ "backend router authority and SNI template must match Stargate" +assert_contains "--advertised-grpc-port=50071" \ + "backend router Watch snapshots must advertise the Stargate gRPC port" +assert_contains "--grpc-pylon-dial-addr=llm-router.example.invalid:443" \ + "backend router Watch snapshots must preserve the Pylon dial endpoint" assert_contains "- '*.llm-request-router-headless.nvcf.svc.cluster.local'" \ "request-router certificate must cover pod-specific backend routing hostnames" assert_contains "image: registry.example.invalid/nvcf/stargate:next" \ @@ -219,6 +223,7 @@ helm template llm-request-router "$chart_dir" \ --namespace nvcf \ --set llmRequestRouter.image.registry=registry.example.invalid \ --set llmRequestRouter.image.repository=nvcf/stargate \ + --set llmRequestRouter.replicaCount=1 \ --set llmRequestRouter.backendRouter.enabled=false \ >"$disabled" @@ -326,6 +331,7 @@ fi assert_render_fails "llmRequestRouter.certificate.dnsNames is required when certificate.enabled is true" \ --set llmRequestRouter.image.registry=registry.example.invalid \ --set llmRequestRouter.image.repository=nvcf/stargate \ + --set llmRequestRouter.replicaCount=1 \ --set llmRequestRouter.backendRouter.enabled=false \ --set llmRequestRouter.certificate.enabled=true \ --set llmRequestRouter.certificate.issuerRef.name=test-issuer @@ -371,6 +377,7 @@ assert_render_fails "llmRequestRouter.tls.certPath and llmRequestRouter.tls.keyP assert_render_fails "llmRequestRouter.tls.certPath and llmRequestRouter.tls.keyPath must use the same directory" \ --set llmRequestRouter.image.registry=registry.example.invalid \ --set llmRequestRouter.image.repository=nvcf/stargate \ + --set llmRequestRouter.replicaCount=1 \ --set llmRequestRouter.backendRouter.enabled=false \ --set llmRequestRouter.tls.secretName=stargate-quic-tls \ --set llmRequestRouter.tls.certPath=/etc/stargate/tls/tls.crt \ diff --git a/deploy/helm/llm-request-router/scripts/check-multi-replica-render.sh b/deploy/helm/llm-request-router/scripts/check-multi-replica-render.sh index 08f5d4fb9..eb63b07f1 100755 --- a/deploy/helm/llm-request-router/scripts/check-multi-replica-render.sh +++ b/deploy/helm/llm-request-router/scripts/check-multi-replica-render.sh @@ -2,7 +2,7 @@ # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 -set -euo pipefail +set -eu chart_dir="${1:-./llm-request-router}" release="${RELEASE:-llm-request-router}" @@ -25,62 +25,117 @@ render() { helm template "${release}" "${chart_dir}" \ --namespace "${namespace}" \ --values "${chart_dir}/values.yaml" \ + --set llmRequestRouter.image.repository=stargate \ "$@" \ > "${output}" } -statefulset_field() { +workload_field() { local manifest="$1" - local expression="$2" - yq -r "select(.kind == \"StatefulSet\" and .metadata.name == \"llm-request-router\") | ${expression}" "${manifest}" | head -n1 + local kind="$2" + local expression="$3" + yq -r "select(.kind == \"${kind}\" and .metadata.name == \"llm-request-router\") | ${expression}" "${manifest}" | head -n1 } -statefulset_args() { +workload_args() { local manifest="$1" - yq -r 'select(.kind == "StatefulSet" and .metadata.name == "llm-request-router") | .spec.template.spec.containers[0].args[]' "${manifest}" + local kind="$2" + yq -r "select(.kind == \"${kind}\" and .metadata.name == \"llm-request-router\") | .spec.template.spec.containers[0].args[]" "${manifest}" +} + +backend_router_args() { + local manifest="$1" + yq -r 'select(.kind == "Deployment" and .metadata.name == "llm-request-router-backend-router") | .spec.template.spec.containers[0].args[]' "${manifest}" +} + +assert_render_fails() { + local expected_error="$1" + shift + local error_file="${tmp_dir}/render-error" + if helm template "${release}" "${chart_dir}" \ + --namespace "${namespace}" \ + --values "${chart_dir}/values.yaml" \ + --set llmRequestRouter.image.repository=stargate \ + "$@" \ + > /dev/null 2> "${error_file}"; then + fail "expected render failure: ${expected_error}" + fi + grep -Fq "${expected_error}" "${error_file}" || fail "render did not return expected error: ${expected_error}" } default_manifest="${tmp_dir}/default.yaml" render "${default_manifest}" -[ "$(statefulset_field "${default_manifest}" ".kind")" = "StatefulSet" ] || fail "default render did not create StatefulSet" -[ "$(statefulset_field "${default_manifest}" ".spec.serviceName")" = "llm-request-router-headless" ] || fail "default StatefulSet serviceName is not llm-request-router-headless" -[ "$(statefulset_field "${default_manifest}" ".spec.replicas")" = "3" ] || fail "default replica count is not 3" -[ "$(statefulset_field "${default_manifest}" ".spec.podManagementPolicy")" = "Parallel" ] || fail "default StatefulSet podManagementPolicy is not Parallel" - -default_args="$(statefulset_args "${default_manifest}")" -printf '%s\n' "${default_args}" | grep -qx -- "--stargate-discovery-dns-name=llm-request-router-headless.${namespace}.svc.cluster.local" || fail "default render missing headless discovery DNS arg" -printf '%s\n' "${default_args}" | grep -qx -- "--advertised-hostname-template={pod_name}.llm-request-router-headless.${namespace}.svc.cluster.local" || fail "default render missing per-pod advertised hostname template" -printf '%s\n' "${default_args}" | grep -qx -- '--reverse-tunnel-pylon-dial-addr=$(POD_IP):50072' || fail "default render missing reverse tunnel pylon dial addr" -if printf '%s\n' "${default_args}" | grep -qx -- "--disable-dns-discovery"; then - fail "default multi-replica render must not disable DNS discovery" -fi - -invalid_error="${tmp_dir}/invalid.err" -if helm template "${release}" "${chart_dir}" \ - --namespace "${namespace}" \ - --values "${chart_dir}/values.yaml" \ +[ "$(workload_field "${default_manifest}" Deployment .kind)" = "Deployment" ] || fail "default render did not create Deployment" +[ -z "$(workload_field "${default_manifest}" StatefulSet .kind)" ] || fail "default render also created StatefulSet" +[ "$(workload_field "${default_manifest}" Deployment .spec.replicas)" = "3" ] || fail "default replica count is not 3" +[ "$(workload_field "${default_manifest}" Deployment .spec.strategy.type)" = "RollingUpdate" ] || fail "default Deployment strategy is not RollingUpdate" +[ "$(workload_field "${default_manifest}" Deployment '.spec.strategy.rollingUpdate.maxSurge')" = "1" ] || fail "default Deployment maxSurge is not 1" +[ "$(workload_field "${default_manifest}" Deployment '.spec.strategy.rollingUpdate.maxUnavailable')" = "0" ] || fail "default Deployment maxUnavailable is not 0" +[ "$(workload_field "${default_manifest}" Deployment '.spec.serviceName // ""')" = "" ] || fail "Deployment must not render StatefulSet serviceName" +[ "$(workload_field "${default_manifest}" Deployment '.spec.podManagementPolicy // ""')" = "" ] || fail "Deployment must not render StatefulSet podManagementPolicy" +[ "$(workload_field "${default_manifest}" Deployment '.spec.updateStrategy // ""')" = "" ] || fail "Deployment must not render StatefulSet updateStrategy" +default_backend_kind="$(yq -r 'select(.kind == "Deployment" and .metadata.name == "llm-request-router-backend-router") | .kind' "${default_manifest}" | head -n1)" +[ "${default_backend_kind}" = "Deployment" ] || fail "default multi-replica Deployment did not infer backend-router enablement" + +default_args="$(workload_args "${default_manifest}" Deployment)" +default_backend_args="$(backend_router_args "${default_manifest}")" +printf '%s\n' "${default_args}" | grep -qx -- "--advertised-hostname-template={pod_name}.llm-request-router-headless.${namespace}.svc.cluster.local" || fail "default Deployment missing per-pod advertised hostname template" +printf '%s\n' "${default_args}" | grep -qx -- "--grpc-pylon-dial-addr=llm-request-router-backend-router.${namespace}.svc.cluster.local:50071" || fail "default Deployment missing inferred backend-router gRPC dial address" +printf '%s\n' "${default_args}" | grep -qx -- "--watch-heartbeat-ms=5000" || fail "default Deployment missing Watch heartbeat arg" +printf '%s\n' "${default_backend_args}" | grep -qx -- "--watch-heartbeat-ms=5000" || fail "backend router missing Watch heartbeat arg" + +multi_deployment_manifest="${tmp_dir}/multi-deployment.yaml" +render "${multi_deployment_manifest}" \ --set llmRequestRouter.replicaCount=3 \ - --set llmRequestRouter.discovery.disableDnsDiscovery=true \ - > "${tmp_dir}/invalid.yaml" 2> "${invalid_error}"; then - fail "multi-replica render with disabled DNS discovery unexpectedly succeeded" -fi -grep -Fq "llmRequestRouter.discovery.disableDnsDiscovery cannot be true when llmRequestRouter.replicaCount is greater than 1; multi-replica routers require DNS discovery" "${invalid_error}" || fail "invalid render did not return the expected guard message" - -single_manifest="${tmp_dir}/single.yaml" -render "${single_manifest}" \ + --set llmRequestRouter.backendRouter.enabled=true + +multi_deployment_args="$(workload_args "${multi_deployment_manifest}" Deployment)" +printf '%s\n' "${multi_deployment_args}" | grep -qx -- "--advertised-hostname-template={pod_name}.llm-request-router-headless.${namespace}.svc.cluster.local" || fail "multi-replica Deployment missing per-pod advertised hostname template" +printf '%s\n' "${multi_deployment_args}" | grep -qx -- "--grpc-pylon-dial-addr=llm-request-router-backend-router.${namespace}.svc.cluster.local:50071" || fail "multi-replica Deployment missing backend-router gRPC dial address" + +statefulset_manifest="${tmp_dir}/statefulset.yaml" +render "${statefulset_manifest}" \ + --set llmRequestRouter.workload.kind=StatefulSet \ + --set llmRequestRouter.replicaCount=3 + +[ "$(workload_field "${statefulset_manifest}" StatefulSet .kind)" = "StatefulSet" ] || fail "explicit StatefulSet render did not create StatefulSet" +[ -z "$(workload_field "${statefulset_manifest}" Deployment .kind)" ] || fail "explicit StatefulSet render also created Deployment" +[ "$(workload_field "${statefulset_manifest}" StatefulSet .spec.serviceName)" = "llm-request-router-headless" ] || fail "StatefulSet serviceName is not llm-request-router-headless" +[ "$(workload_field "${statefulset_manifest}" StatefulSet .spec.podManagementPolicy)" = "Parallel" ] || fail "StatefulSet podManagementPolicy is not Parallel" +[ "$(workload_field "${statefulset_manifest}" StatefulSet .spec.updateStrategy.type)" = "RollingUpdate" ] || fail "StatefulSet updateStrategy is not RollingUpdate" +[ "$(workload_field "${statefulset_manifest}" StatefulSet '.spec.strategy // ""')" = "" ] || fail "StatefulSet must not render Deployment strategy" +statefulset_backend_kind="$(yq -r 'select(.kind == "Deployment" and .metadata.name == "llm-request-router-backend-router") | .kind' "${statefulset_manifest}" | head -n1)" +[ -z "${statefulset_backend_kind}" ] || fail "pinned StatefulSet unexpectedly inferred backend-router enablement" + +statefulset_args="$(workload_args "${statefulset_manifest}" StatefulSet)" +printf '%s\n' "${statefulset_args}" | grep -qx -- "--stargate-discovery-dns-name=llm-request-router-headless.${namespace}.svc.cluster.local" || fail "StatefulSet direct mode missing headless discovery DNS arg" +printf '%s\n' "${statefulset_args}" | grep -qx -- '--reverse-tunnel-pylon-dial-addr=$(POD_IP):50072' || fail "StatefulSet direct mode missing per-pod reverse tunnel address" + +assert_render_fails "llmRequestRouter.workload.kind must be Deployment or StatefulSet, got \"DaemonSet\"" \ + --set llmRequestRouter.workload.kind=DaemonSet + +assert_render_fails "llmRequestRouter.backendRouter.enabled must be true when llmRequestRouter.workload.kind is Deployment and replicaCount is greater than 1" \ + --set llmRequestRouter.workload.kind=Deployment \ + --set llmRequestRouter.replicaCount=2 \ + --set llmRequestRouter.backendRouter.enabled=false + +single_deployment_manifest="${tmp_dir}/single-deployment.yaml" +render "${single_deployment_manifest}" \ + --set llmRequestRouter.workload.kind=Deployment \ --set llmRequestRouter.replicaCount=1 \ - --set llmRequestRouter.discovery.disableDnsDiscovery=true + --set llmRequestRouter.backendRouter.enabled=false -single_args="$(statefulset_args "${single_manifest}")" -printf '%s\n' "${single_args}" | grep -qx -- "--disable-dns-discovery" || fail "single-replica self-only render missing --disable-dns-discovery" +single_deployment_args="$(workload_args "${single_deployment_manifest}" Deployment)" +printf '%s\n' "${single_deployment_args}" | grep -qx -- "--disable-dns-discovery" || fail "single-replica direct Deployment missing --disable-dns-discovery" -custom_template_manifest="${tmp_dir}/custom-template.yaml" -render "${custom_template_manifest}" \ +assert_render_fails "llmRequestRouter.discovery.disableDnsDiscovery cannot be true when llmRequestRouter.replicaCount is greater than 1; multi-replica routers require DNS discovery" \ + --set llmRequestRouter.workload.kind=StatefulSet \ --set llmRequestRouter.replicaCount=3 \ - --set-string llmRequestRouter.kubernetes.advertisedHostnameTemplate=router.example.internal + --set llmRequestRouter.backendRouter.enabled=false \ + --set llmRequestRouter.discovery.disableDnsDiscovery=true -custom_template_args="$(statefulset_args "${custom_template_manifest}")" -printf '%s\n' "${custom_template_args}" | grep -qx -- '--reverse-tunnel-pylon-dial-addr=$(POD_IP):50072' || fail "custom multi-replica advertised hostname template missing reverse tunnel pylon dial addr" +assert_render_fails "llmRequestRouter.discovery.watchHeartbeatMs must be greater than 0" \ + --set llmRequestRouter.discovery.watchHeartbeatMs=0 -echo "multi-replica render checks passed" +echo "dual workload render checks passed" diff --git a/deploy/helm/llm-request-router/scripts/check-pki-render.sh b/deploy/helm/llm-request-router/scripts/check-pki-render.sh index 579fb4551..34964ac1f 100644 --- a/deploy/helm/llm-request-router/scripts/check-pki-render.sh +++ b/deploy/helm/llm-request-router/scripts/check-pki-render.sh @@ -35,6 +35,8 @@ render_certificate_case() { --namespace nvcf \ --values ./llm-request-router/values.yaml \ --set llmRequestRouter.image.repository=stargate \ + --set llmRequestRouter.workload.kind=StatefulSet \ + --set llmRequestRouter.replicaCount=3 \ --set llmRequestRouter.certificate.enabled=true \ --set llmRequestRouter.certificate.secretName=stargate-quic-tls \ --set llmRequestRouter.certificate.issuerRef.name=nvcf-openbao-pki \ @@ -60,12 +62,12 @@ default_cert="$(yq -rN 'select(.kind == "Certificate") | .metadata.name' "${defa default_job="$(yq -rN 'select(.kind == "Job" and .metadata.name == "addons-llm-migrations") | .metadata.name' "${defaults_manifest}" | head -n1)" [ -z "${default_job}" ] || { echo "FAIL: addons-llm-migrations Job rendered with default values" >&2; exit 1; } -# StatefulSet should still render (chart's primary purpose) but with no +# The selected request-router workload should still render but with no # stargate-tls volume or volumeMount. -default_workload="$(yq -rN 'select(.kind == "StatefulSet" and .metadata.name == "llm-request-router") | .metadata.name' "${defaults_manifest}" | head -n1)" -[ "${default_workload}" = "llm-request-router" ] || { echo "FAIL: llm-request-router StatefulSet did not render at defaults" >&2; exit 1; } +default_workload="$(yq -rN 'select((.kind == "Deployment" or .kind == "StatefulSet") and .metadata.name == "llm-request-router") | .metadata.name' "${defaults_manifest}" | head -n1)" +[ "${default_workload}" = "llm-request-router" ] || { echo "FAIL: llm-request-router workload did not render at defaults" >&2; exit 1; } -default_workload_args="$(yq -rN 'select(.kind == "StatefulSet" and .metadata.name == "llm-request-router") | .spec.template.spec.containers[0].args[]' "${defaults_manifest}")" +default_workload_args="$(yq -rN 'select((.kind == "Deployment" or .kind == "StatefulSet") and .metadata.name == "llm-request-router") | .spec.template.spec.containers[0].args[]' "${defaults_manifest}")" if printf '%s\n' "${default_workload_args}" | grep -qx -- "--metrics-prefix=llm_request_router_"; then echo "FAIL: --metrics-prefix is not supported by the pinned stargate 0.3.0 image" >&2 exit 1 @@ -75,10 +77,10 @@ if printf '%s\n' "${default_workload_args}" | grep -qx -- "--otel-service-name=l exit 1 fi -default_tls_mount="$(yq -rN 'select(.kind == "StatefulSet" and .metadata.name == "llm-request-router") | .spec.template.spec.containers[0].volumeMounts[]? | select(.name == "stargate-tls") | .name' "${defaults_manifest}" | head -n1)" +default_tls_mount="$(yq -rN 'select((.kind == "Deployment" or .kind == "StatefulSet") and .metadata.name == "llm-request-router") | .spec.template.spec.containers[0].volumeMounts[]? | select(.name == "stargate-tls") | .name' "${defaults_manifest}" | head -n1)" [ -z "${default_tls_mount}" ] || { echo "FAIL: stargate-tls volumeMount rendered with default values" >&2; exit 1; } -default_tls_volume="$(yq -rN 'select(.kind == "StatefulSet" and .metadata.name == "llm-request-router") | .spec.template.spec.volumes[]? | select(.name == "stargate-tls") | .name' "${defaults_manifest}" | head -n1)" +default_tls_volume="$(yq -rN 'select((.kind == "Deployment" or .kind == "StatefulSet") and .metadata.name == "llm-request-router") | .spec.template.spec.volumes[]? | select(.name == "stargate-tls") | .name' "${defaults_manifest}" | head -n1)" [ -z "${default_tls_volume}" ] || { echo "FAIL: stargate-tls volume rendered with default values" >&2; exit 1; } # Pass 2: PKI, certificate, and TLS are fully enabled. Assert that every @@ -114,7 +116,7 @@ cert_dns_name="$(yq -rN 'select(.kind == "Certificate" and .metadata.name == "st [ "${cert_issuer_name}" = "nvcf-openbao-pki" ] [ "${cert_dns_name}" = "*.stargate.localhost" ] -workload_args="$(yq -rN 'select(.kind == "StatefulSet" and .metadata.name == "llm-request-router") | .spec.template.spec.containers[0].args[]' "${manifest}")" +workload_args="$(yq -rN 'select((.kind == "Deployment" or .kind == "StatefulSet") and .metadata.name == "llm-request-router") | .spec.template.spec.containers[0].args[]' "${manifest}")" printf '%s\n' "${workload_args}" | grep -qx -- "--tls-cert-path=/etc/stargate/tls/tls.crt" printf '%s\n' "${workload_args}" | grep -qx -- "--tls-key-path=/etc/stargate/tls/tls.key" if printf '%s\n' "${workload_args}" | grep -qx -- "--quic-insecure"; then @@ -122,8 +124,8 @@ if printf '%s\n' "${workload_args}" | grep -qx -- "--quic-insecure"; then exit 1 fi -tls_mount_name="$(yq -rN 'select(.kind == "StatefulSet" and .metadata.name == "llm-request-router") | .spec.template.spec.containers[0].volumeMounts[] | select(.name == "stargate-tls" and .mountPath == "/etc/stargate/tls" and .readOnly == true) | .name' "${manifest}")" -tls_volume_name="$(yq -rN 'select(.kind == "StatefulSet" and .metadata.name == "llm-request-router") | .spec.template.spec.volumes[] | select(.name == "stargate-tls" and .secret.secretName == "stargate-quic-tls") | .name' "${manifest}")" +tls_mount_name="$(yq -rN 'select((.kind == "Deployment" or .kind == "StatefulSet") and .metadata.name == "llm-request-router") | .spec.template.spec.containers[0].volumeMounts[] | select(.name == "stargate-tls" and .mountPath == "/etc/stargate/tls" and .readOnly == true) | .name' "${manifest}")" +tls_volume_name="$(yq -rN 'select((.kind == "Deployment" or .kind == "StatefulSet") and .metadata.name == "llm-request-router") | .spec.template.spec.volumes[] | select(.name == "stargate-tls" and .secret.secretName == "stargate-quic-tls") | .name' "${manifest}")" [ "${tls_mount_name}" = "stargate-tls" ] [ "${tls_volume_name}" = "stargate-tls" ] @@ -180,7 +182,7 @@ helm template llm-request-router ./llm-request-router \ single_replica_dns_name="$(yq -rN 'select(.kind == "Certificate" and .metadata.name == "stargate-quic-tls") | .spec.dnsNames[0]' "${single_replica_manifest}")" [ "${single_replica_dns_name}" = "llm-request-router.nvcf.svc.cluster.local" ] || fail "single-replica Certificate SAN did not render the advertised hostname" -single_replica_args="$(yq -rN 'select(.kind == "StatefulSet" and .metadata.name == "llm-request-router") | .spec.template.spec.containers[0].args[]' "${single_replica_manifest}")" +single_replica_args="$(yq -rN 'select((.kind == "Deployment" or .kind == "StatefulSet") and .metadata.name == "llm-request-router") | .spec.template.spec.containers[0].args[]' "${single_replica_manifest}")" printf '%s\n' "${single_replica_args}" | grep -qx -- "--advertised-hostname-template=llm-request-router.nvcf.svc.cluster.local" || fail "single-replica render missing the advertised hostname template" # Certificate validation resolves both supported placeholders. The pod name @@ -197,6 +199,7 @@ if helm template llm-request-router ./llm-request-router \ --namespace nvcf \ --values ./llm-request-router/values.yaml \ --set llmRequestRouter.image.repository=stargate \ + --set llmRequestRouter.workload.kind=StatefulSet \ --set llmRequestRouter.certificate.enabled=true \ --set llmRequestRouter.certificate.issuerRef.name=nvcf-openbao-pki \ > /dev/null 2> "${empty_dns_error}"; then @@ -329,13 +332,13 @@ existing_secret_cert="$(yq -rN 'select(.kind == "Certificate") | .metadata.name' existing_secret_job="$(yq -rN 'select(.kind == "Job" and .metadata.name == "addons-llm-migrations") | .metadata.name' "${existing_secret_manifest}" | head -n1)" [ -z "${existing_secret_job}" ] || fail "existing-Secret mode rendered the OpenBao provisioning hook" -existing_secret_volume="$(yq -rN 'select(.kind == "StatefulSet" and .metadata.name == "llm-request-router") | .spec.template.spec.volumes[] | select(.name == "stargate-tls" and .secret.secretName == "operator-quic-tls") | .name' "${existing_secret_manifest}")" +existing_secret_volume="$(yq -rN 'select((.kind == "Deployment" or .kind == "StatefulSet") and .metadata.name == "llm-request-router") | .spec.template.spec.volumes[] | select(.name == "stargate-tls" and .secret.secretName == "operator-quic-tls") | .name' "${existing_secret_manifest}")" [ "${existing_secret_volume}" = "stargate-tls" ] || fail "existing-Secret mode did not mount the pre-created Secret" -existing_secret_mount="$(yq -rN 'select(.kind == "StatefulSet" and .metadata.name == "llm-request-router") | .spec.template.spec.containers[0].volumeMounts[] | select(.name == "stargate-tls" and .mountPath == "/etc/stargate/tls" and .readOnly == true) | .name' "${existing_secret_manifest}")" +existing_secret_mount="$(yq -rN 'select((.kind == "Deployment" or .kind == "StatefulSet") and .metadata.name == "llm-request-router") | .spec.template.spec.containers[0].volumeMounts[] | select(.name == "stargate-tls" and .mountPath == "/etc/stargate/tls" and .readOnly == true) | .name' "${existing_secret_manifest}")" [ "${existing_secret_mount}" = "stargate-tls" ] || fail "existing-Secret mode did not mount stargate-tls read-only" -existing_secret_args="$(yq -rN 'select(.kind == "StatefulSet" and .metadata.name == "llm-request-router") | .spec.template.spec.containers[0].args[]' "${existing_secret_manifest}")" +existing_secret_args="$(yq -rN 'select((.kind == "Deployment" or .kind == "StatefulSet") and .metadata.name == "llm-request-router") | .spec.template.spec.containers[0].args[]' "${existing_secret_manifest}")" printf '%s\n' "${existing_secret_args}" | grep -qx -- "--tls-cert-path=/etc/stargate/tls/tls.crt" || fail "existing-Secret mode did not pass the certificate path" printf '%s\n' "${existing_secret_args}" | grep -qx -- "--tls-key-path=/etc/stargate/tls/tls.key" || fail "existing-Secret mode did not pass the private key path" if printf '%s\n' "${existing_secret_args}" | grep -qx -- "--quic-insecure"; then diff --git a/deploy/stacks/self-managed/environments/base.yaml b/deploy/stacks/self-managed/environments/base.yaml index 5558a7ce3..6b6b65401 100644 --- a/deploy/stacks/self-managed/environments/base.yaml +++ b/deploy/stacks/self-managed/environments/base.yaml @@ -308,6 +308,11 @@ addons: replicaCount: 3 # Set only for source-tree validation. Empty uses the released OCI chart. chartPath: "" + workload: + # Deployment matches managed NVCF. Existing installations can pin + # StatefulSet and migrate workload kinds in a controlled maintenance + # window; Kubernetes cannot mutate one workload kind into the other. + kind: Deployment # Authority and SNI aware backend router. A worker holds one registration # stream and one reverse QUIC tunnel per router replica, and each must diff --git a/deploy/stacks/self-managed/global.yaml.gotmpl b/deploy/stacks/self-managed/global.yaml.gotmpl index e54fe1a0e..29b22f69b 100644 --- a/deploy/stacks/self-managed/global.yaml.gotmpl +++ b/deploy/stacks/self-managed/global.yaml.gotmpl @@ -876,6 +876,8 @@ llmRequestRouter: enabled: {{ dig "addons" "llm" "enabled" false .Values }} fullnameOverride: llm-request-router replicaCount: {{ dig "addons" "llm" "requestRouter" "replicaCount" 3 .Values }} + workload: + kind: {{ dig "addons" "llm" "requestRouter" "workload" "kind" "Deployment" .Values | quote }} service: grpcPort: {{ $llmRequestRouterGrpcPort }} {{- if .Values.global.imagePullSecrets }} diff --git a/deploy/stacks/self-managed/tests/check-llm-pki-issuer.sh b/deploy/stacks/self-managed/tests/check-llm-pki-issuer.sh index 40f280d2b..e56e05364 100755 --- a/deploy/stacks/self-managed/tests/check-llm-pki-issuer.sh +++ b/deploy/stacks/self-managed/tests/check-llm-pki-issuer.sh @@ -328,7 +328,7 @@ expect_existing_secret_router() { fi local mounted_secret - mounted_secret="$(yq -rN 'select(.kind == "StatefulSet") | .spec.template.spec.volumes[] | select(.name == "stargate-tls") | .secret.secretName' "$manifests_file" | head -1)" + mounted_secret="$(yq -rN 'select((.kind == "Deployment" or .kind == "StatefulSet") and .metadata.name == "llm-request-router") | .spec.template.spec.volumes[] | select(.name == "stargate-tls") | .secret.secretName' "$manifests_file" | head -1)" test "$mounted_secret" = "$secret_name" || fail "$case_name mounted secret $mounted_secret, expected $secret_name" diff --git a/deploy/stacks/self-managed/tests/llm-router-local-chart.sh b/deploy/stacks/self-managed/tests/llm-router-local-chart.sh index 7cd203010..d10557764 100755 --- a/deploy/stacks/self-managed/tests/llm-router-local-chart.sh +++ b/deploy/stacks/self-managed/tests/llm-router-local-chart.sh @@ -5,6 +5,7 @@ stack_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" chart_path="../../../helm/llm-request-router/llm-request-router" work_dir="$(mktemp -d)" values_file="$work_dir/llm-request-router-values.yaml" +stateful_values_file="$work_dir/llm-request-router-stateful-values.yaml" trap 'rm -rf "$work_dir"' EXIT result="$(cd "$stack_dir" && HELMFILE_ENV=base helmfile \ @@ -44,4 +45,29 @@ test "$backend_repository" = "$main_repository" || { exit 1 } +default_workload_kind="$(yq -r '.llmRequestRouter.workload.kind' "$values_file")" +test "$default_workload_kind" = "Deployment" || { + echo "llm-router-local-chart: expected default workload kind Deployment, got ${default_workload_kind:-missing}" >&2 + exit 1 +} + +(cd "$stack_dir" && HELMFILE_ENV=base helmfile \ + --file helmfile.d/02-core.yaml.gotmpl \ + --environment default \ + --state-values-set addons.llm.enabled=true \ + --state-values-set-string "addons.llm.requestRouter.chartPath=$chart_path" \ + --state-values-set-string addons.llm.requestRouter.workload.kind=StatefulSet \ + --state-values-set ingress.gatewayApi.gateways.shared.name=shared-gw \ + --state-values-set ingress.gatewayApi.gateways.shared.namespace=envoy-gateway-system \ + --state-values-set ingress.gatewayApi.gateways.grpc.name=grpc-gw \ + --state-values-set ingress.gatewayApi.gateways.grpc.namespace=envoy-gateway-system \ + --selector name=llm-request-router \ + write-values --output-file-template "$stateful_values_file" >/dev/null) + +stateful_workload_kind="$(yq -r '.llmRequestRouter.workload.kind' "$stateful_values_file")" +test "$stateful_workload_kind" = "StatefulSet" || { + echo "llm-router-local-chart: expected explicit workload kind StatefulSet, got ${stateful_workload_kind:-missing}" >&2 + exit 1 +} + echo "llm-router-local-chart: all checks passed" diff --git a/docs/user/llm-request-router-load-balancing.md b/docs/user/llm-request-router-load-balancing.md index 2b568affc..fce795550 100644 --- a/docs/user/llm-request-router-load-balancing.md +++ b/docs/user/llm-request-router-load-balancing.md @@ -43,9 +43,18 @@ is set, Stargate uses its built-in `power-of-two` default and accepts a routing-method override when it is in the allowlist of built-in algorithms. Stargate reads and validates the file only during process startup. The -StatefulSet does not include a load-balancer ConfigMap checksum in its pod -template. After a ConfigMap-only update, restart the StatefulSet so every -replica loads the same configuration. +request-router workload does not include a load-balancer ConfigMap checksum in +its pod template. New installations default to a Deployment; existing +installations can pin a StatefulSet. After a ConfigMap-only update, restart the +selected workload so every replica loads the same configuration. + +Existing StatefulSet installations must set +`addons.llm.requestRouter.workload.kind=StatefulSet` before upgrading. Changing +the workload kind is a controlled migration, not an in-place Kubernetes kind +mutation. A plain Helm upgrade across workload kinds can temporarily run both +the Deployment and StatefulSet; use the chart migration procedure to remove or +rename the old workload and verify that only the selected kind owns the router +Pods before scaling it. ## Distinguish router algorithms from nvcf-cli routing methods @@ -149,9 +158,9 @@ helm template llm-request-router \ --values ``` -Confirm that the rendered StatefulSet has the expected -`--lb-config-path` argument and that inline JSON creates one ConfigMap with the -`lb-config.json` key. +Confirm that the rendered request-router workload (a Deployment by default) has +the expected `--lb-config-path` argument and that inline JSON creates one +ConfigMap with the `lb-config.json` key. If the LLM route accepts untrusted traffic, inspect the rendered or live HTTPRoute and confirm that the trusted-header filter is present: @@ -172,18 +181,24 @@ For an inline configuration, inspect the live file and start argument: ```bash kubectl get configmap -n nvcf llm-request-router-lb \ -o jsonpath='{.data.lb-config\.json}' -kubectl get statefulset -n nvcf llm-request-router \ +kubectl get deployment -n nvcf llm-request-router \ -o jsonpath='{.spec.template.spec.containers[0].args}' ``` +Replace `deployment` with `statefulset` in these commands when +`addons.llm.requestRouter.workload.kind` is pinned to `StatefulSet`. + Restart after a ConfigMap-only change, then wait for all replicas: ```bash -kubectl rollout restart statefulset/llm-request-router -n nvcf -kubectl rollout status statefulset/llm-request-router -n nvcf +kubectl rollout restart deployment/llm-request-router -n nvcf +kubectl rollout status deployment/llm-request-router -n nvcf kubectl get pods -n nvcf -l app.kubernetes.io/name=llm-request-router ``` +Use `statefulset/llm-request-router` instead when the StatefulSet workload is +explicitly selected. + Confirm every listed pod was recreated after the ConfigMap update. For each pod, check for the `load balancer config loaded` startup log and compare its reported default and model-override count: @@ -230,15 +245,18 @@ metric names, labels, and scrape configuration. | HTTP `400` before backend selection | Compare the function `routingMethod` with the configured algorithm and `request_algorithms`. Check required and numeric gateway headers. | | HTTP `400` for affinity-aware routing | Confirm the gateway generated a nonblank affinity key when `require_cache_affinity_key` is enabled. | | HTTP `503` with no eligible candidates | Confirm pylons are registered and publish the capacity, queue, and optional KV-cache statistics required by the algorithm. | -| New ConfigMap value has no effect | Confirm the pod creation time. Restart the StatefulSet because Stargate does not reload the file. | +| New ConfigMap value has no effect | Confirm the pod creation time. Restart the selected Deployment or StatefulSet because Stargate does not reload the file. | | Unexpected fallback or retries | Compare routing selection, proxy attempt, retry, and retry-exhaustion metrics. Check pylon retry reasons. | | Affinity differs between replicas | Compare the ConfigMap, pod creation times, startup summaries, seed values, and registered candidate set. | Use these logs together: ```bash -kubectl logs -n nvcf statefulset/llm-request-router \ +kubectl logs -n nvcf deployment/llm-request-router \ --all-pods=true --tail=100 kubectl logs -n nvcf deploy/llm-api-gateway --tail=100 kubectl logs -n nvcf-backend -c llm-worker --tail=100 ``` + +Use `statefulset/llm-request-router` for the first command when the StatefulSet +workload is explicitly selected. diff --git a/src/libraries/rust/stargate/crates/stargate-k8s-router/src/endpoints.rs b/src/libraries/rust/stargate/crates/stargate-k8s-router/src/endpoints.rs index 453a65e43..939be05eb 100644 --- a/src/libraries/rust/stargate/crates/stargate-k8s-router/src/endpoints.rs +++ b/src/libraries/rust/stargate/crates/stargate-k8s-router/src/endpoints.rs @@ -72,7 +72,7 @@ impl TargetSnapshot { self.ready_targets().map_or(0, <[PodTarget]>::len) } - fn ready_targets(&self) -> Option<&[PodTarget]> { + pub fn ready_targets(&self) -> Option<&[PodTarget]> { self.ready.as_deref() } } @@ -187,7 +187,6 @@ fn endpoint_pod_name(endpoint: &Endpoint) -> Option<&str> { .as_ref() .filter(|target| target.kind.as_deref().is_none_or(|kind| kind == "Pod")) .and_then(|target| target.name.as_deref()) - .or(endpoint.hostname.as_deref()) } fn socket_addr(address: &str, port: u16) -> String { @@ -339,6 +338,28 @@ mod tests { ); } + #[test] + fn snapshot_normalizes_duplicate_pod_targets_across_slices() { + let first = slice( + "slice-a", + vec![endpoint("request-router-abc", "10.0.0.10", None)], + ); + let replacement = slice( + "slice-b", + vec![endpoint("request-router-abc", "10.0.0.11", None)], + ); + + let snapshot = snapshot_from_slices([&first, &replacement], &config()); + + assert_eq!(snapshot.ready_count(), 1); + assert_eq!( + snapshot + .target_for_pod("request-router-abc") + .map(|target| target.grpc_addr), + Some("10.0.0.11:50071".to_string()) + ); + } + #[test] fn snapshot_includes_ready_pod_targets() { let slice = slice( @@ -359,6 +380,40 @@ mod tests { ); } + #[test] + fn snapshot_uses_target_ref_name_instead_of_endpoint_hostname_alias() { + let mut pod = endpoint("request-router-7d9c6f64f5-mz2qk", "10.0.0.10", None); + pod.hostname = Some("10-0-0-10".to_string()); + let slice = slice("slice-a", vec![pod]); + + let snapshot = snapshot_from_slices([&slice], &config()); + + assert_eq!(snapshot.ready_count(), 1); + assert!( + snapshot + .target_for_pod("request-router-7d9c6f64f5-mz2qk") + .is_some() + ); + assert!(snapshot.target_for_pod("10-0-0-10").is_none()); + } + + #[test] + fn snapshot_ignores_hostname_alias_without_pod_target_ref() { + let slice = slice( + "slice-a", + vec![Endpoint { + addresses: vec!["10.0.0.10".to_string()], + hostname: Some("10-0-0-10".to_string()), + ..Endpoint::default() + }], + ); + + let snapshot = snapshot_from_slices([&slice], &config()); + + assert_eq!(snapshot.ready_count(), 0); + assert!(snapshot.target_for_pod("10-0-0-10").is_none()); + } + #[test] fn snapshot_treats_missing_conditions_with_kubernetes_defaults() { let slice = slice("slice-a", vec![endpoint("stargate-0", "10.0.0.10", None)]); diff --git a/src/libraries/rust/stargate/crates/stargate-k8s-router/src/grpc.rs b/src/libraries/rust/stargate/crates/stargate-k8s-router/src/grpc.rs index 6d48929ea..6bc63243c 100644 --- a/src/libraries/rust/stargate/crates/stargate-k8s-router/src/grpc.rs +++ b/src/libraries/rust/stargate/crates/stargate-k8s-router/src/grpc.rs @@ -14,18 +14,17 @@ // limitations under the License. use std::pin::Pin; -use std::sync::Arc; -use std::sync::atomic::{AtomicUsize, Ordering}; use std::time::Duration; use futures::Stream; -use stargate_forwarding::{HostnameMatcher, forward_stream_messages}; +use stargate_forwarding::{HostnameMatcher, forward_stream_messages, render_hostname}; use stargate_proto::pb::stargate_control_plane_client::StargateControlPlaneClient; use stargate_proto::pb::stargate_control_plane_server::{ StargateControlPlane, StargateControlPlaneServer, }; use stargate_proto::pb::{ - InferenceServerAck, InferenceServerRegistration, WatchStargatesRequest, WatchStargatesResponse, + InferenceServerAck, InferenceServerRegistration, StargateInfo, WatchStargatesRequest, + WatchStargatesResponse, }; use tokio::net::TcpListener; use tokio::sync::watch; @@ -60,16 +59,23 @@ impl From<&PodTarget> for GrpcTarget { #[derive(Clone, Debug)] pub struct GrpcRouterConfig { pub advertised_hostname_template: String, + pub advertised_grpc_port: u16, + pub grpc_pylon_dial_addr: String, pub target_namespace: String, pub connect_timeout: Duration, + pub watch_heartbeat_interval: Duration, } #[derive(Clone)] pub struct RouterControlPlane { connect_timeout: Duration, + advertised_hostname_template: String, + advertised_grpc_port: u16, + grpc_pylon_dial_addr: String, + target_namespace: String, + watch_heartbeat_interval: Duration, hostname_matcher: Option, targets: watch::Receiver, - round_robin: Arc, } impl RouterControlPlane { @@ -80,36 +86,65 @@ impl RouterControlPlane { ); Self { connect_timeout: config.connect_timeout, + advertised_hostname_template: config.advertised_hostname_template, + advertised_grpc_port: config.advertised_grpc_port, + grpc_pylon_dial_addr: config.grpc_pylon_dial_addr, + target_namespace: config.target_namespace, + watch_heartbeat_interval: config.watch_heartbeat_interval, hostname_matcher, targets, - round_robin: Arc::new(AtomicUsize::new(0)), } } - fn target_for_watch<'a, T>( - &self, - request: &Request, - snapshot: &'a TargetSnapshot, - ) -> Result<&'a PodTarget, Status> { - if let Some(pod_name) = - request - .extensions() - .get::() - .and_then(|authority| { - self.hostname_matcher - .as_ref()? - .extract_pod(authority.host()) - }) - { - return snapshot.target_for_pod_ref(pod_name).ok_or_else(|| { - Status::unavailable(format!("target stargate {pod_name} is not ready")) - }); - } - - let offset = self.round_robin.fetch_add(1, Ordering::Relaxed); - snapshot - .first_ready_ref(offset) - .ok_or_else(|| Status::unavailable("no ready stargate targets")) + fn watch_stargates_stream(&self) -> WatchStargatesStream { + let mut targets = self.targets.clone(); + let advertised_hostname_template = self.advertised_hostname_template.clone(); + let advertised_grpc_port = self.advertised_grpc_port; + let grpc_pylon_dial_addr = self.grpc_pylon_dial_addr.clone(); + let target_namespace = self.target_namespace.clone(); + let watch_heartbeat_interval = self.watch_heartbeat_interval; + Box::pin(async_stream::try_stream! { + let mut heartbeat = tokio::time::interval(watch_heartbeat_interval); + heartbeat.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip); + let mut last_snapshot = None; + loop { + let snapshot = targets.borrow_and_update().clone(); + if snapshot.is_initialized() { + last_snapshot = Some(snapshot.clone()); + heartbeat.reset(); + yield watch_response_from_snapshot( + &snapshot, + &advertised_hostname_template, + advertised_grpc_port, + &grpc_pylon_dial_addr, + &target_namespace, + ); + } + loop { + tokio::select! { + changed = targets.changed() => { + if changed.is_err() { + return; + } + break; + } + _ = heartbeat.tick(), if last_snapshot.is_some() => { + let snapshot = last_snapshot + .as_ref() + .expect("heartbeat branch requires an initialized snapshot"); + heartbeat.reset(); + yield watch_response_from_snapshot( + snapshot, + &advertised_hostname_template, + advertised_grpc_port, + &grpc_pylon_dial_addr, + &target_namespace, + ); + } + } + } + } + }) } fn target_for_registration<'a, T>( @@ -163,23 +198,9 @@ impl StargateControlPlane for RouterControlPlane { async fn watch_stargates( &self, - request: Request, + _request: Request, ) -> Result, Status> { - let target = { - let snapshot = self.targets.borrow(); - GrpcTarget::from(self.target_for_watch(&request, &snapshot)?) - }; - info!( - target_pod = %target.pod_name, - target_addr = %target.grpc_addr, - "forwarding WatchStargates to stargate target" - ); - - let mut peer_client = self.connect_target_addr(&target.grpc_addr).await?; - Ok(peer_client - .watch_stargates(request) - .await? - .map(|stream| Box::pin(stream) as WatchStargatesStream)) + Ok(Response::new(self.watch_stargates_stream())) } async fn register_inference_server( @@ -233,6 +254,37 @@ impl StargateControlPlane for RouterControlPlane { } } +fn watch_response_from_snapshot( + snapshot: &TargetSnapshot, + advertised_hostname_template: &str, + advertised_grpc_port: u16, + grpc_pylon_dial_addr: &str, + target_namespace: &str, +) -> WatchStargatesResponse { + let stargates = snapshot + .ready_targets() + .unwrap_or_default() + .iter() + .map(|target| { + let hostname = render_hostname( + advertised_hostname_template, + &target.pod_name, + target_namespace, + ); + StargateInfo { + stargate_id: target.pod_name.clone(), + advertise_addr: format!("{hostname}:{advertised_grpc_port}"), + http_advertise_addr: String::new(), + grpc_pylon_dial_addr: grpc_pylon_dial_addr.to_string(), + } + }) + .collect(); + WatchStargatesResponse { + stargates, + watch_stargate_urls: Vec::new(), + } +} + pub async fn serve_grpc_router( listener: TcpListener, config: GrpcRouterConfig, @@ -260,7 +312,8 @@ pub async fn serve_grpc_router( mod tests { use std::hint::black_box; use std::net::SocketAddr; - use std::sync::Mutex; + use std::sync::atomic::{AtomicUsize, Ordering}; + use std::sync::{Arc, Mutex}; use std::time::Instant; use crate::perf_tests::assert_twenty_percent_faster; @@ -446,6 +499,13 @@ mod tests { config: GrpcRouterConfig, ) -> RunningServer { let (_tx, rx) = watch::channel(snapshot); + start_router_with_receiver(rx, config).await + } + + async fn start_router_with_receiver( + rx: watch::Receiver, + config: GrpcRouterConfig, + ) -> RunningServer { let listener = TcpListener::bind("127.0.0.1:0").await.expect("bind router"); let addr = listener.local_addr().expect("router local addr"); let shutdown = CancellationToken::new(); @@ -471,8 +531,11 @@ mod tests { fn router_config() -> GrpcRouterConfig { GrpcRouterConfig { advertised_hostname_template: "{pod_name}.stargate.external".to_string(), + advertised_grpc_port: 50071, + grpc_pylon_dial_addr: "https://stargate-router.external:443".to_string(), target_namespace: String::new(), connect_timeout: Duration::from_secs(2), + watch_heartbeat_interval: Duration::from_secs(5), } } @@ -546,7 +609,7 @@ mod tests { } #[tokio::test] - async fn watch_stargates_with_target_authority_routes_to_that_target() { + async fn watch_stargates_builds_one_canonical_identity_per_ready_target() { let recorder_a = Recorder::default(); let recorder_b = Recorder::default(); let fake_a = start_fake_stargate("stargate-0", recorder_a.clone()).await; @@ -557,50 +620,90 @@ mod tests { ])) .await; - let mut client = router.client("stargate-1.stargate.external"); + let mut client = router.client("stargate.stargate-local.svc.cluster.local"); let response = watch_once(&mut client).await; - assert_eq!(response.metadata().get("x-upstream").unwrap(), "watch"); let first = first_message(response).await; - assert_eq!(first.stargates[0].stargate_id, "stargate-1"); + assert_eq!( + first + .stargates + .iter() + .map(|stargate| ( + stargate.stargate_id.as_str(), + stargate.advertise_addr.as_str() + )) + .collect::>(), + vec![ + ("stargate-0", "stargate-0.stargate.external:50071"), + ("stargate-1", "stargate-1.stargate.external:50071"), + ] + ); + assert!(first.stargates.iter().all(|stargate| { + stargate.grpc_pylon_dial_addr == "https://stargate-router.external:443" + })); assert_eq!(recorder_a.watch_hits.load(Ordering::Relaxed), 0); - assert_eq!(recorder_b.watch_hits.load(Ordering::Relaxed), 1); + assert_eq!(recorder_b.watch_hits.load(Ordering::Relaxed), 0); } #[tokio::test] - async fn watch_stargates_without_target_authority_uses_ready_round_robin_target() { + async fn watch_stargates_stream_replaces_removed_targets() { let recorder_a = Recorder::default(); + let recorder_b = Recorder::default(); let fake_a = start_fake_stargate("stargate-0", recorder_a.clone()).await; - let router = start_router(snapshot(&[("stargate-0", fake_a.addr)])).await; + let fake_b = start_fake_stargate("stargate-1", recorder_b.clone()).await; + let (targets_tx, targets_rx) = watch::channel(snapshot(&[ + ("stargate-0", fake_a.addr), + ("stargate-1", fake_b.addr), + ])); + let router = start_router_with_receiver(targets_rx, router_config()).await; let mut client = router.client("stargate.stargate-local.svc.cluster.local"); let response = watch_once(&mut client).await; - let first = first_message(response).await; + let mut stream = response.into_inner(); + let first = stream + .message() + .await + .expect("first snapshot read should succeed") + .expect("first snapshot should be published"); + assert_eq!(first.stargates.len(), 2); + + targets_tx + .send(snapshot(&[("stargate-1", fake_b.addr)])) + .expect("updated snapshot should publish"); + let replacement = stream + .message() + .await + .expect("replacement snapshot read should succeed") + .expect("replacement snapshot should be published"); - assert_eq!(first.stargates[0].stargate_id, "stargate-0"); - assert_eq!(recorder_a.watch_hits.load(Ordering::Relaxed), 1); + assert_eq!(replacement.stargates.len(), 1); + assert_eq!(replacement.stargates[0].stargate_id, "stargate-1"); + assert_eq!(recorder_a.watch_hits.load(Ordering::Relaxed), 0); + assert_eq!(recorder_b.watch_hits.load(Ordering::Relaxed), 0); } #[tokio::test] - async fn watch_stargates_without_target_authority_round_robins_ready_targets() { - let recorder_a = Recorder::default(); - let recorder_b = Recorder::default(); - let fake_a = start_fake_stargate("stargate-0", recorder_a.clone()).await; - let fake_b = start_fake_stargate("stargate-1", recorder_b.clone()).await; - let router = start_router(snapshot(&[ - ("stargate-0", fake_a.addr), - ("stargate-1", fake_b.addr), - ])) - .await; + async fn watch_stargates_heartbeats_an_unchanged_snapshot() { + let (targets_tx, targets_rx) = watch::channel(synthetic_snapshot(1)); + let mut config = router_config(); + config.watch_heartbeat_interval = Duration::from_millis(25); + let router = start_router_with_receiver(targets_rx, config).await; let mut client = router.client("stargate.stargate-local.svc.cluster.local"); - for _ in 0..2 { - let response = watch_once(&mut client).await; - first_message(response).await; - } + let mut stream = watch_once(&mut client).await.into_inner(); + let first = stream + .message() + .await + .expect("initial snapshot read should succeed") + .expect("initial snapshot should be published"); + let heartbeat = tokio::time::timeout(Duration::from_millis(250), stream.message()) + .await + .expect("unchanged snapshot heartbeat should arrive") + .expect("heartbeat read should succeed") + .expect("heartbeat should contain a snapshot"); - assert_eq!(recorder_a.watch_hits.load(Ordering::Relaxed), 1); - assert_eq!(recorder_b.watch_hits.load(Ordering::Relaxed), 1); + assert_eq!(heartbeat, first); + drop(targets_tx); } #[tokio::test] @@ -612,8 +715,11 @@ mod tests { GrpcRouterConfig { advertised_hostname_template: "{pod_name}.{namespace}.stargate.external" .to_string(), + advertised_grpc_port: 50071, + grpc_pylon_dial_addr: "https://stargate-router.external:443".to_string(), target_namespace: "prod".to_string(), connect_timeout: Duration::from_secs(2), + watch_heartbeat_interval: Duration::from_secs(5), }, ) .await; @@ -698,15 +804,16 @@ mod tests { } #[tokio::test] - async fn watch_stargates_returns_unavailable_for_unready_target_pod() { + async fn watch_stargates_publishes_initialized_empty_snapshot() { let router = start_router(TargetSnapshot::initialized([])).await; let mut client = router.client("stargate-9.stargate.external"); - let error = client + let response = client .watch_stargates(WatchStargatesRequest {}) .await - .expect_err("missing target should be unavailable"); + .expect("initialized empty snapshot should be routable"); + let snapshot = first_message(response).await; - assert_eq!(error.code(), tonic::Code::Unavailable); + assert!(snapshot.stargates.is_empty()); } } diff --git a/src/libraries/rust/stargate/crates/stargate-k8s-router/src/main.rs b/src/libraries/rust/stargate/crates/stargate-k8s-router/src/main.rs index 0902367a5..d13dfe2bf 100644 --- a/src/libraries/rust/stargate/crates/stargate-k8s-router/src/main.rs +++ b/src/libraries/rust/stargate/crates/stargate-k8s-router/src/main.rs @@ -38,6 +38,7 @@ use tracing::{debug, error, info}; use tracing_subscriber::EnvFilter; const DEFAULT_CONNECT_TIMEOUT_MS: u64 = 5_000; +const DEFAULT_WATCH_HEARTBEAT_MS: u64 = 5_000; const DEFAULT_RELAY_MAX_IDLE_TIMEOUT_MS: u64 = 300_000; const DEFAULT_RELAY_KEEP_ALIVE_MS: u64 = 10_000; @@ -68,12 +69,20 @@ struct Args { value_name = "TEMPLATE" )] advertised_hostname_template: String, + /// Endpoint Pylon dials for every Stargate identity advertised by WatchStargates. + #[arg(long, value_name = "URI")] + grpc_pylon_dial_addr: String, + #[arg(long, default_value_t = 50071, value_name = "PORT")] + advertised_grpc_port: u16, #[arg(long, default_value = "grpc", value_name = "NAME")] grpc_port_name: String, #[arg(long, default_value = "quic", value_name = "NAME")] quic_port_name: String, #[arg(long, default_value_t = DEFAULT_CONNECT_TIMEOUT_MS, value_name = "MS")] connect_timeout_ms: u64, + /// Maximum interval between unchanged WatchStargates snapshots. + #[arg(long, default_value_t = DEFAULT_WATCH_HEARTBEAT_MS, value_name = "MS")] + watch_heartbeat_ms: u64, #[arg(long, default_value_t = DEFAULT_RELAY_MAX_IDLE_TIMEOUT_MS, value_name = "MS")] relay_idle_timeout_ms: u64, /// QUIC keepalive interval for relayed reverse tunnels; 0 disables keepalive @@ -108,6 +117,18 @@ enum RouterTunnelConfig { impl RouterStartupConfig { fn from_args(args: Args) -> Result { + ensure!( + !args.grpc_pylon_dial_addr.trim().is_empty(), + "--grpc-pylon-dial-addr must not be empty" + ); + ensure!( + args.advertised_grpc_port > 0, + "--advertised-grpc-port must be greater than 0" + ); + ensure!( + args.watch_heartbeat_ms > 0, + "--watch-heartbeat-ms must be greater than 0" + ); let relay_endpoint_config = relay_endpoint_config_from_args(&args)?; let server_identity_reloader = server_identity_reloader_from_args(&args)?; // Read the mounted pair once. The reloader validated and owns these @@ -125,8 +146,11 @@ impl RouterStartupConfig { }; let grpc = GrpcRouterConfig { advertised_hostname_template: args.advertised_hostname_template.clone(), + advertised_grpc_port: args.advertised_grpc_port, + grpc_pylon_dial_addr: args.grpc_pylon_dial_addr, target_namespace: args.target_namespace.clone(), connect_timeout: Duration::from_millis(args.connect_timeout_ms), + watch_heartbeat_interval: Duration::from_millis(args.watch_heartbeat_ms), }; let tunnel = match args.tunnel_protocol { RouterTunnelProtocol::RawQuic => { @@ -338,9 +362,12 @@ fn log_startup(config: &RouterStartupConfig) { target_namespace = %config.grpc.target_namespace, target_service_name = %config.target_build_config.service_name, advertised_hostname_template = %config.grpc.advertised_hostname_template, + advertised_grpc_port = config.grpc.advertised_grpc_port, + grpc_pylon_dial_addr = %config.grpc.grpc_pylon_dial_addr, grpc_port_name = %config.target_build_config.grpc_port_name, quic_port_name = %config.target_build_config.quic_port_name, connect_timeout_ms = config.grpc.connect_timeout.as_millis(), + watch_heartbeat_ms = config.grpc.watch_heartbeat_interval.as_millis(), relay_idle_timeout_ms = relay_idle_timeout.as_millis(), relay_keep_alive_ms = relay_keep_alive.map_or(0, |duration| duration.as_millis()), quic_insecure, @@ -401,9 +428,15 @@ mod tests { use super::*; fn router_argv<'a>(extra: &'a [&'a str]) -> impl Iterator { - ["stargate-k8s-router", "--target-namespace", "prod"] - .into_iter() - .chain(extra.iter().copied()) + [ + "stargate-k8s-router", + "--target-namespace", + "prod", + "--grpc-pylon-dial-addr", + "https://stargate-router.example:443", + ] + .into_iter() + .chain(extra.iter().copied()) } fn router_args(extra: &[&str]) -> Args { @@ -449,6 +482,33 @@ mod tests { ); } + #[test] + fn router_cli_requires_a_nonempty_pylon_grpc_dial_address() { + let missing = Args::try_parse_from(["stargate-k8s-router", "--target-namespace", "prod"]); + assert!(missing.is_err(), "Pylon dial address must be required"); + + let mut empty_args = router_args(&[]); + empty_args.grpc_pylon_dial_addr.clear(); + let empty = RouterStartupConfig::from_args(empty_args) + .err() + .expect("empty Pylon dial address must be rejected"); + assert_eq!( + empty.to_string(), + "--grpc-pylon-dial-addr must not be empty" + ); + } + + #[test] + fn router_cli_rejects_a_zero_watch_heartbeat() { + let error = RouterStartupConfig::from_args(router_args(&["--watch-heartbeat-ms", "0"])) + .err() + .expect("zero Watch heartbeat must be rejected"); + assert_eq!( + error.to_string(), + "--watch-heartbeat-ms must be greater than 0" + ); + } + #[test] fn raw_quic_rejects_the_webtransport_only_upstream_trust_option() { let upstream_ca = test_file(b"upstream-ca-bytes"); @@ -483,12 +543,16 @@ mod tests { "stargate-ready", "--advertised-hostname-template", "{pod_name}.{namespace}.example", + "--advertised-grpc-port", + "41071", "--grpc-port-name", "grpc-control", "--quic-port-name", "quic-tunnel", "--connect-timeout-ms", "2500", + "--watch-heartbeat-ms", + "1750", "--relay-idle-timeout-ms", "20000", "--relay-keep-alive-ms", @@ -510,7 +574,16 @@ mod tests { config.grpc.advertised_hostname_template, "{pod_name}.{namespace}.example" ); + assert_eq!(config.grpc.advertised_grpc_port, 41071); + assert_eq!( + config.grpc.grpc_pylon_dial_addr, + "https://stargate-router.example:443" + ); assert_eq!(config.grpc.connect_timeout, Duration::from_millis(2500)); + assert_eq!( + config.grpc.watch_heartbeat_interval, + Duration::from_millis(1750) + ); assert_eq!( config.target_build_config, TargetBuildConfig { diff --git a/src/libraries/rust/stargate/crates/stargate-k8s-router/src/watcher.rs b/src/libraries/rust/stargate/crates/stargate-k8s-router/src/watcher.rs index 1bd3ba4f1..394a1548b 100644 --- a/src/libraries/rust/stargate/crates/stargate-k8s-router/src/watcher.rs +++ b/src/libraries/rust/stargate/crates/stargate-k8s-router/src/watcher.rs @@ -79,16 +79,23 @@ pub async fn run_endpoint_slice_watcher( } struct WatcherState { - store: BTreeMap, - init_store: BTreeMap, + store: BTreeMap, + init_store: BTreeMap, + next_revision: u64, build_config: TargetBuildConfig, } +struct ObservedSlice { + revision: u64, + slice: EndpointSlice, +} + impl WatcherState { fn new(build_config: TargetBuildConfig) -> Self { Self { store: BTreeMap::new(), init_store: BTreeMap::new(), + next_revision: 0, build_config, } } @@ -100,20 +107,39 @@ impl WatcherState { return None; } Event::InitApply(slice) => { - self.init_store.insert(slice_key(&slice), slice); + let key = slice_key(&slice); + let observed = self.observe(slice); + self.init_store.insert(key, observed); return None; } Event::InitDone => { self.store = std::mem::take(&mut self.init_store); } Event::Apply(slice) => { - self.store.insert(slice_key(&slice), slice); + let key = slice_key(&slice); + let observed = self.observe(slice); + self.store.insert(key, observed); } Event::Delete(slice) => { self.store.remove(&slice_key(&slice)); } } - Some(snapshot(self.store.values(), &self.build_config)) + Some(self.snapshot()) + } + + fn observe(&mut self, slice: EndpointSlice) -> ObservedSlice { + let revision = self.next_revision; + self.next_revision = self.next_revision.wrapping_add(1); + ObservedSlice { revision, slice } + } + + fn snapshot(&self) -> TargetSnapshot { + let mut observed = self.store.values().collect::>(); + observed.sort_unstable_by_key(|item| item.revision); + snapshot( + observed.into_iter().map(|item| &item.slice), + &self.build_config, + ) } } @@ -221,4 +247,83 @@ mod tests { assert!(snapshot.target_for_pod("stargate-0").is_none()); assert!(snapshot.target_for_pod("stargate-1").is_some()); } + + #[test] + fn apply_replaces_rolled_pod_without_leaving_stale_target() { + let mut state = WatcherState::new(config()); + state + .apply(Event::Apply(slice( + "slice-a", + "request-router-old", + "10.0.0.10", + ))) + .expect("initial Apply should publish a snapshot"); + + let snapshot = state + .apply(Event::Apply(slice( + "slice-a", + "request-router-new", + "10.0.0.11", + ))) + .expect("replacement Apply should publish a snapshot"); + + assert_eq!(snapshot.ready_count(), 1); + assert!(snapshot.target_for_pod("request-router-old").is_none()); + assert_eq!( + snapshot + .target_for_pod("request-router-new") + .map(|target| target.grpc_addr), + Some("10.0.0.11:50071".to_string()) + ); + } + + #[test] + fn apply_and_delete_track_scale_up_and_scale_down() { + let mut state = WatcherState::new(config()); + let first = slice("slice-a", "request-router-a", "10.0.0.10"); + let second = slice("slice-b", "request-router-b", "10.0.0.11"); + state + .apply(Event::Apply(first.clone())) + .expect("first Apply should publish a snapshot"); + let scaled_up = state + .apply(Event::Apply(second.clone())) + .expect("second Apply should publish a snapshot"); + + assert_eq!(scaled_up.ready_count(), 2); + + let scaled_down = state + .apply(Event::Delete(first)) + .expect("Delete should publish a snapshot"); + assert_eq!(scaled_down.ready_count(), 1); + assert!(scaled_down.target_for_pod("request-router-a").is_none()); + assert!(scaled_down.target_for_pod("request-router-b").is_some()); + } + + #[test] + fn latest_slice_observation_wins_for_duplicate_pod_identity() { + let mut state = WatcherState::new(config()); + state + .apply(Event::Apply(slice( + "slice-z", + "request-router-0", + "10.0.0.10", + ))) + .expect("initial Apply should publish a snapshot"); + + let snapshot = state + .apply(Event::Apply(slice( + "slice-a", + "request-router-0", + "10.0.0.11", + ))) + .expect("replacement Apply should publish a snapshot"); + + assert_eq!(snapshot.ready_count(), 1); + assert_eq!( + snapshot + .target_for_pod("request-router-0") + .map(|target| target.grpc_addr), + Some("10.0.0.11:50071".to_string()) + ); + } }