fix(recipes): accelerated-node selector routed to a gpu-operator value the chart ignores

[yuanchen8911]

Summary

recipes/registry.yaml routes the accelerated-node selector to daemonsets.nodeSelector on gpu-operator, but the chart has no such value. It is silently ignored, so --accelerated-node-selector does not constrain GPU Operator operand placement. The docs state that it does.

Evidence

The chart has no daemonsets.nodeSelector. helm show values gpu-operator --version v26.7.0 renders a daemonsets: block containing labels, annotations, priorityClassName, tolerations, updateStrategy and rollingUpdate — and no nodeSelector. The only nodeSelector values the chart exposes are operator.nodeSelector (the controller Deployment) and driver.nvidiaDriverCRD.nodeSelector (the NvidiaDriver CRD path, which AICR leaves enabled: false).

Rendering confirms it is dropped, on both the version main pins and the one #2439 bumps to:

$ helm template go gpu-operator --repo https://helm.ngc.nvidia.com/nvidia \
    --version <v> -n gpu-operator --set daemonsets.nodeSelector.nodeGroup=gpu-worker
# ClusterPolicy daemonsets.nodeSelector:
v26.3.3   ABSENT
v26.7.0   ABSENT

Observed on a live cluster (EKS GB300, gpu-operator v26.7.0, deployed from an AICR bundle). The release values carry the routed selector:

$ helm get values gpu-operator -n gpu-operator -o json | jq .daemonsets
{ "nodeSelector": { "nodeGroup": "gpu-worker" },
  "tolerations": [ { "key": "dedicated", "value": "worker-workload", ... } ] }

The resulting ClusterPolicy does not:

$ kubectl get clusterpolicy -o json | jq '.items[0].spec.daemonsets | {nodeSelector, tolerations: (.tolerations|length)}'
{ "nodeSelector": null, "tolerations": 2 }

And every operand selects only on the operator's own deploy labels:

gpu-feature-discovery                    {"nvidia.com/gpu.deploy.gpu-feature-discovery":"true"}
nvidia-container-toolkit-daemonset       {"nvidia.com/gpu.deploy.container-toolkit":"true"}
nvidia-dcgm                              {"nvidia.com/gpu.deploy.dcgm":"true"}
nvidia-dcgm-exporter                     {"nvidia.com/gpu.deploy.dcgm-exporter":"true"}
nvidia-device-plugin-daemonset           {"nvidia.com/gpu.deploy.device-plugin":"true"}
nvidia-device-plugin-mps-control-daemon  {"nvidia.com/gpu.deploy.device-plugin":"true","nvidia.com/mps.capable":"true"}
nvidia-driver-daemonset                  {"nvidia.com/gpu.deploy.driver":"true"}
nvidia-mig-manager                       {"nvidia.com/gpu.deploy.mig-manager":"true"}
nvidia-operator-validator                {"nvidia.com/gpu.deploy.operator-validator":"true"}

daemonsets.tolerations is a real chart value and does land — which is why the failure is asymmetric and easy to miss.

What is and is not established

Established: the routed value is ignored; the docs claim otherwise. docs/user/cli-reference.md states "GPU DaemonSets (driver, device-plugin, toolkit, dcgm): nodeSelector=nodeGroup=gpu-worker", which has never held on either pinned version.

Not established: that this causes incorrect placement in any supported configuration. GPU Operator targets operands at GPU nodes through its own nvidia.com/gpu.deploy.* labels, driven by GFD/NFD detection, so operands already land only on GPU nodes — which is what the accelerated selector was meant to achieve. This is filed as a correctness-of-configuration and documentation problem, not a demonstrated placement defect.

Where it could matter is a cluster with multiple GPU pools where only some should run operands: AICR's selector would not constrain that, and the operator would claim every GPU node. Whether that is a configuration AICR intends to support is the open question below.

Proposed work

1. Decide whether AICR still needs to constrain GPU Operator operand placement. The chart exposes no lever for it, so this is a design decision, not a mechanical fix:
   • if no, drop the routing and the documented guarantee;
   • if yes, a different mechanism is needed — the operator's deploy labels are operator-owned, so this likely means an upstream ask.
2. Remove or correct nodeSelectorPaths: [daemonsets.nodeSelector] in recipes/registry.yaml. Today it writes a value into every bundle containing gpu-operator that has no effect. Note removing it moves the stock-render golden digests for every leaf containing gpu-operator. OCP leaves resolve gpu-operator-ocp-olm, which carries no such routing.
3. Correct the docs. docs/user/cli-reference.md (the GPU DaemonSets placement claim), plus any other page repeating it.

Scope is gpu-operator only — grep -n 'daemonsets.nodeSelector' recipes/registry.yaml returns a single routing entry, and the -ocp variant does not route it.

Not verified

• Whether any AICR recipe or overlay relies on the routed selector taking effect.

Reading GPU Operator's reconcile code is not required to establish that the current route is ineffective: the ClusterPolicy CRD schema itself has no daemonsets.nodeSelector field — kubectl explain clusterpolicy.spec.daemonsets lists only labels, priorityClassName and tolerations — so the value is dropped before admission. (v26.7's GPUCluster template does copy the daemonsets map wholesale, but that CRD schema does not permit nodeSelector either.) Reconcile behaviour matters only when evaluating replacement mechanisms under proposed work item 1.

Context

Found while reviewing #2471 (making the DRA eviction node label opt-in), where an early draft of the recovery documentation incorrectly assumed the routed selector reached the driver DaemonSet. Deliberately kept out of that PR: it is pre-existing on both pinned versions, independent of the opt-in change, and needs the design decision above. Also kept out of #2439, since the bump neither introduces nor changes the behavior.

Related: #2471, #2439, #2469

[yuanchen8911]

Corrected — my earlier wording conflated two distinct reachability questions.

Separating them properly:

• The routing being ignored is demonstrably reachable, not merely possible: it occurs in every bundle that enables gpu-operator and passes an accelerated-node selector. That is established by the evidence in the body — chart values, helm template on both pinned versions, and the live ClusterPolicy showing nodeSelector: null.
• Harmful placement — an operand landing outside its intended pool — remains unproven. The operator's own nvidia.com/gpu.deploy.* labels already scope operands to GPU nodes.

The issue body's conservative framing is the correct one, and my original comment blurred the first point into the second. Retaining the current title, Enhancement type and P2 until harmful cross-pool placement is actually demonstrated.

The alternative title suggested during the #2471 review, if the eventual disposition warrants it: bug(bundler): GPU Operator drops the accelerated node selector for operand DaemonSets.