DRA kubelet plugin silently fails to schedule when GPU nodes lack the eviction label

[yuanchen8911]

Problem

PR #2401 (closing #1253) wired the DRA kubelet-plugin eviction contract by writing one node label into two places:

• nvidia-dra-driver-gpu kubelet-plugin nodeSelector — the plugin only runs on labeled nodes
• gpu-operator driver.manager.env.NODE_LABEL_FOR_GPU_POD_EVICTION — the driver-manager knows which pods to evict before a driver upgrade

The design is sound. The driver-manager deschedules the plugin by blanking the label and reschedules by restoring it, so the label must gate placement for the mechanism to work at all. nvidia.com/dra-kubelet-plugin=true in the nodeSelector is load-bearing, not incidental.

The problem is the failure mode, not the design. Deploying a bundle onto GPU nodes that do not carry the label produces:

• nvidia-dra-driver-gpu-kubelet-plugin DaemonSet at DESIRED=0
• no DRA kubelet plugins on any node
• no ResourceSlices — so no ComputeDomain/IMEX capability
• no error anywhere. helm upgrade/install reports success and deploy.sh exits 0.

Nothing the operator runs mentions the requirement. Verified on a generated bundle:

• README.md — no mention
• deploy.sh — no mention
• the only occurrence is 015-nvidia-dra-driver-gpu/values.yaml:18, as a nodeSelector entry

The prerequisite is documented (docs/user/cli-reference.md:1643, docs/user/bundling.md:197) but only in pages an operator would have to already know to consult.

Observed

EKS GB300 cluster (aws-us-east-2-nhensley-gb300), 2026-08-28. An AICR stack deployed 2026-08-26 — before #2401 merged — had a working kubelet-plugin DaemonSet selecting on nodeGroup=gpu-worker alone, 2/2 ready.

Regenerating the bundle from current main and running helm upgrade on gpu-operator and nvidia-dra-driver-gpu added the second selector. Both upgrades reported Upgrade complete. Both kubelet plugins silently disappeared; the DaemonSet went to DESIRED=0.

The driver-manager was not at fault — its logs show correct save/restore behaviour:

21:45:57  Current value of "nvidia.com/dra-kubelet-plugin"=          <- already empty
21:50:34  Rescheduling all GPU clients on the current node by enabling their component-specific nodeSelector labels

It captured the pre-existing (empty) value and faithfully restored it. There was simply nothing to restore.

Applying the label resolved it completely — DESIRED 0 → 2, both plugins Running, ResourceSlices published within ~40s:

kubectl label node <gpu-node> nvidia.com/dra-kubelet-plugin=true

Why this had not been seen before

Pure timing. The cluster's stack was deployed 2026-08-26; #2401 merged 2026-08-27 16:27 PDT. Revision 2 of the deployed DRA release carried kubeletPlugin.nodeSelector: {"nodeGroup": "gpu-worker"} with no eviction label, so the original bundle predates the change. The upgrade described above was the first deployment anywhere on this cluster to carry the new selector.

Expected to affect fresh installs too — inferred, not yet observed

A fresh ./deploy.sh onto unlabelled nodes should fail identically: the plugin never schedules, DESIRED=0, no error. Upgrade is worse only in that working functionality disappears rather than never appearing.

Stated precisely: this has not been demonstrated. No fresh install with a post-#2401 bundle has been run. The inference rests on three verified facts — the selector is present in the rendered values, the bundle's README.md and deploy.sh never mention the label, and nothing in recipes/ writes that label (no component, no nodewright-customizations, no manifest), so no automation would rescue a fresh install.

Worth confirming on a clean cluster before deciding scope: if fresh installs are somehow unaffected, this narrows to the upgrade path and the fix is smaller.

The durable risk is node lifecycle

A one-off kubectl label is a repair, not a configuration. It does not survive:

• node replacement or recycling
• cluster autoscaling adding GPU nodes
• a nodegroup scaled from zero

Any GPU node added afterwards arrives unlabeled and silently runs without the DRA kubelet plugin, leaving the cluster in a partially-DRA-enabled state — worse than uniform failure, because it is intermittent and node-dependent.

The label therefore belongs in the node pool definition (EKS managed nodegroup labels, Karpenter nodeClass, etc.), alongside nodeGroup=gpu-worker which is already provisioned that way.

Proposed

Any one of these would close the silent-failure gap; the first is closest to existing precedent.

1. Emit a bundle-time warning, matching what aicr bundle already does for StorageClass: when the DRA and GPU Operator components are both enabled and the eviction label is applied, warn that every GPU node must carry it and that the plugin will not schedule otherwise. aicr bundle already warns when a PVC would depend on unverifiable cluster state (docs/user/cli-reference.md:1684) — this is the same class of dependency and currently gets nothing.
2. Surface it in the bundle artifact — a line in the generated README.md, and/or a pre-flight check in deploy.sh that fails loudly when a rendered nodeSelector matches zero nodes.
3. Documentation updates, which are needed regardless of which mechanism above is chosen:
   • docs/user/bundling.md (around L197) and docs/user/cli-reference.md (around L1643) currently present the labelling as a one-off kubectl label node step. Both should state that the label must be applied at node-pool provisioning time — EKS managed nodegroup labels, Karpenter nodeClass, or equivalent — because an ad hoc label does not survive node replacement, recycling, autoscaling, or a nodegroup scaled from zero.
   • Both should call out the existing-cluster upgrade case explicitly. The current text reads as a fresh-install prerequisite, so an operator regenerating a bundle and running helm upgrade has no reason to revisit it — which is exactly how this was hit.
   • Both should state the failure mode: an unlabelled node yields DESIRED=0, no ResourceSlices, and no error from Helm or deploy.sh, so the absence is not self-announcing.
   • docs/integrator/ deployment guides that walk through node-pool creation should include the label alongside the existing nodeGroup=gpu-worker guidance, so it is set where the other required node labels are set.
   • Whichever warning is added in (1) or (2) should be documented next to the existing StorageClass warning in docs/user/cli-reference.md:1684, so the two cluster-state dependencies are described in one place.

Out of scope

• Changing the eviction mechanism itself. The nodeSelector is required for blank/restore to work; Auto-wire DRA kubelet-plugin eviction with GPU Operator driver upgrades #1253's contract is correct and its implementation behaved exactly as designed here.
• Auto-labeling nodes from AICR. Node labels are cluster infrastructure, the same category as StorageClass, and AICR deliberately does not own them.

Related: #1253 (closed by #2401), #2439 (surfaced during its cluster validation; not caused by it)