Surface NodeClaim drift/rollout progress in NodePool status

[javanthropus]

Description

What problem are you trying to solve?

Today, drift is only observable per-NodeClaim, via the Drifted status condition. There is no aggregate, NodePool-level signal that tells you how far a NodePool is through reconciling drift across the nodes it owns.

This makes it hard for external systems to gate on "this NodePool has finished rolling out a change" without listing and aggregating NodeClaims themselves. Concretely, we run Karpenter under Argo CD (GitOps). When a NodePool/EC2NodeClass change triggers drift, we want Argo CD to report the Application as Progressing until the drift-driven node replacement is substantially complete, and Healthy once it is.

Argo CD's health is evaluated per resource with a sandboxed Lua check that has no access to other resources. So a NodePool health check can only read the NodePool's own status. A NodeClaim health check could in principle read each NodeClaim's Drifted condition, but:

• It would require NodeClaims to appear as children of the NodePool in Argo's resource tree (which, in our multi-cluster setup, they currently do not), and
• It forces all-or-nothing (100%) semantics, with no notion of a tolerance threshold or a settling window.

As a result we maintain a PostSync Job that polls NodeClaims, groups them by karpenter.sh/nodepool, computes the percentage no longer Drifted, and blocks until each NodePool crosses a threshold (e.g. 90%). This is exactly the kind of aggregation we'd expect Karpenter itself to be able to expose, since it already owns the NodeClaims and tracks their drift state.

Proposal

Expose drift/rollout progress at the NodePool level, in NodePool.status. Any of the following would be sufficient for our use case (in rough order of preference):

1. Counts in NodePool.status, e.g. status.driftedNodeClaims / status.nodeClaims (analogous to the existing status.nodes and status.resources), so consumers can compute completion percentage directly.
2. A NodePool-level condition such as Drifted (status: "True" while any owned NodeClaim is drifted, "False" once reconciliation is complete), mirroring how the per-NodeClaim Drifted condition works.
3. Both — a condition for a simple boolean gate plus counts for threshold-based logic.

This would let any GitOps/automation system gate on a NodePool's own status without re-implementing NodeClaim aggregation, and it would make rollout progress visible in kubectl get nodepool and dashboards.

How important is this feature to you?

Moderately important. We have a working PostSync Job that does the aggregation today, so we are not blocked, but it adds operational surface (a Job + ServiceAccount + ClusterRole + a maintained image per cluster) purely to compute information Karpenter already has internally. A NodePool-level status field/condition would let us replace that machinery with a standard Argo CD custom health check and would benefit anyone integrating Karpenter rollouts with external orchestration.

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[k8s-ci-robot]

This issue is currently awaiting triage.

If Karpenter contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[sumukha-radhakrishna]

Right, Karpenter doesn't expose aggregate drift progress at the NodePool level today. The only signals are:

• Per-NodeClaim: status.conditions[Drifted] on each NodeClaim
• NodePool-level: status.resources (aggregate resource counts) and status.nodes exist, but no
  drift-specific counters

Karpenter's disruption controller processes NodeClaims individually and doesn't maintain a "rollout" concept. Drift is detected per-node and resolved opportunistically (respecting disruption budgets, PDBs, etc.). There's no internal state tracking "X of Y drift replacements complete." But we can look into surfacing this thru status

[jigisha620]

@javanthropus,
Curious, what do you think about having a metric that exposes this information?

[javanthropus]

Our immediate need is a way to cause ArgoCD to report that a rollout for a NodePool is in progress. We want to use this as a signal for users and Argo to know when an Argo app that deploys a NodePool is "done" so that they can move on to other apps in sequence. We currently do that now with a hacky script run in a pod as part of a postSync action. This feature will simplify that by eliminating the need for the script.

Argo itself can't query metrics as far as I know. Its health check logic only has access to the NodePool resource, so all details must come from there. @cnmcavoy is already working on a custom health check that we'll use with Argo for this purpose to go with his PR. If it works well, we'll probably submit it to the Argo project for inclusion so that others may benefit.

[DerekFrank]

If the ArgoCD maintainers think metric-awareness is not within scope of their project, I think we would be willing to surface a drift progression status on the nodepool. Can you update this issue with a link to the issue in their repository so we can discuss this? I'd like to come to a single, opinionated stance for monitoring Karpenter drift rollouts with ArgoCD

[cheeseandcereal]

Wanted to add a second concrete consumer for this, since I think it shows the request isn't Argo-specific and generalizes to any orchestration layer whose readiness is scoped to a single resource.

We're looking at managing NodePool + EC2NodeClass through kro (https://github.com/kubernetes-sigs/kro) (grouping them into a single custom API via a ResourceGraphDefinition). kro decides whether an instance is still rolling out vs. settled by evaluating readyWhen CEL expressions against the resources it manages. That works great for resources kro declares in its graph, since it can read nodePool.status directly, but it hits the exact wall described here:

• The NodePool and EC2NodeClass are in the graph, so their status is readable.
• The NodeClaims are not. Karpenter creates them as a side effect, their count/names aren't known at authoring time, and there's no way to declaratively fold arbitrary controller-owned children into the instance's readiness.

So without a NodePool-level signal, a kro instance wrapping a NodePool flips to "ready" as soon as the NodePool object is admitted, not when the drift-driven replacement actually completes. This is the same premature-healthy behavior described for Argo CD, and the same "re-implement NodeClaim aggregation yourself" workaround.

If the proposal lands, the counts option (status.driftedNodeClaims / status.nodeClaims) would let us express rollout readiness directly, including a tolerance threshold, without any external aggregation job:

readyWhen:
  - ${nodePool.status.nodeClaims > 0 &&
      nodePool.status.driftedNodeClaims <= nodePool.status.nodeClaims / 10}

A NodePool-level Drifted condition would cover the simple all-or-nothing gate too, but the counts are what unlock threshold-based logic for us. Between the two, counts (option 1) are the more valuable for this use case, and both together (option 3) would be ideal.

+1 on exposing this at the NodePool level, since it benefits any external orchestrator gating on a single resource's status, not just GitOps controllers doing per-resource health checks.

[alegermer]

Wanted to add some context on why putting aggregate NodeClaim state in NodePool.status makes sense, beyond just the ArgoCD use case.

This is basically how every core workload controller already works. A Deployment tells you replicas, updatedReplicas, readyReplicas, and unavailableReplicas, plus a Progressing condition that flips as the rollout moves along. DaemonSets do the same with desiredNumberScheduled, updatedNumberScheduled, and numberReady. That's the whole reason kubectl rollout status works: you can ask the parent resource "are we done yet?" without going and counting children yourself.

I believe NodePool to NodeClaim conveys the same type of relationship: a parent driving a set of children toward a desired state, with drift reconciliation playing the role of a rolling update. NodePool already aggregates status.nodes and status.resources, but the one question people actually ask during a rollout ("is this pool still churning or has it settled?") can only be answered by going through individual NodeClaims. Karpenter already knows the answer internally, it just doesn't write it anywhere readable on the NodePool as @javanthropus correctly pointed out.

ArgoCD makes the gap visible, but it's not really an ArgoCD problem. Its health checks can only look at the resource being evaluated, and the docs tell CRD authors to surface child health in the parent's status for exactly this reason. kstatus and similar tooling make the same assumption. Follow the convention and all of that tooling works for free.

Regarding the shape of these status fields, I'd lean toward counts as the main thing, same as Deployments. Counts let each consumer pick their own threshold (say, call it settled at 90% undrifted), which fits how disruption budgets already treat node rollouts as gradual.

[jmdeal]

I believe NodePool to NodeClaim conveys the same type of relationship: a parent driving a set of children toward a desired state, with drift reconciliation playing the role of a rolling update.

This is only true to an extent. Drifted, how it is defined today, is a property of the NodeClaim rather than the NodePool. The reason I draw the distinction is that there are multiple vectors that can drive a NodeClaim to be drifted: explicit NodePool updates, explicit NodeClass updates, out-of-band updates (e.g. new AMI release with the EC2 provider). A NodeClaim's drifted status is the union of all of those conditions, but we don't offer a more granular view.

The goal of this issue, as I understand it, is to provide ArgoCD with a mechanism to understand when drift induced by a change to a NodePool spec has finished propagating. This would mean only making the following drift actions visible:

	Static	Dynamic
NodePool	y	n (arguably doesn't exist?)
NodeClass	n	n

Whether or not this distinction matters is situational I think. The scenario that I envision being undesirable is when you have frequent out-of-band updates (e.g. AMI releases, capacity reservation reshuffling, etc) which keep NodeClaims drifted even after static NodePool configuration updates have been rolled out. It would be great if the community could provide examples for scenarios where they would need this differentiation, where it doesn't matter, and where you need the union of all drift types. We should understand this before we naively surface the existing drift condition on the NodePool.

One other technical problem we'll need to address is linearizability. The existing linked PR reconciles the status condition counts onto the NodePool status asynchronously. While this probably works most of the time, there are edge case. For example, if there's CPU starvation or the count controller is delayed for any other reason (like throttling), the last drifted count could be persisted for an unbounded period of time. This could cause gates to be prematurely passed. You can't rely on an edge (e.g. the number of drifted nodes increasing) because an update to a NodePool is not guaranteed to induce drift (e.g. restricting requirements but the pruned instance types weren't present in the cluster). We need some way to associate a drifted count with the revision of the NodePool (likely generation).

/kind feature
/triage needs-information