Skip to content

feat: label NodeClaim termination and lifetime histograms with instance and capacity type - #62

Merged
pfernandes21 merged 1 commit into
mainfrom
devin/1788444704-lifecycle-instance-type-labels
Sep 3, 2026
Merged

feat: label NodeClaim termination and lifetime histograms with instance and capacity type#62
pfernandes21 merged 1 commit into
mainfrom
devin/1788444704-lifecycle-instance-type-labels

Conversation

@pfernandes21

Copy link
Copy Markdown

Summary

The GPU Spot Availability dashboard's interruption panels (instance termination time, end-to-end termination, spot node lifetime before interruption) could only follow $cloud via a nodepool join — the underlying histograms had no instance-type dimension. This stamps the launched instance/capacity type on every histogram recorded when a NodeClaim is torn down so those panels can honor $instance_type / $gpu_type directly.

 karpenter_nodeclaims_instance_termination_duration_seconds{nodepool, cause}
+karpenter_nodeclaims_instance_termination_duration_seconds{nodepool, instance_type, capacity_type, cause}
 karpenter_nodeclaims_termination_duration_seconds{nodepool, cause}
+karpenter_nodeclaims_termination_duration_seconds{nodepool, instance_type, capacity_type, cause}
 karpenter_nodeclaims_lifetime_seconds{nodepool, capacity_type, origin, cause}
+karpenter_nodeclaims_lifetime_seconds{nodepool, instance_type, capacity_type, origin, cause}

Values come from the labels the cloud provider stamps on the NodeClaim at launch (node.kubernetes.io/instance-type, karpenter.sh/capacity-type), read via a shared terminationLabels(nodeClaim) in the lifecycle controller. A NodeClaim deleted before it launched (failed launch, registration timeout) reports unknown instead of an empty label — this also changes lifetime_seconds{capacity_type=""} to capacity_type="unknown" for that case. Cardinality is bounded by the instance types a NodePool can actually launch, as with the existing karpenter_nodeclaims_terminated_total labels.

Tests: the cloud_interrupted termination test now asserts the launched instance/capacity type on all three histograms, and a new case covers the never-launched → unknown path. go test ./pkg/controllers/nodeclaim/lifecycle/ -race passes against envtest 1.36.

Follow-up in the monorepo once merged: repin infra/kraftsman/fork, bump exa-scale, and switch the dashboard's interruption/lifetime panels from the nodepool→cloud join to direct instance_type=~"$instance_type" matchers.

Link to Devin session: https://app.devin.ai/sessions/78f39945389b48c8b60f35de366cb138
Open in Devin Desktop: https://app.devin.ai/desktop/session/78f39945389b48c8b60f35de366cb138?variant=devin
Requested by: @pfernandes21

…ce and capacity type

karpenter_nodeclaims_{instance_termination,termination,lifetime}_seconds only carried
nodepool (+ capacity_type on lifetime), so spot interruption and node-lifetime panels
could not be filtered by instance type. Every histogram recorded at NodeClaim teardown
now carries instance_type and capacity_type, read from the labels the cloud provider
stamped at launch, with `unknown` for a NodeClaim deleted before it ever launched.

Assisted-by: devin:claude-opus-4.5
@devin-ai-integration

Copy link
Copy Markdown

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@pfernandes21
pfernandes21 merged commit 8db0a71 into main Sep 3, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant