feat: label NodeClaim termination and lifetime histograms with instance and capacity type - #62
Merged
pfernandes21 merged 1 commit intoSep 3, 2026
Conversation
…ce and capacity type
karpenter_nodeclaims_{instance_termination,termination,lifetime}_seconds only carried
nodepool (+ capacity_type on lifetime), so spot interruption and node-lifetime panels
could not be filtered by instance type. Every histogram recorded at NodeClaim teardown
now carries instance_type and capacity_type, read from the labels the cloud provider
stamped at launch, with `unknown` for a NodeClaim deleted before it ever launched.
Assisted-by: devin:claude-opus-4.5
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The GPU Spot Availability dashboard's interruption panels (instance termination time, end-to-end termination, spot node lifetime before interruption) could only follow
$cloudvia a nodepool join — the underlying histograms had no instance-type dimension. This stamps the launched instance/capacity type on every histogram recorded when a NodeClaim is torn down so those panels can honor$instance_type/$gpu_typedirectly.karpenter_nodeclaims_instance_termination_duration_seconds{nodepool, cause} +karpenter_nodeclaims_instance_termination_duration_seconds{nodepool, instance_type, capacity_type, cause} karpenter_nodeclaims_termination_duration_seconds{nodepool, cause} +karpenter_nodeclaims_termination_duration_seconds{nodepool, instance_type, capacity_type, cause} karpenter_nodeclaims_lifetime_seconds{nodepool, capacity_type, origin, cause} +karpenter_nodeclaims_lifetime_seconds{nodepool, instance_type, capacity_type, origin, cause}Values come from the labels the cloud provider stamps on the NodeClaim at launch (
node.kubernetes.io/instance-type,karpenter.sh/capacity-type), read via a sharedterminationLabels(nodeClaim)in the lifecycle controller. A NodeClaim deleted before it launched (failed launch, registration timeout) reportsunknowninstead of an empty label — this also changeslifetime_seconds{capacity_type=""}tocapacity_type="unknown"for that case. Cardinality is bounded by the instance types a NodePool can actually launch, as with the existingkarpenter_nodeclaims_terminated_totallabels.Tests: the
cloud_interruptedtermination test now asserts the launched instance/capacity type on all three histograms, and a new case covers the never-launched →unknownpath.go test ./pkg/controllers/nodeclaim/lifecycle/ -racepasses against envtest 1.36.Follow-up in the monorepo once merged: repin
infra/kraftsman/fork, bump exa-scale, and switch the dashboard's interruption/lifetime panels from the nodepool→cloud join to directinstance_type=~"$instance_type"matchers.Link to Devin session: https://app.devin.ai/sessions/78f39945389b48c8b60f35de366cb138
Open in Devin Desktop: https://app.devin.ai/desktop/session/78f39945389b48c8b60f35de366cb138?variant=devin
Requested by: @pfernandes21