Skip to content

🐛 Measure nodeDeletionTimeout from node deletion start - #14026

Open
saisankargochhayat wants to merge 1 commit into
kubernetes-sigs:mainfrom
saisankargochhayat:machine-node-deletion-phase-clock
Open

🐛 Measure nodeDeletionTimeout from node deletion start#14026
saisankargochhayat wants to merge 1 commit into
kubernetes-sigs:mainfrom
saisankargochhayat:machine-node-deletion-phase-clock

Conversation

@saisankargochhayat

@saisankargochhayat saisankargochhayat commented Aug 3, 2026

Copy link
Copy Markdown

What this PR does / why we need it:
nodeDeletionTimeoutSeconds is evaluated against the Machine's deletionTimestamp, so node drain, volume detach and infrastructure deletion all spend the same budget. Since nodeDrainTimeoutSeconds and nodeVolumeDetachTimeoutSeconds default to 0 (unlimited) and infrastructure deletion has no timeout, the budget is typically already exhausted by the time Node deletion is attempted — so raising nodeDeletionTimeoutSeconds has no effect unless the value exceeds the time those phases consume.

This tracks nodeDeletionStartTime in status.deletion and measures the timeout from it, consistent with how nodeDrainTimeoutSeconds and nodeVolumeDetachTimeoutSeconds were changed in #11166.

Scope and behaviour change:

  • This does change behaviour: when Node deletion fails, the Machine now retries for up to nodeDeletionTimeoutSeconds measured from when Node deletion started, rather than giving up against a budget that was already spent.
  • It does not change the default. nodeDeletionRetryTimeout (10s, not configurable) and the nodeDeletionTimeoutSeconds default (10s) are equal, so a single poll window consumes the whole budget and the finalizer is still removed after the first failed window. The change takes effect once the configured budget exceeds the poll window — e.g. 30s gives three windows.
  • Total Machine deletion time was already effectively unbounded, since drain and detach default to 0 and infra deletion has no timeout. deletionTimestamp + nodeDeletionTimeoutSeconds was never a ceiling on deletion — it only truncated the final phase.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #14024

This PR was written in part with the assistance of generative AI.

@linux-foundation-easycla

linux-foundation-easycla Bot commented Aug 3, 2026

Copy link
Copy Markdown

CLA Signed
The committers listed above are authorized under a signed CLA.

  • ✅ login: saisankargochhayat / name: Sai Sankar Gochhayat (a85b3f5)

@kubernetes-prow kubernetes-prow Bot added the do-not-merge/needs-area PR is missing an area label label Aug 3, 2026
@kubernetes-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign fabriziopandini for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubernetes-prow

Copy link
Copy Markdown
Contributor

Welcome @saisankargochhayat!

It looks like this is your first PR to kubernetes-sigs/cluster-api 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/cluster-api has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

@kubernetes-prow kubernetes-prow Bot added cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Aug 3, 2026
@kubernetes-prow

Copy link
Copy Markdown
Contributor

Hi @saisankargochhayat. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@saisankargochhayat saisankargochhayat changed the title Measure nodeDeletionTimeout from node deletion start 🐛 Measure nodeDeletionTimeout from node deletion start Aug 3, 2026
@kubernetes-prow kubernetes-prow Bot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Aug 3, 2026
@saisankargochhayat

Copy link
Copy Markdown
Author

/area machine

@kubernetes-prow kubernetes-prow Bot added area/machine Issues or PRs related to machine lifecycle management and removed do-not-merge/needs-area PR is missing an area label labels Aug 3, 2026
@saisankargochhayat
saisankargochhayat force-pushed the machine-node-deletion-phase-clock branch from 7f9fe40 to f0f2d2f Compare August 3, 2026 23:17
nodeDeletionTimeoutSeconds was evaluated against the Machine's deletionTimestamp,
so node drain, volume detach and infrastructure deletion all spent the same
budget. Since nodeDrainTimeoutSeconds and nodeVolumeDetachTimeoutSeconds default
to 0 (unlimited) and infrastructure deletion has no timeout, the budget is
typically already exhausted by the time Node deletion is attempted, so raising
nodeDeletionTimeoutSeconds has no effect.

Track nodeDeletionStartTime in status.deletion and measure the timeout from it,
consistent with how nodeDrainTimeoutSeconds and nodeVolumeDetachTimeoutSeconds
were changed in kubernetes-sigs#11166.
@saisankargochhayat
saisankargochhayat force-pushed the machine-node-deletion-phase-clock branch from f0f2d2f to ee79bc9 Compare August 3, 2026 23:21
// if the nodeDeletionTimeoutSeconds is exceeded.
// Only present when the Machine has a deletionTimestamp and deleting the Node had been started.
// +optional
NodeDeletionStartTime metav1.Time `json:"nodeDeletionStartTime,omitempty,omitzero"`

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we be adding this? Not sure but it seems like we would in effect be creating drift for an established api that already exists through several versions.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was taking some inspiration from #14024 111614024 6, finding a way to persist phase specific times.
We need a way to persist the timestamp for when Node deletion actually begins; otherwise nodeDeletionTimeoutSeconds includes time spent draining, waiting for hooks/volumes, and deleting infrastructure.
Any recommendations on how to persist this phase transition without extending MachineDeletionStatus?

@troy0820

troy0820 commented Aug 7, 2026

Copy link
Copy Markdown
Member

/hold

@kubernetes-prow kubernetes-prow Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 7, 2026
@troy0820

troy0820 commented Aug 7, 2026

Copy link
Copy Markdown
Member

Also, please abide by the AI usage policy https://kubernetes.io/blog/2026/06/26/open-source-maintainership-in-the-age-of-ai/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/machine Issues or PRs related to machine lifecycle management cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

nodeDeletionTimeout is measured from deletionTimestamp, making the 10s default unreachable in practice

2 participants