🐛 Perform draining and volume detachment once until completion

[Danil-Grigorev]

What this PR does / why we need it:

The draining logic continues to access the cluster even after the hook is removed or completed, causing a deadlock on machine removal in a non KCP backed CP provider implementation.

In a cluster with a kubelet local mode, competed draining and etcd membership removal causes inability to access API server externally, so the operation continues to error out even after success. This does not allow underlying machine to be removed.

The solution makes the PreTeminateHook agnostic to the provider, and ensures that completion of the operation does not lead to further attempts to access the cluster.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #11591

/area machine

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign chrischdi for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Danil-Grigorev]

/area bootstrap

[chrischdi]

We should not start to rely on this condition to be set as it is because its gonna be deprecated/removed in v1beta2.

Also propably a change in the behavior for all VMs (previously followed reconciliation may also run again through drain).

[Danil-Grigorev]

I unified the check under the PreTerminateDeleteHookSucceededCondition, which allows for both methods to run until completion. It is not listed in the proposal as a part of deprecation/removal AFAICT

[sbueringer]

It is not listed in the proposal as a part of deprecation/removal AFAICT

@Danil-Grigorev I think that was just not explicitly called out.

The proposal states the full list of new Machine conditions and then below

To better evaluate proposed changes, below you can find the list of current Machine's conditions:
Ready, InfrastructureReady, BootstrapReady, NodeHealthy, PreDrainDeleteHookSucceeded, VolumeDetachSucceeded, DrainingSucceeded.

(We mentioned the PreDrainDeleteHookSucceeded condition in the list of current conditions, but forgot to mention PreTerminateDeleteHookSucceeded)

[sbueringer]

Anyway. I talked to @fabriziopandini and want to propose an alternative

• We already have .status.deletion.{nodeDrainStartTime,waitForNodeVolumeDetachStartTime}
• We would propose to also add .status.deletion.{waitForPreDrainHookStartTime,waitForPreTerminateHookStartTime}
• They should be set the same way. as the other timeouts
• Then we can modify isNodeDrainAllowed & isNodeVolumeDetachingAllowed to skip drain / wait for volume detach if waitForPreTerminateHookStartTime is set
• Additionally it would be nice to skip drain / wait for volume detach if the InfraMachine either doesn't exist anymore or has a deletionTimestamp set

WDYT?

[furkatgofurov7]

Thanks for review @chrischdi 👍🏼
cc @sbueringer @enxebre @fabriziopandini hey folks, would you please have a look at this when you have time since it is blocking us (CAPRKE2 provider) to bump and use CAPI v1.9.x series?

[alexander-demicev]

/lgtm

A backport would be very much appreciated 🙏

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: 0bcc510758d20e84eb7b2d56a889fcf26def4356

[Danil-Grigorev]

/area machine

[enxebre]

is this a control plane Node? wouldn't this scenario make any other upcoming node deletion fail to query through the remote client as well?

[Danil-Grigorev]

Thanks @enxebre, I think in this case it is eventually passing the deletion. Once the ETCD member is drained, the API server healthz on the node fails, which causes exclusion from LB. It is important to get the code to remove infra machine, so it completes the deletion on the provider side. Node is removed once API server connectivity is restored, and then the Machine follow.

Thanks to @chrischdi suggestion the behavior is replicated in our CI with setting 2 annotations #11591 (comment), and it works well as a temporary solution. But other providers may hit the same issue.