[bug] Cluster scale down should not hang on destroying a disconnected machine

[steverfrancis]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

One a 3 node cluster (all CP nodes), if one of the nodes is in "Disconnected" state (because it is powered off), and I attempt to remove it from the cluster, Omni will get stuck in "Scaling down", with that machine showing "Destroying".
And just sit there. (Maybe there is a timeout, but it has been more than 5 minutes so far.)

Expected Behavior

Given that Omni knows that machine is in "DIsconnected" state, there is no reason to wait for its state to reconcile, or whatever it is doing.
The only way to get Omni to proceed was to remove the machine from the Machines tab.

This should be an instantaneous action when scaling down the cluster, with a warning "Machine X is currently disconnected. Removing it from the cluster will require the machine to be unregistered from Omni. The machine will need to wiped and reinstalled to be used again with Omni. Proceed?"

Steps To Reproduce

Create a 3 node cluster.
Power off one node, wait for it to be "Disconnected"
Attempt to scale down the cluster by removing the disconnected node.

What browsers are you seeing the problem on?

No response

Anything else?

No response

[smira]

This flow has its issues - as the machine is not wiped this way, if we simply remove it from Omni, if the machine comes back, it will re-establish itself in the cluster (or it will at least try to).

That's why removing machines forcefully should be the last resort path, and action should be taken outside of Omni to wipe it properly.

[steverfrancis]

The current state of hanging the operation in perpetuity and blocking all other operations on the cluster (like patching) also has issues.

Agree this should be a last resort, but given it is the only path, putting in an Alert warning and actually offering the user a way forward seems correct.

[utkuozdemir]

We could improve the UX by eliminating the need to go to machines tab and deleting the disconnected machine from there, instead, allow "force-destroy" in a popup when removing a machine from a cluster.

[jd4883]

I have a very similar issue (same problem). This is my scenario (4 node cluster, 3 CP 1 worker, node-3 is the worker):

1. I pushed a change which due to a misconfiguration on node 4 (the wrong startup disk was setup so it blew out the disk. Expected but unfortunate. As a result node-04 is orphaned. I also removed it via kubernetes (kubectl delete node)
2. I removed it from the machines tab without issue, but it still shows as a member of the cluster.
3. I tried to scale down the cluster which was a huge mistake, hoping the orphaned node would drop. Node-02 got marked "Preparing to Destroy". Ideally I would like to cancel this operation but the UI gives me no option to do so. I should be ok on data loss with how my longhorn storage is setup, but obviously this would be much simpler to cancel the undesired operation. Cancel destroy fails with this message: Failed to restore machine set node 00000000-0000-0000-0000-0cc47ab01056 for going-merry: The resource creation failed with the error: failed to validate: 1 error occurred: * adding machine set node to the machine set "going-merry-control-planes" is not allowed: the machine set is using automated machine allocation
4. The working control plane (node-01) is having some intermittent connectivity which I think is related to etcd elections.
5. The commands below indicate the node-04 is still part of etcd and removing it should likely help but the omni bootstrapped talos cluster gives me permission errors:

% talosctl etcd members -n 172.20.10.101

NODE            ID                 HOSTNAME                 PEER URLS                                                                                                                                                                                                                                                               CLIENT URLS                                             LEARNER
172.20.10.101   133803d7bc6046cb   k8s-supermicro-node-01   https://172.20.10.101:2380,https://172.20.10.201:2380,https://172.20.12.1:2380,https://172.20.12.4:2380,https://172.20.12.6:2380,https://172.20.12.7:2380,https://172.20.14.103:2380,https://172.20.14.104:2380,https://172.20.15.253:2380,https://172.20.15.254:2380   https://172.20.10.101:2379                              false
172.20.10.101   43df374cbbce218d   k8s-supermicro-node-04   https://172.20.10.104:2380,https://172.20.10.204:2380                                                                                                                                                                                                                   https://172.20.10.104:2379,https://172.20.10.204:2379   false
172.20.10.101   800fc068a9d0eab0   k8s-supermicro-node-02   https://172.20.10.102:2380,https://172.20.10.202:2380,https://172.20.12.2:2380,https://172.20.12.8:2380,https://172.20.12.9:2380,https://172.20.14.102:2380                                                                                                             https://172.20.10.102:2379,https://172.20.10.202:2379   false
# attempting to remove it fails due to permissions
% talosctl etcd remove-member 43df374cbbce218d -n 172.20.10.101
1 error occurred:
	* 172.20.10.101: rpc error: code = PermissionDenied desc = not authorized

6. I can also see the node-01 behavior changing periodically with this command, node-02 and node-04 have been persistent in the same state:

% omnictl cluster status going-merry
Cluster "going-merry" SCALING_UP Not Ready (1/4) (healthy/total)
├── Kubernetes Upgrade Done
├── Talos Upgrade InstallingExtensions current machine k8s-supermicro-node-02
├── Control Plane "going-merry-control-planes" ScalingUp Not Ready (0/3)
│   ├── Load Balancer Not Ready
│   ├── Status Checks Failing: [Etcd]
│   ├── Machine "00000000-0000-0000-0000-0cc47ab01056" BEFORE_DESTROY
│   ├── Machine "00000000-0000-0000-0000-0cc47ab01498" RUNNING Not Ready
│   └── Machine "00000000-0000-0000-0000-ac1f6b8184e0" UNKNOWN Config Outdated
└── Workers "going-merry-workers" Running Ready (1/1)
    └── Machine "00000000-0000-0000-0000-0cc47ab0109e" RUNNING Ready

7. I also tried removing node-04 from the cluster view in the nodes section of the UI. I get Failed to Destroy The Node Cannot read properties of undefined (reading 'metadata') each time. It would be really nice if there was a force-destroy approach (CLI or UI) to fix this

What option do I have to remove so I can re-add node-04? Additionally, if there is any way I can force stop the attempt to remove node-02 (CLI or UI) that would really help. If I am missing something please point me in the right direction, I can open a new issue if this seems too different though I am fairly sure the problem is identical.

Thanks in advance,

Jacob

[flpajany]

Hello,

I have the same kind of a problem here when I tried to scale down a cluster in my OnPrem configuration. The node is rebooting to go to Maintenance Mode but it seems that omni is not taking that into account.

The cluster stays in this state indefinitly :

Regards,
Floris