Automatic upgrade and statefulset workloads

[smokfyz]

Description

Hello! There is limited information in the documentation regarding how node auto-upgrades function when using a database in Kubernetes or Longhorn. Could you please explain how auto-upgrades work in such a scenario? Just disable it or how?

[mysticaltech]

@smokfyz For k3s auto upgrades, nothing happens basically, the binary is replaced without even kube going down. Now for node upgrades, that's more invasive. Node upgrades provoke the nodes to be drained and uncordoned from the cluster for reboot, one after another, so super important to be in HA.

This I guess would be problematic unless you have longhorn correctly setup in an HA fashion, if you are in HA it should be seemless IMHO as longhorn would have distributed the data in duplicates across the cluster. Same for the statefulset it should not be a problem if configured in an HA fashion. But I must admit am not an expert in working with them.

@kube-hetzner/core Please correct me if I'm wrong 🙏

[mysticaltech]

Here's what I was able to get out of GPT-4:

Node upgrades and the management of StatefulSets in a High Availability (HA) Kubernetes cluster, particularly with automatic node upgrades, present a complex but manageable scenario. Here are some thoughts and considerations:

1. Importance of HA in Node Upgrades: In an HA cluster, automatic node upgrades are less likely to cause service disruptions. HA setups typically involve multiple replicas of nodes and pods, ensuring that even if one node is taken down for an upgrade, the others can continue to handle the workload. This redundancy is crucial for maintaining service availability.

2. Challenges with StatefulSets: StatefulSets, which are used for managing stateful applications (like databases), have their own complexities. They maintain a sticky identity for each of their pods. During node upgrades, care must be taken to ensure that the state (like persistent data) is not lost or corrupted. This requires a well-thought-out data replication and backup strategy.

3. Graceful Handling of Pod Eviction: During automatic node upgrades, pods are evicted as nodes are drained. For StatefulSets, Kubernetes tries to reschedule the pods to other nodes while maintaining their identity and connection to the right data volumes. Ensuring that other nodes have the capacity and configuration to accept these pods is essential.

4. Storage Considerations with Longhorn or Similar Solutions: If you're using a distributed storage solution like Longhorn, it's crucial to ensure that your storage layer is also highly available and resilient to node failures. Data should be replicated across multiple nodes so that if one node goes down during an upgrade, the data remains accessible.

5. Testing and Monitoring: Before implementing automatic upgrades in production, it’s vital to test the process in a staging environment. Monitoring tools should be in place to quickly identify and address any issues that arise during the upgrade process.

6. Version Compatibility and Rollback Plans: Ensure that the new versions of Kubernetes or other components are compatible with your existing workloads. Have a rollback plan in case the upgrade leads to unexpected issues.

7. Coordination and Scheduling: While automatic upgrades are convenient, it might be beneficial to have some control over the scheduling to ensure upgrades occur during low-traffic periods, thereby minimizing potential impacts.

In summary, while automatic node upgrades in an HA Kubernetes cluster offer many benefits in terms of reducing manual overhead and ensuring up-to-date systems, they require careful planning, robust infrastructure setup (especially for stateful applications), and thorough testing to ensure smooth operations without significant service disruptions.

[aleksasiriski]

@smokfyz If you're using Longhorn I suggest disabling automatic upgrades since there's a bug in Longhorn preventing the nodes to fully drain and this bug leaves the nodes cordoned meaning you're paying for resources that aren't being utilized. I'm currently manually updating my nodes once per week until the bug is fixed.

On the other hand, cloudnative-pg works flawlessly with automatic upgrades. Just ensure you have backups and WAL on an external S3 bucket if something goes wrong.

[valkenburg-prevue-ch]

I would like to add that if you use Hashicorp Vault together with Longhorn storage, there are some caveats too.

Vault uses a pod disruption budget on the persistent volumes which is configured such that its pods cannot be evicted unless another three replicas would remain after the eviction. So you need 4 replicas. But on top of that, once a pod was evicted, the node updated, and the pod is rescheduled, this Vault pod does not become ready until you unseal it. As a consequence, until you unseal it, the three remaining replicas cannot be evicted, and the update process comes to a halt (and one node remains cordoned, waiting for the update process to continue).

All in all, for me it only works if you have 4 vault replicas and auto-unseal configured, so the pods come online again after rescheduling and the next vault pod can be evicted for the update process.

It took me quite a while to understand this. The key to understanding why a note is stuck on cordoned state, is to check the logs of the kured instance on that machine. Use kubectl describe node <the cordoned node>, to find which is the kured pod on that node, and then get its logs. You get things like

evicting pod longhorn-system/instance-manager-8b782478248407465761290b23d085d3
error when evicting pods/"instance-manager-8b782478248407465761290b23d085d3" -n "longhorn-system" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.

[mysticaltech]

@smokfyz FYI.