Inconsistencies with MachinePools during Kubernetes upgrade

[MadJlzz]

/kind bug

What steps did you take and what happened:

Thursday 3rd of April, we suffered from 2 problems, one being a major disruption of our services during a MachinePool upgrade.

The initial goal was to upgrade "selfmanaged" BYO network clusters created through capi/capz from Kubernetes 1.29 towards Kubernetes 1.32

AzureMachinePool partial upgrade (preprod)

We started by performing multiple rolling upgrades of the workload cluster control plane, one version at a time.

No problems occurred and everything went smooth. We also deployed the associated Kubernetes version of cloud-provider-azure.

It all went bad when we started to run the upgrade on our MachinePool/AzureMachinePool objects. (k999azc-endor001)

To do so we've

• updated k999azc-endor001 MachinePool's spec.template.spec.version field to v1.32.0
• updated k999azc-endor001 AzureMachinePool's spec.template.image.computeGallery.version field to the name of the
  image we just build.

We're baking images our own way, not using image-builder as we feel it is still too complex to customize output images.

Here's the section of the AzureMachinePool that might be interesting

orchestrationMode: Flexible
strategy:
  rollingUpdate:
    deletePolicy: Oldest
      maxSurge: 1
      maxUnavailable: 0
  type: RollingUpdate

And a screenshot of clusterctl describe after applying the change

So far, so good, the VMSS scaled up to 4 instances (thanks to the maxSurge of 1), the VM got provisioned and the freshly created node joined the cluster. (we have 3 replicas for that pool)

And then, the rollout just stopped. It feels that capi or capz behaves like if all the machines were upgraded even though that's not true.

As a workaround, we ran multiple manual scale using kubectl until we reach the desired state.

# Scale up to deploy a machine with the new image reference.  
kubectl scale machinepool k999azc-endor001 --replicas=4 -n k999azc  
# Scale down and delete the oldest VM.  
kubectl scale machinepool k999azc-endor001 --replicas=3 -n k999azc  
# Rinse and repeat until all machines are upgraded.

Well, that was not smooth but no biggies there.

Upon debugging, I saw two things that felt weird to me, as a user when looking at the Azure Cloud console on the underlying VMSS:

• the scale-in-policy is set to Default even though the strategy of the AzureMachinePool is Oldest
• the upgrade policy is set to Manual

I am not sure if it's the VMSS that is responsible of rolling out all the machines of if capz is handling the process so that might be unrelated.

Also, we can reproduce this problem and we didn't saw anything useful/concerning from the logs.

AzureMachinePool upgrade goes bonkers (prod)

Since the upgrade went fine on a preprod cluster, we decided to perform it on production one but it didn't went as planned.

Also, sorry if this section might be obscure, we worked hard to have the shortest downtime so we weren't able to take as many notes.

We followed the same process described in the previous section but we forgot to delete azure-container-registry-config: /etc/kubernetes/azure.json from the kubeletExtraArgs in the underlying MachinePool's KubeadmConfig.

Because of that, the freshly provisioned node couldn't join the cluster because the kubelet flag --azure-container-registry-config being removed starting from Kubernetes 1.30 (if I recall correctly)

Then, because of that line removal we performed on the Kubeadm config, I believe capi triggered a second upgrade of the nodes.

Scaling by hand like we did created a bunch of problems such as the VMSS live replicas not being in sync with the spec.providerIDList of the MachinePool object - recreating every time ghost Machine and AzureMachinePoolMachine we were deleting. (ghost meaning nodes unexistant in AzureCloud but present in CAPI state)

The worse happened when we tried to scale to two replicas for that pool and that for an unknown reason, capi/capz scaled the VMSS to 1, degrading our stack drastically.

To solve the problem we had to:

• edit the Cluster object and set the reconciliation to be paused: true
• delete the ghost objects (MachinePool, AzureMachinePool ) and update the spec.providerIDList from the MachinePool object.
• synchronize the number of replicas of the MachinePool with the number we had on the VMSS on Azure Cloud
• we killed the pods inside capz-system in case there's in-memory cache mechanism
• we unpaused the cluster

Following those manipulations, new nodes were created and everything went back in order.

Environment:

Regarding the setup we're using:

• A GKE Kubernetes cluster

$ kubectl version
Client Version: v1.32.0
Kustomize Version: v5.5.0
Server Version: v1.31.6-gke.1020000    

• CAPI and CAPZ (deployed through the operator)

> kubectl get bootstrapproviders,controlplaneproviders,controlplaneproviders,infrastructureproviders -A
NAMESPACE                  NAME                                                  INSTALLEDVERSION   READY
kubeadm-bootstrap-system   bootstrapprovider.operator.cluster.x-k8s.io/kubeadm   v1.9.5             True

NAMESPACE                      NAME                                                     INSTALLEDVERSION   READY
kubeadm-control-plane-system   controlplaneprovider.operator.cluster.x-k8s.io/kubeadm   v1.9.5             True

NAMESPACE     NAME                                                     INSTALLEDVERSION   READY
capa-system   infrastructureprovider.operator.cluster.x-k8s.io/aws     v2.8.1             True
capz-system   infrastructureprovider.operator.cluster.x-k8s.io/azure   v1.17.4            True

[dtzar]

Just curious - have you tried doing the upgrade using the latest 1.19 version of CAPZ (versus as-shown 1.17.4)?

[MadJlzz]

I did not, I wanted to play safe and wait for #5410 to be fixed first before upgrading. I saw it was merged so if you do a v1.19.2 I can try to reproduce on my end

[nawazkh]

@MadJlzz could you please retry since #5410 has been fixed ?

[MadJlzz]

@nawazkh even after the upgrade the behaviour is similar, it does not rollout all the machines from the pool

[mboersma]

See also kubernetes-sigs/cluster-api#12392

[MadJlzz]

I've retested this morning an upgrade of a test cluster and found the root issue of my first point AzureMachinePool partial upgrade (preprod)

The problem comes from the fact that my VMSS is orchestrated in Flexible mode. Even though, I understand that it is technically possible for a Flexible scale set to orchestrate different machine type, machine image - I believe it's the responsability of capz to conduct an upgrade until the end though.

I have the feeling it might come from somewhere around here:

cluster-api-provider-azure/azure/services/scalesets/spec.go

Lines 119 to 129 in 1916969

	hasModelChanges := hasModelModifyingDifferences(&existingInfraVMSS, vmss)
	isFlex := s.OrchestrationMode == infrav1.FlexibleOrchestrationMode
	updated := true
	if !isFlex {
	updated = existingInfraVMSS.HasEnoughLatestModelOrNotMixedModel()
	}
	if s.MaxSurge > 0 && (hasModelChanges || !updated) && !s.HasReplicasExternallyManaged {
	// surge capacity with the intention of lowering during instance reconciliation
	surge := s.Capacity + int64(s.MaxSurge)
	vmss.SKU.Capacity = ptr.To[int64](surge)
	}

The first time we reconcile, hasModelChanges is true because the local model differs from the API state. So we run the update once. Next reconciliation loop, the model and the API results are equal since the VMSS was updated in the meantime. Since the VMSS is flexible, we do not check if all VMs model has changed - the capacity does not get updated which does stop the upgrade of the rest of the VMSS VMs.

Is this intended? I undestand the concept of Flexible orchestration mode - it's just that back in the days, I followed Azure's guidance, not thinking that this would have an impact on my upgrade strategy.

https://learn.microsoft.com/en-us/azure/virtual-machine-scale-sets/virtual-machine-scale-sets-orchestration-modes

I can reproduce the problem every time I am trying to perform a rolling updare so that might help.

[mboersma]

@MadJlzz and I talked about this one and think there's something to fix. Let's see if we can add an e2e test that covers VMSS flex + upgrade as a starting point.