Problem while upgrading existing cluster from 1.32 to 1.33

[quantr001]

/kind bug

What steps did you take and what happened:
I have an existing cluster running k8s 1.32 deployed roughly a year ago, and now I want to upgrade it to 1.33. My upgrade procedure involves:

• Create a new VSphereMachineTemplate based on the existing one, with new name and template image.
• Manually edit KubeadmControlPlane's .spec.machineTemplate.infrastructureRef.name and .spec.version to point to the new machine template and k8s version. I also edited kube-vip version from 0.6.4 to 0.8.10 since I have been having a lot of problems with 0.6.4.
• Wait for the new nodes to be rolled out. However, the new node is stuck in Provisioned state while capi-kubeadm-control-plane-controller-manager complains "Waiting for control plane to pass preflight checks to continue reconciliation".
• On the new node, kube-apiserver's log only contains one line similar to this From cluster create template Investigate and remove cloud-provider flag for kube-apiserver due to removal in v1.33 cluster-api-provider-ibmcloud#2224.
• I tried to remove the extraArgs in the KubeadmControlPlane and rollout again, this time the new node's kube-apiserver doesn't complain about cloud-provider argument anymore, however there are still errors in during startup. After some manual restart of the kube-* pods by moving manifests out of and back into /etc/kubernetes/manifests, I managed to get those pods to start without errors, but capi-kubeadm-control-plane-controller-manager still complains "Waiting for control plane to pass preflight checks to continue reconciliation".

What did you expect to happen:
New nodes to be rolled out properly.

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• Cluster-api-provider-vsphere version:

$ clusterctl upgrade plan
Checking if cert-manager needs upgrade...
Cert-Manager will be upgraded from "v1.16.2" to "v1.19.3"

Checking new release availability...

Latest release available for the v1beta1 Cluster API contract version:

NAME                     NAMESPACE                           TYPE                     CURRENT VERSION   NEXT VERSION
bootstrap-kubeadm        capi-kubeadm-bootstrap-system       BootstrapProvider        v1.9.4            v1.10.10
control-plane-kubeadm    capi-kubeadm-control-plane-system   ControlPlaneProvider     v1.9.4            v1.10.10
cluster-api              capi-system                         CoreProvider             v1.9.4            v1.10.10
infrastructure-vsphere   capv-system                         InfrastructureProvider   v1.12.0           v1.15.2

The current version of clusterctl could not upgrade to v1beta1 contract (only v1beta2 supported).


Latest release available for the v1beta2 Cluster API contract version:

NAME                     NAMESPACE                           TYPE                     CURRENT VERSION   NEXT VERSION
bootstrap-kubeadm        capi-kubeadm-bootstrap-system       BootstrapProvider        v1.9.4            v1.12.3
control-plane-kubeadm    capi-kubeadm-control-plane-system   ControlPlaneProvider     v1.9.4            v1.12.3
cluster-api              capi-system                         CoreProvider             v1.9.4            v1.12.3
infrastructure-vsphere   capv-system                         InfrastructureProvider   v1.12.0           v1.15.2

You can now apply the upgrade by executing the following command:

clusterctl upgrade apply --contract v1beta2

• Kubernetes version: (use kubectl version): v1.30.1+vmware.1-fips (tanzu)
• OS (e.g. from /etc/os-release):

$ cat /etc/os-release 
PRETTY_NAME="Ubuntu 24.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.4 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo

[fabriziopandini]

At first sight the doesn't seems a CAPI/CAPV problem but more a configuration / environment problem that leads to control plane components to not start properly. You have to check those components inside the workload cluster.

Might be you can also take a look at artifacts from our CI test that is performing 1.32 to 1.33 upgrade (e.g. compare your KCP with the one in this test): https://testgrid.k8s.io/cluster-api-provider-vsphere-main#periodic-e2e-govmomi-main-1-32-1-33

[quantr001]

I have decided to delete the whole cluster and redeploy from scratch instead of upgrading. I deployed with 3 control nodes and 5 worker nodes, however in the first volley, only 1 control nodes was deployed. However all the machines are stuck at Provisioned phase instead of transitioning to Running phase, even though I can access the new cluster. Once I deployed calico on the new cluster, all worker nodes now only had one taint node.cluster.x-k8s.io/uninitialized:NoSchedule.

Do you know if there was any changes between k8s 1.32 and 1.33+? I'm aware that I'm on an old version of cluster-api (1.9.4) but until recently I was able to deploy new k8s 1.32 clusters. Also, do you know which script is run to remove that taint, and does it leave any kind of log behind?

EDIT: It seems my old version of cluster-api doesn't support version 1.33+ (according to this), so I will attempt to upgrade cluster-api version. My question is, what will happen to the existing v1beta1 objects? Will vsphere-provider controller update those to v1beta2 automatically?

[fabriziopandini]

Good catch, I did not payed attention to the version you were running.

WRT to you question about migrations, you can find a few threads in the CAPI channel answering to this question, the last one that I'm aware of is https://kubernetes.slack.com/archives/C8TSNPY4T/p1772567300348209 (CAPI leverage API machinery for version management, so it behaves like any other system upgrading CRDs)