microk8s enable ha-cluster fails

[mshlain]

ha-cluster-upgrade.log
inspection-report-20251216_075926.tar.gz

Summary

The command 'microk8s enable ha-cluster' fails and corrupts the whole cluster

What Should Happen Instead?

Calico CNI should be enabled

Reproduction Steps

I have a single-node microk8s cluster that is running few applications
When i installed it back in 2023 i used 'microk8s disable ha-cluster' because we did not need the full-blow calico CNI and flannel was enough.
We also installed coredns from helm
Today we need calico CNI

I did:

1. Drain with:
   kubectl delete nodes --all
   kubectl delete pods --all --all-namespaces
   (no errors returned)

2. Execute: microk8s enable ha-cluster wich printed:

microk8s enable ha-cluster
Infer repository core for addon ha-cluster
No resources found
Enabling HA
Upgrading the network CNI
Waiting for the cluster to restart

3. After

Introspection Report

Inspecting system
Inspecting Certificates
Inspecting services
  Service snap.microk8s.daemon-cluster-agent is running
  Service snap.microk8s.daemon-containerd is running
  Service snap.microk8s.daemon-kubelite is running
 FAIL:  Service snap.microk8s.daemon-flanneld is not running
For more details look at: sudo journalctl -u snap.microk8s.daemon-flanneld
  Service snap.microk8s.daemon-etcd is running
  Service snap.microk8s.daemon-apiserver-kicker is running
  Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
  Copy processes list to the final report tarball
  Copy disk usage information to the final report tarball
  Copy memory usage information to the final report tarball
  Copy server uptime to the final report tarball
  Copy openSSL information to the final report tarball
  Copy snap list to the final report tarball
  Copy VM name (or none) to the final report tarball
  Copy current linux distribution to the final report tarball
  Copy asnycio usage and limits to the final report tarball
  Copy inotify max_user_instances and max_user_watches to the final report tarball
  Copy network configuration to the final report tarball
Inspecting kubernetes cluster
  Inspect kubernetes cluster

Building the report tarball
  Report tarball is at /var/snap/microk8s/8511/inspection-report-20251216_075926.tar.gz

[mshlain]

I have found somehow strange workaround: tried to apply few times, then applied the cni resource manually then tried to apply the addon one more time and it sompleted successfully:

myadmin@debian:> microk8s enable ha-cluster
Infer repository core for addon ha-cluster
No resources found
Enabling HA
Upgrading the network CNI
Waiting for the cluster to restart
myadmin@debian:> microk8s status
microk8s is not running. Use microk8s inspect for a deeper inspection.
myadmin@debian:> microk8s enable ha-cluster
Infer repository core for addon ha-cluster
Enabling HA
Upgrading the network CNI
Waiting for the cluster to restart
myadmin@debian:> microk8s status
microk8s is not running. Use microk8s inspect for a deeper inspection.
myadmin@debian:> microk8s enable ha-cluster
Infer repository core for addon ha-cluster
Enabling HA
Upgrading the network CNI
Waiting for the cluster to restart
myadmin@debian:> sudo cp /var/snap/microk8s/8511/args/cni-network/cni.yaml .
myadmin@debian:> sudo chown myadmin:myadmin ./cni.yaml 
myadmin@debian:> kubectl apply -f ./cni.yaml 
poddisruptionbudget.policy/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
serviceaccount/calico-node created
serviceaccount/calico-cni-plugin created
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgpfilters.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrole.rbac.authorization.k8s.io/calico-cni-plugin created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-cni-plugin created
daemonset.apps/calico-node created
deployment.apps/calico-kube-controllers created
myadmin@debian:> microk8s enable ha-cluster
Infer repository core for addon ha-cluster
Enabling HA
Upgrading the network CNI
Waiting for the cluster to restart
Waiting for the CNI to deploy
Waiting for daemon set "calico-node" rollout to finish: 0 of 1 updated pods are available...
daemon set "calico-node" successfully rolled out
Waiting for deployment "calico-kube-controllers" rollout to finish: 0 of 1 updated replicas are available...
deployment "calico-kube-controllers" successfully rolled out
Configuring the datastore
Waiting for the cluster to start with the new datastore

The node is now HA ready. Join three or more nodes to form an HA cluster.

[louiseschmidtgen]

Hello @mshlain,

Thank you for reporting your issue.

Unfortunately, your set-up diverges from the common supported use case.

I may offer you some pointers and guidance to migrate your cluster to calico:

Under the hood ha-enable runs a script 000-switch-to-calico which is defined here.

You can try to manually invoke this script to upgrade your cluster to calico. For supported upgrade paths it is best to stay as close as possible to the default configuration for a seamless upgrade experience.

Best regards,
Louise