Canonical K8s MAAS cluster creation with CAPI gets stuck when default CNI is set to false

[vishu2498]

Issue:

When deploying a Canonical K8s MAAS cluster using CAPI, if the default CNI option is set to false, the managment cluster is forever waiting for the bootstrap cluster to get created and arrive in running state since the CNI needs to be installed manually on the MAAS instance.

Steps to reproduce:

An option is present to set default CNI to false as shown here:

spec:
  initConfig:
    enableDefaultNetwork: false

Reference: https://documentation.ubuntu.com/canonical-kubernetes/latest/capi/reference/configs/#initconfig

Create a cluster using this configuration in CK8ControlPlane object and the issue will be reproducible.

Expected Behaviour:

The bootstrap cluster inside MAAS instance should be up and running and setting default CNI to false should not impact the boostrap cluster. Later when the bootstrap cluster is up (possibly using default CNI), user/CAPI can install their own CNI on the created cluster.

Actual Behaviour:

Inside the config used by script, it disables the CNI installation (cilium CNI installation). This methods works correct when deploying with k8s-snap package, but it fails via the CAPI method.

The bootstrap cluster is created without CNI and is stuck with only these pods:

NAMESPACE     NAME                              READY   STATUS    RESTARTS   AGE
kube-system   coredns-fc9c778db-bhqwc           0/1     Pending   0          11s
kube-system   metrics-server-8694c96fb7-6b79j   0/1     Pending   0          11s

At this stage, k8s status doesn't show ready status and stays the same.

From the CAPMAAS side, it only proceeds if it is able to create a client to target cluster (using kubeconfig secret) and is able to list the nodes (only then it confirms that API server is online).

It waits with logs like this:

I0717 05:05:24.427993       1 maasmachine_controller.go:353] "API Server is not online; requeue" logger="controllers.MaasMachine" maasmachine="vishu-maas-1-cp-61386-8vvln" cluster="vishu-maas-1" state="Deploying" m-id="aqhn73"

Now since the bootstrap cluster inside the MAAS instance has CNI disabled, there is nothing from management side that can be done to get this cluster up. As per the documentation, the k8s cluster deployed with snap also needs to disable CNI first and then manually apply alternative CNI. However, CAPI can’t proceed here because user first needs to SSH inside the instance, access the k8s cluster and manually install CNI for the process to go through.

[ader1990]

Hello,

When used with cluster api tooling like Sylva (https://sylva-projects.gitlab.io/), the CNI installation is managed using the CAPI workflow, using either direct kubectl apply if the cluster is reachable or by using https://github.com/canonical/cluster-api-k8s post run commands available to install kube-vip and thus making the cluster reachable. Then, it is a matter of user commands to install the CNI and then, the CAPI control plane see the cluster as ready.

This discussion is more over-arching and has been discussed on several other threads like kubernetes-sigs/cluster-api#11766 and the question is: What is a Ready cluster? For example, kubeadm clusters somehow cheat the workflow and mark the cluster ready via the Node Ready in an optimistic non-realistic fashion, and after a few minutes the nodes revert to NotReady State if there is no installation of the CNI via other methods (manual or automated by cloud-init post run commands). But the cluster api sees the Node Ready and completes the workflow. The https://github.com/canonical/cluster-api-k8s workflow is not the same as the kubeadm's workflow. The https://github.com/canonical/cluster-api-k8s workflow is to do nothing and not mark the cluster node as Ready, thus the issue arising.

[vishu2498]

@ader1990 I understand your point of view and the way Canonical K8s CAPI has been differentiated from the kubeadm workflow.

However, isn't keeping EnableDefaultNetwork as option in CK8sConfigSpec irrelevant then? What it eventually does is when the option is given as false , it adds the field in the config yaml for the bootstrap cluster and the scripts triggered by CAPI CK8s use this config to bootstrap (create) the cluster. If it keeps the cluster stuck forever from the CAPI, that doesn't amount to a good user experience as anyone would not know the internals until they read through the documentation.

As per how the cluster can be deployed using snap, the documentation provides an option to use an alternative CNI by disabling the network (either at time of cluster creation or updation). This can be done manually from CAPI side too but it can introduce race conditions.

While user is expected to install CNI manually in case of any CAPI cluster, for the CK8s counterpart, the bootstrap cluster inside the machine should be up and running regardless of how user configures default network option. Following this, the CAPI stack on management cluster with CAPMAAS will be able to connect to the cluster and then let users install their CNI. The crux of the idea being that bootstrap cluster inside machine should be able to get into ready state and management cluster CAPI can go through the further processes (even when user disables or enables EnableDefaultNetwork).

[HomayoonAlimohammadi]

If I understand this correctly, a Canonical K8s cluster is considered "not ready" if there's no CNI installed. So, setting enableDefaultNetwork: false prevents the bootstrap cluster from reporting ready, until a CNI is manually installed on it. This is in fact, by design. I believe the workaround presented by @ader1990 about installing kube-vip should resolve this issue, unless there's a feature request for natively supporting this in Canonical K8s CAPI.

[vishu2498]

Hi @HomayoonAlimohammadi . Thanks for your insight on this!

However, even by design that the CNI installation should be done manually, the ready status from the bootstrap cluster should be returned as true so that CAPI can proceed ahead and install any CNI that the user wants to have.

We haven't explored the kube-vip option, but we would discuss this and probably create a feature request for this scenario in Canonical K8s CAPI.