Canonical K8s CAPI should have an option for control-plane only role

[vishu2498]

Issue:

Citing the documentation, when deploying any Canonical K8s cluster using CAPI, the default behaviour is that the bootstrap (creation) process starts the cluster where the control-plane nodes have both the roles:

• control-plane
• worker

By going through the design, it is understandable that CK8s was made for single-node clusters too where a single control-plane node can handle both the cluster's core and workloads.

However for certain scenarios (even mentioned in the production usage), there can be various conditions where user just wants the control-plane to handle cluster's core and shift the workloads to the worker nodes. In this case, since the control-plane also has the worker role, the workloads will also get scheduled on control-plane node. It was noticed in our testing where some pods got created on control-plane node and went in CrashLoopBackOff state since the ports they wanted to use were already bind by some of the core pods on control-plane node.

In the documentation, a workaround for this scenario is suggested where user can taint the control-plane nodes (after cluster creation) and results will be the expected ones.

However, this process works fine for a cluster created from snap but there is no option in API fields where CAPI detects this and taints the control-plane node(s) by itself. The solution can either be tainting control-plane nodes after cluster creation (this can introduce race conditions) OR to mark control-plane role only for multi-node cluster and control-plane, worker roles for single-node cluster (which is the default behaviour currently) for control-plane node.

Steps to reproduce:

Create a single-node or multi-node cluster using Canonical K8s CAPI and the issue will be reproducible.

Expected Behaviour:

For atleast a multi-node cluster, the control-plane node should have control-plane role only and worker node should have worker role only.

There must be an API field for CK8s where user can mention that they want worker role to be present on control-plane node or not.

Actual Behaviour:

Output shows that control-plane node holds both control-plane & worker role for a multi-node cluster:

root@small-4x8-lxd-003:/home/ubuntu# kubectl get nodes
NAME         STATUS   ROLES                  AGE   VERSION
bm-az1-142   Ready    control-plane,worker   26h   v1.32.6
bm-az3-32    Ready    worker                 26h   v1.32.6

[HomayoonAlimohammadi]

Hi @vishu2498 and thanks for bringing this up.
I believe there's a nodeTaints option in the cluster template (ref) that you can use. It should look like the following:

apiVersion: controlplane.cluster.x-k8s.io/v1beta2
kind: CK8sControlPlane
metadata:
  ...
spec:
  ...
  spec:
    controlPlane:
      nodeTaints: ...

[vishu2498]

Hi @HomayoonAlimohammadi . Thanks for responding!

Yeah we understand that adding nodeTaints is a workaround to resolve the issue. With this, the workloads would start running on worker nodes and not on control-plane nodes but the control-plane node would still have worker role (as shows in example).

By design, there should be an API field which would restrict the control-plane to only have control-plane role and work accordingly.