PSA: Using cilium install vs. other methods brings you sadness when later adding new Control Plane nodes

[boldandbusted]

Howdy. I've got a homelab with Talos Linux and it's fun. I'm learning new things, and breaking things in interesting ways. 🥇

I'm learning that if you want Cilium CNI, and you want to later scale your control planes (adding new control plane nodes), you'll probably not want to use the Cilium CLI to install itself in the first place. Why? Because Talos won't have the correct manifests to install and configure Cilium on newly added Control Plane hosts.

It is easier to just run 'cilium install' - and I wish that this was better supported. However, that ease at the beginning of your journey are paid later when you need to scale.

I think the hints left in the Talos Linux's Cilium docs should be well-heeded. :) Cheers!

[smira]

I'm not quite sure what you're talking about. Scaling the cluster up/down is handled by Kubernetes/Cilium on its own, as proper components are scheduled to run e.g. on all nodes.

[boldandbusted]

Hello Andrey. Thank you for Talos.

Perhaps I didn't explain my scenario well.

I have a Talos K8s cluster with 1 control plane node and 3 worker nodes, bare-metal. I used the Cilium CLI to install Cilium, as shown in the Talos docs. Works fine.

Now, I wanted to take 2 of the Worker nodes and make them control plane nodes. I talosctl reset them, booted them to Talos maintenance mode, and then followed the general instructions for adding new control plane nodes, using apply-config from the original controlplane.yaml file plus patch files for handling removing the CNI used to install Cilium. Then I tried cilium install again. However...

Those new control plane nodes never came up Ready. But perhaps I missed something?

[smira]

yes, you did something wrong, but it's not clear what exactly. there are no special steps required to make Cilium "scale" to new controlplane or worker nodes. (you only cilium install once for the cluster lifetime).

[boldandbusted]

Thank you for the feedback! I will try again in a day or two. If I encounter the problem again, I would like to give you a good problem description, rather than the vague description I have given you thus far.