apply-config from http/https?

[darkpixel]

I have been playing around with Talos for a few months now and love it.
Nodes are set up to PXE boot passing talos.config=http://10.74.2.253/talos/${mac}/ to get their config.

The app that's listening for http requests looks at the MAC address and hands out a templatized config file that determines if the node is a controlplane or worker node, what region/zone it's in, and apply a few other metedata labels (like if it has spinning rust, SSD, or NVMe storage).

After making changes to the config, it's a bit of a hassle to get all the nodes updated. I basically have to curl that file on behalf of the node, and then apply it.

i.e. this:

for node in $(kubectl get nodes -o wide | grep -v ROLES | awk '{ print $6 }'); do
  for mac in $(talosctl -n $node get link | grep -v down | grep -v veth | grep -v bridge | grep -v flannel | grep -v loopback | grep -v KIND | awk '{ print $7 }'); do
    echo $node - $mac; curl -s http://10.74.2.253/talos/${mac}/ | talosctl apply-config -n $node -f -;
  done ;
done

Hacky as heck.

I'm not sure if the nodes "remember" their talos.config setting after installing, but I'd love to simply call talosctl apply-config and have them re-poll that address.

Or at the very least be able to call talosctl apply-config -u "http://10.74.2.253/talos/${mac}/" and have them pull their updated config.

The slightly less risky way to update their config is that I simply talosctl reset -n 1.2.3.4 --wipe-mode all --reboot node by node and let them update/reinstall while shuffling workloads around. That's somewhat excessive for small config changes that don't need a reboot. :)

[rothgar]

Kernel args and remote Talos config are meant for bootstrapping and not long term maintenance or upgrades. Since the bootstrapping application is passive it doesn't seem like it is able to directly call the talos API to update the configuration automatically which is probably the best place to put this logic and lifecycle management.

Your script looks exactly how I would implement this logic from the client side.

This is exactly why we built Omni. It manages the patches for you, and automatically applies them to the machines when an update happens.

[darkpixel]

I get that. I just wish there was a way to tell the nodes to 'pull' a config during runtime instead of a push.
Pretty much every machine needs a "one off" patch for various attributes and it can become a mess keeping everything straightened out. A templatized config makes this a non-issue...there just no way to easily automate it.

You can see my script is hacky. Change one minor bit of output from talosctl and the thing will blow up.
Heck, a brief network timeout will probably cause unexpected results too. :)

[rothgar]

There are other community templating options (eg talhelper) and generally gitops-like flows which could accomplish similar things. Check out https://github.com/siderolabs/awesome-talos/ for some ideas.

From our experience, nodes should not be responsible for pulling their own configuration. You'll always want something outside the cluster (e.g. a human running a script) because individual machine configuration can break machines or external factors (e.g. network) cause other faults.

The same is true for other systems like Kubernetes. Kubernetes deployments should never be able to directly subscribe/update themselves if an upstream manifest changes. There should always be some "external" controller that understands the full state of the application and makes calls to the k8s API.