Proposal: provider-owned k3s/RKE2 bootstrap (first-class, not a cloud-init recipe)

[manarhusrieh]

Problem

Today Talos and Ubuntu/kubeadm are first-class (imageSelector.family), while
k3s is a documented recipe: the user hand-writes cloud-init userData that
installs the agent and — critically — must also hand-render Karpenter's
registration contract:

• the node labels core's scheduler assumed when it picked the type
  (karpenter.sh/nodepool, node.kubernetes.io/instance-type,
  karpenter.sh/capacity-type, zone/region),
• the karpenter.sh/unregistered:NoExecute taint,
• kubelet args consistent with the provider's advertised allocatable.

Every k3s user re-derives this by hand, and the failure modes are silent:
miss the taint and pods schedule onto a node core hasn't bound yet; miss a
label and the Node doesn't match what scheduling simulated. Worse, the
contract is versioned — it tracks karpenter core, not the user's cluster — so
hand-rolled userData rots quietly across upgrades. A recipe puts the most
correctness-sensitive part of the provider in the least-maintained place.

The concrete case motivating this: your k3s recipe already calls kube-hetzner
"the natural path", and I run kube-hetzner in production. I'd like to see this
provider become integrable there as an optional addon, the way that module
already integrates cluster-autoscaler (module deploys the chart, templates the
NodeClass from its own snapshot/network/firewall/token resources), and I
intend to propose exactly that upstream once the bootstrap story supports it.
The blocker is precisely the recipe model: for that module to fill userData
it would have to template Karpenter's labels/taint contract into Terraform —
i.e. duplicate provider logic downstream and re-couple it to karpenter
versions. With provider-owned bootstrap the module shrinks to passing refs
(join endpoint, token Secret, snapshot selector, plus an extra-userdata hook
for its node-prep), and the version coupling stays where it belongs: in this
repo. That unlocks the provider for the kube-hetzner install base with a
supported integration rather than N hand-rolled cloud-inits.

Proposed solution

An orthogonal bootstrap: block on the NodeClass (distribution: k3s|rke2,
server, tokenSecretRef, extraConfig), composing with any
imageSelector — k3s runs on Ubuntu or custom snapshots (kube-hetzner uses
MicroOS), so "what disk image" and "how it joins" stay separate concerns.
Talos and kubeadm paths remain untouched.

Whatever the final shape, the provider-owned bootstrap should guarantee:

1. Rendered join config (k3s: /etc/rancher/k3s/config.yaml; RKE2:
   /etc/rancher/rke2/config.yaml) with server URL and token, token sourced
   from a Secret ref only — never inline in the CR. (Worth documenting
   regardless: userData is readable from the metadata service inside the
   server, so recommending the k3s agent token over the cluster token.)
2. Node labels + the unregistered taint rendered by the provider, per
   NodeClaim, so they can never drift from what core simulated.
3. kubelet settings (e.g. maxPods) consistent with advertised allocatable.
4. User extension that composes instead of clobbers: structural merge of
   extra userData (list-valued keys like runcmd/write_files unioned,
   provider entries first) — a verbatim append lets a user runcmd: silently
   replace the install commands and the node never joins.
5. Deterministic rendering (stable label ordering etc.), since userData
   inputs presumably feed drift hashing.

Full disclosure on where this comes from: I built and tested a provider in
parallel earlier this year and found this project on the day I was going to
publish — you were first and further along, so I'd rather contribute than
fragment. From that work I have a tested k3s+RKE2 userdata renderer
(deterministic output, structural merge, tests asserting the label/taint
contract) built against karpenter v1.14.0, which I'm glad to adapt to your
conventions and submit — design discussion first, then a PR, DCO signed.

Alternatives considered

• New image families k3s / rke2 next to talos/ubuntu. Fits the
  existing imagefamily resolver, and I'm happy to implement it this way if
  you prefer — the renderer slots in underneath either shape. Downside: it
  conflates the OS image with the join method, and the two pull apart exactly
  in the kube-hetzner case (k3s on a MicroOS snapshot).
• Keep bootstrap out of the provider; maintain a contrib example that
  downstream tools like kube-hetzner can template. Workable but weaker: the
  label/taint contract still gets duplicated downstream and re-coupled to
  karpenter versions — I'd argue contract items 1–5 above are provider
  concerns wherever the rendering lives.
• Status quo (recipe only). Every k3s/RKE2 adopter keeps hand-rolling the
  registration contract, with silent failure modes and per-user drift as core
  evolves.

[stubbi]

Thanks for this — it's a well-argued proposal, and the core claim is one I agree with: the registration contract (nodepool/instance-type/capacity-type/zone labels, the unregistered:NoExecute taint, kubelet settings consistent with advertised allocatable) is provider logic that tracks karpenter core versions, and asking every k3s user to hand-render it in cloud-init is the worst place for it to live. The kube-hetzner integration story is compelling too — "module passes refs, provider owns the contract" is the right division of labor.

On the shape: I prefer your primary proposal (an orthogonal bootstrap: block) over new k3s/rke2 image families. The MicroOS case makes the argument by itself — "which disk image" and "how the node joins" are separate axes, and conflating them means every (image × join-method) pair becomes a new family. Talos stays a family since there the image is the join method.

A few design constraints I'd want the discussion to settle before a PR:

1. Validation of the cross-product. bootstrap: must be rejected (CEL or webhook, at admission) when combined with an image family that can't honor it — bootstrap: k3s + imageSelector.family: talos should fail loudly, not produce a server that never joins.
2. Merge semantics need to be specified, not just "structural". Union for runcmd/write_files with provider entries first sounds right; the spec should say what happens on scalar-key conflicts (user overrides? error?) and pin it with tests, since silent clobbering in either direction is the failure mode you're designing away.
3. Drift hashing. You already flagged deterministic rendering; the other half is deciding which bootstrap inputs participate in the NodeClass hash so a token Secret rotation doesn't roll the fleet but a changed server URL does.
4. Token handling. Secret-ref-only, agent token recommended, documented as metadata-service-readable — all agreed; the provider will need RBAC to read that Secret, so scope it (namespace-local reference, not arbitrary).
5. Allocatable consistency (your item 3) intersects with in-flight work on computing allocatable from real reservations (fix: compute node allocatable from real reservations and usable memory #55, which adds a descriptive spec.kubelet). With a provider-owned bootstrap that field can become prescriptive for k3s/RKE2 — the provider renders the reservations it advertises. Worth designing them to converge rather than shipping two overlapping kubelet knobs.

Given you have a tested renderer already: a short design write-up in this issue covering the points above (schema sketch + merge-semantics table is plenty, no need for a formal doc) and then a PR sounds like the right sequence. One heads-up on timing: we're currently standing up e2e infrastructure for the provider, and I'd want that in place before merging a new bootstrap path so it lands with real cluster coverage rather than unit tests alone — design discussion doesn't need to wait on that, though.