Two ways to run Kubernetes on apple/container — Talos (embedded etcd) and k3s (external-etcd HA)

[BinHsu]

Apple's container runs each container as its own micro-VM with its own vmnet IP. That's an unusual base for Kubernetes — one real VM per node, no shared kernel, no Docker socket. I wanted to know what it can actually host, so I built two small provisioners on it and ran them on hardware. Sharing what the substrate allows, and the landmines.

• Talos Linux — immutable OS, embedded etcd: https://github.com/BinHsu/aegis-apple-container-provisioner-talos
• k3s (k3ac) — external etcd quorum + haproxy HA: https://github.com/BinHsu/aegis-apple-container-provisioner-k3s

Both boot one micro-VM per node and form a real multi-node cluster with per-node IPs. They are proven spikes, not maintained tooling — the code is the receipts (each repo carries an ADR trail and a first-person hardware verification log). Credit where due: @saiyam1814's kiac already explored the kubeadm path; these are the Talos and k3s data points alongside it.

Where the two diverge — HA, and it's the interesting part:

• Talos uses embedded etcd. The provider patches the control-plane endpoint to the real DHCP address before the node boots, so etcd binds the correct IP from the start.
• k3s has no such pre-boot seam. --cluster-init forms the cluster after boot, and embedded etcd's peer membership is IP-bound — cold-restart the VMs, vmnet hands out new DHCP IPs, the members look for peers at the old addresses, and quorum never reforms. So k3ac uses k3s's pluggable external datastore: a 3-node etcd quorum (mutual TLS) addressed by FQDN, fronted by an haproxy L4 load balancer. The cluster reconnects by name after any DHCP shift, and it adds day-2 ops (snapshot/restore, rolling upgrade/rollback, cert/token rotation), all behind a -force dry-run gate.

Substrate gotchas (all verified on hardware), shared by both lanes:

• container cp faults the vsock during k3s/Talos cold-boot image extraction → deliver the kubeconfig by host bind-mount instead.
• A control-plane node needs real memory headroom; the apiserver can be OOM-killed during the boot window at tight limits.
• MetalLB L2 VIP is host-reachable here — gratuitous ARP works on vmnet, where the qemu path drops it ([macOS] Support socket_vmnet / vmnet-bridged networking for QEMU provisioner siderolabs/talos#12834).

The Talos one I pitched upstream; it was declined on principled grounds (siderolabs/talos#13587), so it stays as a dev/CI substrate.

Write-ups — the DHCP traces, the ADRs, the maintainer's reasoning:

• k3s: https://binhsu.github.io/k3s-clusters-apple-container-k3ac/
• Talos: https://binhsu.github.io/a-native-talos-provider-for-apple-container/

Curious whether a first-party Kubernetes story (discussion #1673) lands on k3s or something else.

[afbjorklund]

That's an unusual base for Kubernetes — one real VM per node, no shared kernel, no Docker socket.

Surely having one VM per node is the standard deployment? Like how minikube did it, before kind.

It was Kata containers and now Apple container that went back to the one VM per container model...

I thought it would be an interesting concept of having one VM per pod, as a compromise between the two.

• Support Apple Containers on MacOS kubernetes/minikube#20933 (comment)

[BinHsu]

You're right, "unusual" was the wrong word — one VM per node is the classic minikube driver model (hyperkit/qemu), and kind was the departure from it, not this. I should have said the substrate is unusual, not the topology.

What's actually going on: apple/container's native grain is one micro-VM per container. k3ac aggregates that up to the node — each k3s node is a single micro-VM running containerd, and the pods on it share that node's kernel. So the isolation boundary today is the node, exactly like classic minikube; "no shared kernel" holds between nodes, not between pods on a node. The novelty is just the VM provider (vmnet-native on Apple Silicon, no Docker socket), not the cluster shape.

The one-VM-per-pod compromise you're describing is the genuinely interesting direction — and it's essentially Kata's pod-sandbox model. On apple/container that wouldn't be a node provisioner like k3ac at all; it'd be a CRI runtime shim mapping each pod to its own micro-VM — apple/container playing the role Kata's runtime plays under containerd. That actually fits the substrate's grain better than what I'm doing: I'm flattening per-container VMs up into per-node, where that approach keeps the VM boundary at the pod. Bigger project, but you've pointed at the one that fits the runtime best — probably where the first-party #1673 question gets decided.

[afbjorklund]

The VM provider isn't all that novel either, it is the same VZ, the main difference is the image - from cloud image to container image. That also lends itself to pre-installing the software from the Dockerfile, instead of bootstrapping after the first boot.

I still don't think there will be enough memory to allocate gigabytes for each pod (or app), and there is the whole CRI runtime support missing as well. So until that materializes, one VM per node is the pragmatic approach. Possibly shared, like kind.