Feature Discussion k8s plugin

[jshi991]

Overview

The k8s plugin adds Kubernetes development cluster support to the container CLI. Cluster nodes run k3s, a lightweight Kubernetes distribution, packaged as a container image that runs directly on the Apple container runtime.

This is an experimental feature. The goal is to make spinning up a local Kubernetes cluster fast and simple.

Proposed UX

Intial feature set - run , create, delete, loadImage, write-config, list

The initial feature set covers the core day-to-day workflow: boot a single-node cluster, load images into it, point kubectl at it, and tear it down when finished.

k3s base image — Before any cluster can start the plugin needs a k3s node image. run and create check for the image on first use and build it automatically if it is missing. The image layers rancher/k3s with a custom entrypoint that handles Apple-kernel specifics: native containerd snapshotter, cgroup v2 setup, and the sysctl values required for Kubernetes networking.

run / create / delete — run creates and starts a single control-plane cluster in one step and writes a kubeconfig entry to ~/.kube/config automatically. create creates the container without starting it. delete stops and removes the cluster and cleans up its entries from ~/.kube/config.

container k8s run my-cluster
container k8s create my-cluster
container k8s delete my-cluster
container k8s delete --force my-cluster

load-image — Loads a locally-stored image into the cluster's containerd so kubelet can schedule pods using it without an external registry. The image is saved from the container image store to a temporary OCI tar and streamed into the node via ctr images import. Short references (alpine) are automatically qualified to docker.io/library/alpine:latest.

container k8s load-image my-cluster my-app:latest

write-config / get-kubeconfig — write-config merges the cluster's kubeconfig into ~/.kube/config and sets current-context. get-kubeconfig prints the YAML to stdout for scripts or manual merging. Both rewrite the API server address to 127.0.0.1:<host-port> or to the machine's FQDN when one is detected.

container k8s write-config my-cluster
container k8s get-kubeconfig my-cluster

list — Lists all clusters and their nodes in a table, discovered via the plugin=k8s container label.

container k8s list
# CLUSTER     NODE        ROLE           STATE    CPUS  MEMORY   ADDR          PORTS
# my-cluster  my-cluster  control-plane  running  4     4096 MB  192.168.64.5  6445->6443

Proposed Architecture

Future Work

API Support - all cluster topology requests will map to a well defined configuration object

Multi-node clusters — run will accept a --workers N flag to start N worker nodes alongside the control plane. Each worker runs k3s in agent mode and joins using a token fetched from the control plane. load-image will target all nodes by default so every kubelet can schedule pods regardless of placement.

Load balancing — Allow clusters to expose services through a host-accessible endpoint without manual port-forwarding. k3s ships with ServiceLB; the integration point will be designed to map cleanly to the container networking model.

Registries — Let users point a cluster at a local registry (e.g. one started with container run) so that images pushed there are immediately pullable inside the cluster without running load-image manually. This maps to the k3s registries.yaml mirror configuration.

Questions for the Community

We'd love your feedback:

1. Is this something you'd use? Would this be valuable to your workflow?
2. What features are must-haves?
3. What use cases would this unlock for you?

---

Please share your thoughts and feedback/suggestions. We want to hear from you!

[afbjorklund]

Shouldn't the plugin rather be called k3s? It is expected to run kubeadm for k8s

[afbjorklund]

Previous hacks for container were listed at:

• run kubernetes #81

[jglogan]

I had originally suggested k8s for the subcommand simply because it's an easy-to-type shorthand for Kubernetes in the generic sense. Do you see any issue with using k8s with something like a --driver or --template option to specify a particular node implementation when creating a cluster?

[jglogan]

and the plugin name maybe renamed to something such as cluster

container cluster for Kubernetes cluster management isn't a good name choice IMO.

[afbjorklund]

I had originally suggested k8s for the subcommand simply because it's an easy-to-type shorthand for Kubernetes in the generic sense.

I was more expecting container kubernetes to match container machine and other long names...

You don't have to follow our naming from Lima, but we used k3s and k8s to match get.k3s.io and get.k8s.io (kubeadm)

[jglogan]

container kubernetes is something we can consider since we have a convention of including one-character aliases and Kubernetes would be a leading candidate for the k alias.

[zendes]

I'd love to be able to install Cilium in these clusters. that would require some additional kernel configs to be enabled, which are documented here

[jglogan]

Registries don't need to be implemented immediately but it's worth giving a little thought to them and how they might interact with the cluster feature.

I'd like to see the default container k8s create command bring up a cluster that is generally useful for the most common use cases.

My assumption that the most common use case is local cloud native development, where having a local registry makes it so that the developer can use the same build-tag-push method for making images available, and so that they don't have to vary imagePullPolicy because the local environment requires transferring images directly into node image stores.

The tool has the container registry subcommand today for interacting with remote registries (login, logout, ls). This command could be extended to allow the user to manage registry containers (which would be regular containers after creation, no special guards or filters around them).

If mkcert is installed and configured we could also set up TLS and basic authentication for a registry when it's created.

With the convenience wrapper around registries in place, there are a few different ways to connect clusters and registries.

Most opinionated would be that by default, cluster create first ensures that a registry container with a well known name (such as container-registry or k8s-registry) is created and running, then starts the cluster. If a local DNS domain isn't defined and set as the default, we might need localhost mapping similar to https://kind.sigs.k8s.io/docs/user/local-registry/ during cluster configuration.

I've implemented this pattern in similar tools and it tends to be useful for most users, but it requires options for "don't do anything about a registry" or "use this registry" if you need to escape the built-in opinionation.

Less opinionated would be that by default, cluster create only if a registry container with the well known name exists, or if the user specifies an alternate registry container to use. The user is responsible for explicitly creating the registry container.

[afbjorklund]

The minikube and the Lima template:k8s does not use any registry, they build the images right on the control plane. That keeps the images on the same machine, so there is no need to push anything anywhere... You can still install a registry, as an add-on.

Currently it is a bit hard to expose the docker socket with container machine, since custom mounts are missing. But you can still access the k8s containers over coma --root nerdctl -n k8s.io and the CLI. Installing SSH is another workaround, or using TCP.

[jshi991]

#2043

[wibed]

why not use the kubeclient library? https://github.com/swiftkube/client

[jglogan]

It's worth a look. Certainly there's some appeal in removing shell-outs. On the suspect side it hasn't seen an upgrade in five months.

I wouldn't yolo up a PR; instead create an issue and justify it based on how much it would simplify/remove from the existing ContainerK8s library.

[mcg1969]

Hello folks!

First: let me express my genuine gratitude for this work. My work involves Kubernetes application development, with a healthy dose of Rosetta emulation, so I am also pleased to see the k8s support shaping up. I've already moved from Minikube to Docker Desktop for k8s, and would be happy to move to this as it matures.

Now, that said, I actually saw the apple/container k8s work after I had discovered a very promising alternative, Kiac. I hope the author @saiyam1814 will forgive me for tagging him here: it's a tremendous piece of work already, even in its early stage. It offers a more complete feature set and a smooth experience. This post by the author offers a fuller discussion of what it offers: https://blog.kubesimplify.com/introducing-kiac-kubernetes-in-apple-containers

Given the timelines for both projects I strongly suspect this is a case of independent development by two people with the same great idea. But I think it would be wise for the authors here to take a look at Kiac and consider whether it's worth either throwing some weight behind that project instead, or finding another way to collaborate closely with Kiac.

I asked Claude to compare the two repos with the following in mind—what aspects of the future work discussed here are already shipping in kiac. Here's what it found for me:

• Multi-node clusters — --workers N boots each node as its own VM with real cross-node networking and scheduling. Maps to the plugin's planned --workers N.
• type: LoadBalancer out of the box — kiac-lb assigns node IPs to Services in ~2s, shares IPs across non-colliding Services, and self-heals. Maps to the "Load balancing" item. (kubeadm path; k3s clusters use bundled servicelb.)
• Cilium / eBPF — --cni cilium --kernel full ships a sha-pinned kernel with VXLAN, eBPF, and br_netfilter and drives the official installer. Directly answers the open Cilium request in Feature Discussion `k8s` plugin #1673.
• Real node-failure testing — kiac stop node / start node produce genuine NotReady, eviction, rescheduling, and rejoin — a payoff of the VM-per-node model that single-node clusters can't exercise.
• Reboot survival — kiac resume cluster restarts VMs and heals the control-plane IP across apiserver cert, kubeconfigs, and kube-proxy, idempotently.
• Batteries-included addons — metrics-server (kubectl top), a default StorageClass for PVCs, optional Prometheus/Grafana (--observability), and Gateway API + Traefik (--gateway), all preconfigured on create.
• Distro abstraction — one interface over both kubeadm (kindest/node) and k3s, with pinned K8s 1.32–1.36. Close to the --driver / --template idea raised in-thread.
• Declarative clusters — kiac create cluster --config cluster.yaml defines a full cluster in one file, with flag overrides. A possible model for the planned config/API object.

An argument could be made that there is value in separating the two projects cleanly: let container be the docker to kaic's Kubernetes.

In any case — thank you again, and to @saiyam1814 as well!