k3s server fails to create k3s.yaml with versions greater than v1.32.3+k3s1

[highcoe]

Environmental Info:

I create multiple custom WSL distributions with Debian/RockyLinux running tools like Docker, Podman, K3S, RKE2.
The Linux setup is done using Docker with a Dockerfile, which finally results in a WSL image, which then runs in the WSL2 virtual machine.

K3S Version: v1.32.4+k3s1 (and newer)

Node(s) CPU architecture, OS, and Version:
x86_64, RockyLinux 9.x and Debian 12.x

Cluster Configuration:
K3S single node with 1 server/agent, running perferctly in a WSL2 virtual machine as local development environment

Describe the bug:

During the K3S installation/setup in my Dockerfile, I start k3s server to create k3s.yaml, so that I can then do kubectl- and helm-operations for my Kubernetes setup. However with K3S v1.32.4+k3s1 and newer, k3s.yaml is no longer created at this point. With older K3S versions everything was fine.

Steps To Reproduce:

I have reduced the following Dockerfile to reproduce the issue only, without all the other stuff needed for the final WSL environment.

FROM rockylinux/rockylinux:9.6.20250531

ARG K3S_VERSION="v1.32.3+k3s1"
#ARG K3S_VERSION="v1.32.4+k3s1"
#ARG K3S_VERSION="v1.33.1+k3s1"

RUN	set -eux && \
	dnf install -y nftables systemd && \
	curl -fLO https://github.com/k3s-io/k3s/releases/download/$K3S_VERSION/k3s --output-dir /usr/local/bin && \
	curl -fLO https://github.com/k3s-io/k3s/releases/download/$K3S_VERSION/k3s-airgap-images-amd64.tar.zst --create-dirs --output-dir /var/lib/rancher/k3s/agent/images && \
	chmod +x /usr/local/bin/k3s && \
	curl -sfL 'https://get.k3s.io' | \
		INSTALL_K3S_SKIP_DOWNLOAD=true \
		INSTALL_K3S_SKIP_START=true \
		INSTALL_K3S_SKIP_ENABLE=true \
		INSTALL_K3S_VERSION=$K3S_VERSION \
		sh - && \
	( /usr/local/bin/k3s server & ) && \
	until [ -f '/etc/rancher/k3s/k3s.yaml' ]; do sleep 1; done

k3 server is started before last line in the Dockerfile.

Reproduce:

1. use Docker Engine community version client and server in Linux environment (current version is 28.3.1)
2. build with Dockerfile in current folder> docker build --no-cache --progress=plain .
3. select newer ARG K3S_VERSION in the Dockerfile by commenting out the others
4. repeat building the Dockerfile and inspect the hanging failure

Expected behavior:

• /etc/rancher/k3s/k3s.yaml is created after some seconds with any K3S_VERSION, which is tagged 'latest' in Github now and in future

Actual behavior:

• /etc/rancher/k3s/k3s.yaml is created after some seconds (success) with K3S_VERSION less or equal "v1.32.3+k3s1"
• /etc/rancher/k3s/k3s.yaml is never created (failure) with K3S_VERSION greater than "v1.32.3+k3s1"

Additional context / logs:
I have the same trouble in the corresponding environment with RKE2:
Version "v1.32.3+rke2r1" creates /etc/rancher/rke2/rke2.yaml, newer versions fail to do so.

[brandond]

Why are you making your own K3s docker image instead of just running docker.io/rancher/k3s:v1.32.3-k3s1? If you want, you can just build a new image on top of that with an extra stage to download the airgap images.

[brandond]

If it's not creating the admin kubeconfig, then it's failing to start for some reason. You'd need to look at the logs to figure out why. That said, we do not really expect the github release artifact to be run in docker. Use the k3s docker image if you want do to that; it uses a slightly different build artifact that is designed for containerized use.

[highcoe]

Thank you for the feedback.

The container image of docker.io/rancher/k3s:v1.32.3-k3s1 seems to start k3s agent, not k3s server.

The log files gave a hint to start k3s server with option --snapshotter=native what finally creates the desired admin kubeconfig during Docker build.

I don't know why this this option is newly required with K3S versions from v1.32.4+k3s1 and later.

And I hope that there will not be any conflicts when starting k3s server with snapshotter native during buildtime, while later at runtime it runs with overlayfs in the WSL2 virtual machine.

[brandond]

The container image of docker.io/rancher/k3s:v1.32.3-k3s1 seems to start k3s agent, not k3s server.

You need to run the image with server or agent as the command. By default the command is agent.

The log files gave a hint to start k3s server with option --snapshotter=native

The native snapshotter will always work, but its performance is terrible. This message suggests that you are running k3s with /var/lib/rancher/k3s/agent on a filesystem that does not support overlayfs.

[highcoe]

The filesystem of /var/lib/rancher/k3s/agent during Docker build is always overlayfs.
This directory is created by the 2nd curl-command in the Dockerfile above when downloading the airgap images.

Latest WORKING version v1.32.3+k3s1:

[...]
#5 11.48 + curl -sfL https://get.k3s.io
#5 11.48 + INSTALL_K3S_SKIP_DOWNLOAD=true
#5 11.48 + INSTALL_K3S_SKIP_START=true
#5 11.48 + INSTALL_K3S_SKIP_ENABLE=true
#5 11.48 + INSTALL_K3S_VERSION=v1.32.3+k3s1
#5 11.48 + sh -
#5 12.07 [INFO]  Skipping k3s download and verify
#5 12.07 [INFO]  Skipping installation of SELinux RPM
#5 12.07 [INFO]  Creating /usr/local/bin/kubectl symlink to k3s
#5 12.08 [INFO]  Creating /usr/local/bin/crictl symlink to k3s
#5 12.08 [INFO]  Creating /usr/local/bin/ctr symlink to k3s
#5 12.08 [INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
#5 12.09 [INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
#5 12.10 [INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
#5 12.11 [INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
#5 12.12 + df /var/lib/rancher/k3s/agent
#5 12.12 Filesystem      1K-blocks    Used Available Use% Mounted on
#5 12.12 overlay        1055762868 7723212 994336184   1% /
#5 12.12 + stat -f -c %T /var/lib/rancher/k3s/agent
#5 12.12 overlayfs
#5 12.12 + /usr/local/bin/k3s server
[...]

First FAILING version v1.32.4+k3s1:

[...]
#5 19.05 + curl -sfL https://get.k3s.io
#5 19.05 + INSTALL_K3S_SKIP_DOWNLOAD=true
#5 19.05 + INSTALL_K3S_SKIP_START=true
#5 19.05 + INSTALL_K3S_SKIP_ENABLE=true
#5 19.05 + INSTALL_K3S_VERSION=v1.32.4+k3s1
#5 19.05 + sh -
#5 19.68 [INFO]  Skipping k3s download and verify
#5 19.68 [INFO]  Skipping installation of SELinux RPM
#5 19.68 [INFO]  Creating /usr/local/bin/kubectl symlink to k3s
#5 19.69 [INFO]  Creating /usr/local/bin/crictl symlink to k3s
#5 19.69 [INFO]  Creating /usr/local/bin/ctr symlink to k3s
#5 19.69 [INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
#5 19.70 [INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
#5 19.71 [INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
#5 19.72 [INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
#5 19.73 + df /var/lib/rancher/k3s/agent
#5 19.73 Filesystem      1K-blocks    Used Available Use% Mounted on
#5 19.73 overlay        1055762868 7729408 994329988   1% /
#5 19.73 + stat -f -c %T /var/lib/rancher/k3s/agent
#5 19.73 overlayfs
#5 19.73 + /usr/local/bin/k3s server
[...]

[brandond]

Yeah, don't do that. This is not how k3s is intended to be run in a container. Use the image that we produce for that purpose.

[thiyagarajan-dev]

@highcoe, Do you have any recent findings on why the customized WSL distribution image is incompatible with k3s version 1.32.4?

[highcoe]

It's not incompatible. As I learned, I just have to add the option --snapshotter=native when starting k3s server to get the admin kubeconfig created during Dockerbuild. With older versions of K3S, this option was not necessary. I have no idea why this has changed, but since the same behaviour also changed in RKE2 with the same versions, it's likely to be found in common sources?

Later, inside the WSL2 virtual machine, K3S of the customized WSL distribution runs fine using snapshotter overlayfs.

The distribution also installs a bunch of other tooling available (only) for Debian/RHEL-based distributions, so the Alpine-based image https://github.com/k3s-io/k3s/blob/master/package/Dockerfile#L1 is not really an option for me as baseimage.

[brandond]

You really shouldn't be starting k3s as part of an image build. K3s generates all the CA certs during its initial startup. If you reuse any of the datastore or certs from that initial startup in subsequent nodes, the admin kubeconfig from only node will have full access to any other cluster created from that same base.

If you're not reusing the certs or datastore, and aren't reusing images loaded into the snapshotter... then why are you even starting it up in the first place?