Support pinning/managing a custom runc binary (bundled runc's masked-path mount fails on older kernels)

[miltieIV2]

Summary

k0s bundles and unconditionally re-stages its own runc binary at /var/lib/k0s/bin/runc on every install/start, with no supported way to pin an alternate version or opt out of this management. On older kernels, the bundled runc fails because it relies on a tmpfs masked-path mount technique (nr_blocks=1,nr_inodes=1) that isn't supported there.

Environment / observed behavior

• Fleet of legacy hosts running an older LTS kernel line, bumped to 4.4 (the newest available via the distro's own backport path) specifically to meet a runtime-security agent's minimum kernel requirement.
• With k0s's bundled runc, worker/controller nodes fail to start containers with a masked-path mount error on this kernel.
• Manually swapping in runc v1.2.9 (which predates the masking technique) at /var/lib/k0s/bin/runc fixes it completely — confirmed stable in production across dozens of hosts.
• However, k0s re-stages its own bundled runc on every start, silently reverting the fix. Confirmed this happens more than once per lifecycle: once immediately at install, and again some seconds after control-plane readiness on a controller+worker node (timing varies with host load, observed ~8-10s on an otherwise-idle host). There's no config option to tell k0s "don't manage runc" or "use this specific runc binary."

Current workaround (fragile)

We run a small always-on daemon, triggered on every k0scontroller/k0sworker start (not a one-shot post-install hook), that polls the runc binary's checksum and re-swaps it whenever k0s overwrites it. A one-shot swap-after-install approach was tried first and missed the later re-stage entirely. This works but is a real hack around the absence of first-class support, and cost significant time to discover and get right.

Feature request

Either of these would remove the need for the workaround:

1. A config option (e.g. under spec.k0s.config.spec.workerProfiles, or a top-level key) to point k0s at a specific runc binary path/version instead of always using the bundled one — similar in spirit to how k0sctl's k0sBinaryPath lets you bring your own k0s binary.
2. Or, at minimum, a way to disable k0s's own runc management entirely (skip re-staging if a runc binary already exists at the target path), so external tooling can manage it without fighting k0s's own reconciliation on every start.

Happy to provide more detail on the exact masked-path mount failure if useful (kernel 4.4; bundled runc version failed, v1.2.9 confirmed working).

[jnummelin]

oh, that is interesting and very unexpected finding. Did you by any chance find anything in upstream runc that reports this? Just curious on the details.

In case you are on k0s 1.36, there's a trick that might work. In k0s 1.36, we moved to have the embedded bins attached to the binary as a zip archive. This means that you could actually take the k0s binary and swap out the runc bin within the zip archive. So something like this:

# cp k0s k0s.zip
# zip -A k0s.zip
Zip entry offsets appear off by 103481506 bytes - correcting...
# wget https://github.com/opencontainers/runc/releases/download/v1.2.9/runc.arm64
Connecting to github.com (140.82.121.4:443)
Connecting to release-assets.githubusercontent.com (185.199.108.133:443)
saving to 'runc.arm64'
runc.arm64           100% |**************************************************************************************************************************************| 11.3M  0:00:00 ETA
'runc.arm64' saved
# mv runc.arm64 runc
# zip -0 k0s.zip runc 
updating: runc (stored 0%)
# cp k0s.zip /usr/local/bin/k0s
# k0s install controller --single && k0s start
...
# k0s kc get node,pod -A
NAME                 STATUS   ROLES           AGE     VERSION
node/lima-k0s-demo   Ready    control-plane   3m48s   v1.36.3+k0s

NAMESPACE     NAME                                  READY   STATUS    RESTARTS   AGE
kube-system   pod/coredns-7b99595d6d-mtcg2          1/1     Running   0          3m48s
kube-system   pod/kube-proxy-2rs9c                  1/1     Running   0          3m48s
kube-system   pod/kube-router-srd5m                 1/1     Running   0          3m48s
kube-system   pod/metrics-server-6b7b849cb6-4stpp   1/1     Running   0          3m43s
# /var/lib/k0s/bin/runc --version
runc version 1.2.9
commit: v1.2.9-0-g6524246c
spec: 1.2.0
go: go1.24.10
libseccomp: 2.5.5

The ZIP stuff is currently very undocumented as it's still considered as an internal implementation detail. However, we do have some plans to expose this as a user facing "feature" too.

NOTE: I just tested that I can replace the runc bin with that 1.2.9 version. I did not run any real testing on it's compatibility or anything, so your mileage may vary. 😄

[ncopa]

This zip thingy is super cool! so in theory, you could use zip tooling to remove runc from k0s and then it should pick up the system one?

[jnummelin]

you could use zip tooling to remove runc from k0s and then it should pick up the system one?

Looking at the code, yes. But TBH didn't test that path

[miltieIV2]

The idea here is to get the older runc-1.2.9 added to the k0s release in order to support older kernels out of the box (zero friction) - particularly the hybrid cluster where some nodes need it. It should be straightforward to either use a kernel version comparison or a functional tmpfs test to decide which to use.

/var/lib/k0s/bin/runc --version
runc version 1.2.9
commit: v1.2.9-0-g6524246c
spec: 1.2.0
go: go1.24.10
libseccomp: 2.5.5

---

My use case is to pull all my legacy EC2 instances into per-environment K0S clusters for visibility, management, and security scanning.

[jnummelin]

k0s bundling two different runc versions sounds bit fishy, at least on first thoughts, TBH. Also from the point of view that runc 1.2.9 is kinda old, and has a number of CVEs reported agains it. (Trivy finds 46... 🙈 )

In your proposed feature workarounds:

a way to disable k0s's own runc management entirely (skip re-staging if a runc binary already exists at the target path)

By using zip to remove the embedded runc, you can actually achieve this.

# manually installed runc 1.2.9:
lima-k0s-demo:~# /var/lib/k0s/bin/runc --version
runc version 1.2.9
commit: v1.2.9-0-g6524246c
spec: 1.2.0
go: go1.24.10
libseccomp: 2.5.5
# stripped runc away from the k0s bin with zip -d ...
lima-k0s-demo:~# unzip -l /usr/local/bin/k0s | grep runc
  8061090  1980-00-00 00:00   containerd-shim-runc-v2
# k0s happily uses the manually installed one:
lima-k0s-demo:~# k0s kc get pod -A
NAMESPACE     NAME                              READY   STATUS    RESTARTS   AGE
kube-system   coredns-7b99595d6d-qr7wd          1/1     Running   0          9m41s
kube-system   kube-proxy-jcs5f                  1/1     Running   0          9m41s
kube-system   kube-router-xtfrw                 1/1     Running   0          9m41s
kube-system   metrics-server-6b7b849cb6-gdsfd   1/1     Running   0          9m39s

similar in spirit to how k0sctl's k0sBinaryPath lets you bring your own k0s binary.
You could pre-fabricate the runc-stripped k0s bin and use k0sctl k0sBinaryPath to point to that. And also use the k0sctl files/hooks to push the needed version of runc in place.

That makes me think, k0sctl hooks could probably allow us to "automate" all this, something like:
(very pseudo-code example in the hook)

hosts:
- role: controller+worker
  hooks:
    install:
      before:
        - if [ $(uname -r) == "4.4.xyz"]; then mv /usr/local/bin/k0s k0s.zip && zip -A k0s.zip && zip -d k0s.zip runc && mv k0s.zip /usr/local/bin/k0s; fi

@kke I do not remember by heart at what points the hooks kick-in, can you check if that'd do the job?

[miltieIV2]

My situation is the late stages of AWS EKS migration. Almost all EC2 instances are EKS nodes now. My goal here was to catch up all the legacy AWS EC2 instances so that everything looks like k8s (all instances in some cluster in Lens - no exceptions). At that point, security and compliance scanning can be 100% k8s - KuebArmor, openshift-compliance, etc.

Some of these instances are ancient AWS Ubuntu Trusty AMI derivatives. Those were Liniux 3.13-3.18 - no way to run K0S at all. By upgrading the kernels to the Ubuntu HWE kernels (Trusty distro linux-generic-lts-xenial), I was able to get to a Linux 4.4 kernel. Combining that with GRUB_CMDLINE_LINUX="${GRUB_CMDLINE_LINUX} cgroup_enable=memory swapaccount=1" and k0sctl spec.k0s.config.spec.workerProfiles[default].values.failCgroupV1=false, I was then able get off the ground - cluster up, but pods would not start without the runc downgrade. Adding a clunky always running upstart script to forcibly downgrade runc constantly allowed pods to start (intermittent failures until runc was hacked).

At this point I was able to launch KubeArmor, but since it requires nested eBPF, it failed with a Linux must be >= 4.12 error. Some of these nodes are Ubuntu Xenial distro, so I was able to get those up to Linux 4.15.0-142-generic with the Xenial HWE linux-generic-hwe-16.04 kernel. The Trusty nodes required tracking down a non-Ubuntu kernel that would work with a Trusty distro: linux-{image-unsigned,modules,headers}-4.19.325-0419325-{generic_amd64,all}.deb. All this ended up in a big prep-node.sh script to assimilate the instances to the point that k0sctl could bring them into the clusters. This script is needed because an ancient instance could reappear from an ASG-based instance replacement and have to be re-assimilated.

Now that these clusters are under KubeArmor control, security monitoring and mitigation can be applied.

So for my case, having a dynamic old runc-1.2.9 would get me through the bootstrap phase to get k0s stood up to get to the security tools to figure out what needs to be fixed. These nodes are so ancient that I can't even pull the k0s file with k0sctl - I have to push the k0s file onto the instance. Since I have to do that, I could use the spiffy zip features to repack my own custom k0s binary with the runc-1.2.9 that I need.

Claude and I bashed around for quite a while trying to get the k0sctl hooks to do the heavy lifting and failed - lots of race conditions.

So yeah, a lot of friction just to get to the zero friction bit.

[jnummelin]

a lot of friction just to get to the zero friction bit.

Sounds like a bunch, yeah. Much of that stemming from the fact that Trusty has been EOL since 2019. The whole ecosystem moves forward such fast that we, maybe too easily, ignore older stuff.

These nodes are so ancient that I can't even pull the k0s file with k0sctl - I have to push the k0s file onto the instance.
Claude and I bashed around for quite a while trying to get the k0sctl hooks to do the heavy lifting

So you use the spec.hosts[*].k0sBinaryPath to push the bin from your workstation? If that's the case, then I don't think you need any hooks or anything, unless you use hooks for something else too.

Just tested this:

#!/bin/sh

# Download k0s bin
curl -fsSL -o k0s.zip https://github.com/k0sproject/k0s/releases/download/v1.36.3%2Bk0s.2/k0s-v1.36.3+k0s.2-amd64

# Download runc 1.2.9 bin
curl -fsSL -o runc https://github.com/opencontainers/runc/releases/download/v1.2.9/runc.amd64

# Swap runc binary in k0s zip
zip -A k0s.zip
zip -d k0s.zip runc
zip -0 k0s.zip runc

mv k0s.zip k0s-runc129

Then push that to a host with k0sctl:

apiVersion: k0sctl.k0sproject.io/v1beta1
kind: Cluster
metadata:
  name: single-node-custom-zip
spec:
  hosts:
    - role: controller+worker
      ssh:
        address: 1.2.3.4
        user: root
        port: 22
      uploadBinary: true
      k0sBinaryPath: k0s-runc129

Also tested a hook version:

apiVersion: k0sctl.k0sproject.io/v1beta1
kind: Cluster
metadata:
  name: single-node-custom-zip
spec:
  hosts:
    - role: controller+worker
      ssh:
        address: 1.2.3.4
        user: root
        port: 22
      hooks:
        # Use install pre hook to to rip out runc
        # k0s will then use the one provided by system
        install:
          before:
            - |
              if [ -x /usr/local/bin/k0s ]; then
                mv /usr/local/bin/k0s /usr/local/bin/k0s.zip
                zip -A /usr/local/bin/k0s.zip
                zip -d /usr/local/bin/k0s.zip runc || true
                cp /usr/local/bin/k0s.zip /usr/local/bin/k0s
                chmod 0755 /usr/local/bin/k0s

                curl -fsSL -o /usr/local/bin/runc https://github.com/opencontainers/runc/releases/download/v1.2.9/runc.amd64
                chmod 0755 /usr/local/bin/runc

                echo "Modified k0s at $(date)" >> /tmp/k0sctl-hooks.log
              fi

Both tests on Ubuntu 22, that's the oldest I could easily find on my cloud provider stock images.

[miltieIV2]

By the time I got all this working, the spec.host is huge:

• I have to push the k0s binary
• containerd settings (99-old-kernel.toml):

version = 3
[plugins."io.containerd.cri.v1.runtime"]
  enable_unprivileged_icmp = false
  enable_unprivileged_ports = false

• We have a Harbor docker pull-through caching proxy for docker hub (mirror and hosts).
• Per host SSH settings

...
  hosts:
  - role: controller+worker
    hostname: sensu
    uploadBinary: true
    k0sBinaryPath: k0sctl/remote-bin/k0s-v1.36.3+k0s.0-amd64
    installFlags:
      - --ignore-pre-flight-checks
      - --disable-components=autopilot
    files:
      - src: ../containerd.d/99-old-kernel.toml
        dstDir: /etc/k0s/containerd.d/
        perm: "0644"
      - src: ../containerd.d/98-registry-mirror.toml
        dstDir: /etc/k0s/containerd.d/
        perm: "0644"
      - src: ../containerd.d/certs.d/docker.io/hosts.toml
        dstDir: /etc/k0s/containerd.d/certs.d/docker.io/
        perm: "0644"
    ssh:
      address: 10.30.30.144
      user: admin
      port: 22
      keyPath: ~/.ssh/id_rsa
...

---

When Claude messed around with this, it did not know about the k0s repack stuff (probably because it is so new).
The newer kernel .deb files are xz compressed, so those have to be repacked to gzip so they could be installed, so repacking stuff is already a thing.

I will fire that Claude session back up on this and have it cook up zip-re-packs for specific runc requirements.

[miltieIV2]

Follow-up: the repack approach worked great in production. Confirmed in two stages before touching anything live:

1. In a disposable container: repacked the k0s binary with the compatible runc, confirmed it extracts correctly, and confirmed that corrupting the extracted binary and restarting k0s re-extracts a clean copy every time (i.e. the normal re-staging behavior on restart/reboot becomes the thing that keeps it fixed, instead of something to fight).
2. On a real host: stopped our old always-running "keep re-copying the right runc" daemon, corrupted runc, restarted the worker service with the old (un-repacked) binary as a baseline — confirmed it reverted to the broken bundled version, proving the daemon really was the only thing pinning it before. Installed the repacked binary and repeated the corrupt-and-restart test twice more with the daemon stopped the whole time — self-healed to the compatible version both times.

Rolled it out to our whole fleet (dozens of legacy-kernel hosts across five environments) with zero issues, and retired the daemon entirely.

Really appreciate the pointer to the zip-repack trick — solved this cleanly at the source instead of us needing to keep fighting k0s's own reconciliation. Feel free to close this out on our end; the original ask (native support for an alternate/pinned runc) would still be nice to have someday, but the repack approach is a fully workable path for anyone else who hits this.

[jnummelin]

Great to hear everything worked out, thanks for updating us.

We will eventually document the zip details. Like mentioned originally, this is not, yet, exposed as user facing thing. Once we document all that, we can probably document this use case too as an example. The kernel runc version stuff part of it I mean.

[jnummelin]

I think we should keep this still open, we might wanna have some pre-flight check for the runc/kernel compatibility check.