"Permission denied" on commands when running k3s in docker

[onigoetz]

Hi,

I'm running K3s inside Docker (on Truenas). Some containers fail to start either with "permission denied" when running commands or with Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: exec: "argocd": executable file not found in $PATH.

I run k3s with the following Docker Compose:

services:
  server:
    hostname: k3s-control-1
    image: "rancher/k3s:v1.33.3-k3s1"
    networks:
      default: {}
      k3snet:
        ipv4_address: 192.168.1.6
    command:
    - server
    - --disable=traefik
    - --disable=local-storage
    - --disable-helm-controller
    - --advertise-address=192.168.1.6
    dns:
      - 192.168.1.1
    tmpfs:
    - /run
    - /var/run
    privileged: true
    security_opt:
    - seccomp:unconfined
    environment:
    - K3S_TOKEN=<secret>
    - K3S_KUBECONFIG_OUTPUT=/output/kubeconfig.yaml
    - K3S_KUBECONFIG_MODE=666
    - K3S_DATASTORE_ENDPOINT=mysql://k3s:<password>@tcp(172.16.0.1:3306)/k3s
    volumes:
      - /mnt/tank_nvme/apps/k3s/output:/output
      - /mnt/tank_nvme/apps/k3s/control:/var/lib/rancher/k3s

  agent:
    hostname: k3s-worker-1
    image: "rancher/k3s:v1.33.3-k3s1"
    networks:
      default: {}
      k3snet:
        ipv4_address: 192.168.1.7
    command:
    - agent
    dns:
      - 192.168.1.1
    tmpfs:
    - /run
    - /var/run
    privileged: true
    security_opt:
    - seccomp:unconfined
    volumes:
      - /mnt/tank_nvme/apps/k3s/worker-1:/etc/rancher/node
      - /mnt/tank_nvme/apps/k3s/storage:/var/lib/rancher/k3s/storage
    environment:
    - K3S_URL=https://server:6443
    - K3S_TOKEN=<secret>
networks:
  k3snet:
    driver: macvlan
    driver_opts:
      parent: br0
    ipam:
      driver: default
      config:
        - subnet: 192.168.1.0/24
          gateway: 192.168.1.1

To be honest, I don't think the source of this problem is k3s as I got the same behavior when starting Talos in Docker.
So I think this is caused by the nested nature of containers.

There is already seccomp:unconfined and privileged: true which should give the highest access to the containers but that doesn't seem to be enough.

Does anybody have what this is caused by ? My understanding was that seccomp profiles are inherited so this should have helped. Also not sure if rootless containers would have helped.

[brandond]

Try doing this with k3d, and look at how that sets up the containers.

[onigoetz]

will check, thanks

[onigoetz]

I ran k3d in a VM and inspected the parameters; they are the same as what I gave to Docker Compose.
I also tried

    security_opt:
    - seccomp=unconfined

(unclear if the separator should be a = or a :)

and

    security_opt:
    - label=disable

I also tried to run with both, but that didn't change anything.

Then I tried to install K3S directly on Truenas (without Docker or a VM) and, weirdly, the issue is the same; containers have the same issue as within Docker.

At this stage I conclude that the issue comes from Truenas itself, but I fail to understand how.

[onigoetz]

I found the cause and a working solution.

The cause wasn't a security_opt but a filesystem one; all container filesystems were mounted as noexec

shm on /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/6b0d3f5dc0426d3b774dc47e2beb062eb802e5d71dac5489af2a7f3206b1b2e3/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k,inode64)
shm on /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/f69b02fb47ba32527bc91e722149d3187f04bb23d1d52cfeb7303c85822cfb4e/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k,inode64)
shm on /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/5594e9d8ff604c4307b7373e278becb6290ed904ffe0b95ea42391e265523d41/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k,inode64)

The /run mount has itself a noexec flag which, as far as my understanding goes, is inherited by k3s.
I could not find a way to disable this, even removing the flag from the mount doesn't prevent this from happening.

I tried to start k3s with --container-runtime-endpoint unix:///run/containerd/containerd.sock to use the containerd that's on the system already for Docker but that didn't work, because the containerd version is too old if I understand correctly.

Last try I did is to start with --docker as the docker containers seem to work fine and that did the trick. I now have a working Kubernetes node within Truenas scale