Problem debugging with EphemeralContainers

[ffatghub]

Hi, thanks for your work k3d is great! I have a problem enabling or using EphemeralContainers

k3d cluster create --k3s-server-arg '--kube-apiserver-arg=feature-gates=EphemeralContainers=true' --config mycluster.yaml

I think the feature-gate is enabled (ps -auxfww):

[cut] /sbin/docker-init -- /bin/k3s server --kube-apiserver-arg=feature-gates=EphemeralContainers=true --tls-san 0.0.0.0

then I am following steps in

https://kubernetes.io/docs/tasks/debug-application-cluster/debug-running-pod/#ephemeral-container

and I get this error at debug time

Defaulting debug container name to debugger-mfj6k. error: error updating ephemeral containers: EphemeralContainers in version "v1" cannot be handled as a Pod: no kind "EphemeralContainers" is registered for version "v1" in scheme "k8s.io/kubernetes/pkg/api/legacyscheme/scheme.go:30"

What am I missing? Any help is appreciated.
thanks
fra

k3d/k3s versions

k3d version v4.4.6
k3s version v1.21.1-k3s1 (default)

[sgandon]

I was trying the same thing tonight and managed to have it working. The command for creating the cluster is the following

k3d cluster create k3d-122 -i rancher/k3s:v1.22.3-k3s1 --k3s-arg '--kube-apiserver-arg=feature-gates=EphemeralContainers=true@server:0'

[sgandon]

According to K8s doc you need a cluster v1.22
And above I am using the latest k3d (v5.0.2)

[ffatghub]

Hi @sgandon , thanks a lot. I updated my environment with no luck:

k3d v5.1.0
k3s v1.22.3-k3s1
kubectl v1.22.3

Now the schema error is gone but "kubectl debug" hangs indefinitely waiting for a reply. The issue is described here:

https://stackoverflow.com/questions/65306212/kubectl-debug-hangs-on-1-20-with-feature-gate-enabled
https://stackoverflow.com/questions/60903633/kubernetes-alpha-debug-command

As far as I understand the container is created as a resource in the control plane (text below comes from a describe of the target pod)

ephemeralContainers:

• image: busybox
  imagePullPolicy: Always
  name: debugger-vcqgf
  resources: {}
  stdin: true
  targetContainerName: ephemeral-demo
  terminationMessagePath: /dev/termination-log
  terminationMessagePolicy: File
  tty: true

however it is not instantiated by the k3s agent... maybe the EphemeralContainers=true flag must be propagated to the k3s agent too?

any help is appreciated
thanks
ciao

[sgandon]

Can you give the commands you are using, because the command I pasted above is working for me, then I did use the k8s example

kubectl debug -it mypod --image=busybox --target=mycontainer

[ffatghub]

Hi @sgandon,

thanks again. You are right the command you pasted is working as expected with latest k3d/k3s. It must be something related to the way I use the k3d with configuration file.

k3d cluster create --config cluster.yaml

cluster.yaml:

apiVersion: k3d.io/v1alpha3
kind: Simple
name: test
servers: 1
agents: 4
image: rancher/k3s:v1.22.3-k3s1
volumes:
  - volume: /mnt/data:/mnt/data
    nodeFilters:
      - server:*
      - agent:*
options:
  k3d:
    wait: true
    timeout: "120s"
    disableLoadbalancer: true
  k3s:
    extraArgs:
      - arg: --node-taint=CriticalAddonsOnly=true:NoSchedule
        nodeFilters:
          - server:*
      - arg: --no-deploy=traefik
        nodeFilters:
          - server:*
      - arg: --kube-apiserver-arg=feature-gates=EphemeralContainers=true
        nodeFilters:
          - server:*
  kubeconfig:
    updateDefaultKubeconfig: true
    switchCurrentContext: true

I thought the extra args statement was wrong but the resulting command executed is correct (?)

sbin/docker-init -- /bin/k3s server --node-taint=CriticalAddonsOnly=true:NoSchedule --no-deploy=traefik --kube-apiserver-arg=feature-gates=EphemeralContainers=true --tls-san 0.0.0.0

any idea how to debug this?

thanks a lot
ciao

[ffatghub]

Hi @sgandon, all,

guess I figured out at least the apparent cause. If the ephemeral-demo pod is scheduled on anything different then the server node, the ephemeral container does not start.

Could you please try your command with -a 4?

Make sure ephemeral-demo is scheduled on agent-N (via nodeSelector for example). It might also land on agent-N if you execute cluster creation and the pod run after a very short time like “k3d cluster create ... && kubectl run ephemeral-demo ...“.

Under this condition, the subsequent debug command is failing in my environment.

hope this helps
ciao

[ffatghub]

I found I way to deterministically replicate the problem

k3d cluster create k3d-122 -i rancher/k3s:v1.22.3-k3s1 -a 4 --k3s-arg '--kube-apiserver-arg=feature-gates=EphemeralContainers=true@server:0'

kubectl run ephemeral-demo --image=k8s.gcr.io/pause:3.1 --restart=Never --overrides='{ "spec": { "nodeSelector": { "kubernetes.io/hostname": "k3d-k3d-122-agent-1" } } }'

[iwilltry42]

Hi @ffatghub , thanks for starting this discussion and sorry for my late reply!
@sgandon , thanks for chiming in here 🙏

So far, you've enabled the feature gate at the API-Server and you may have to enable it on the kube-scheduler and the kubelets as well:

k3d cluster create k3d-122 -i rancher/k3s:v1.22.3-k3s1 -a 4 --k3s-arg '--kube-apiserver-arg=feature-gates=EphemeralContainers=true@server:*' --k3s-arg='--kube-scheduler-arg=feature-gates=EphemeralContainers=true@server:* --k3s-arg='--kubelet-arg=feature-gates=EphemeralContainers=true@agent:*''

At least this worked for with the following commands:

$ kubectl run ephemeral-demo --image=k8s.gcr.io/pause:3.1 --restart=Never --overrides='{ "spec": { "nodeSelector": { "kubernetes.io/hostname": "k3d-k3d-122-agent-1" } } }'

pod/ephemeral-demo created

$ kubectl debug -it ephemeral-demo --image=busybox --target=ephemeral-demo

Targeting container "ephemeral-demo". If you don't see processes from this container it may be because the container runtime doesn't support this feature.
Defaulting debug container name to debugger-vn2zg.
If you don't see a command prompt, try pressing enter.
/ # ps aux
PID   USER     TIME  COMMAND
    1 root      0:00 /pause
    8 root      0:00 sh
   14 root      0:00 ps aux
/ # hostname
ephemeral-demo
/ # 

# in a different shell
kdp ephemeral-demo  
Name:         ephemeral-demo
Namespace:    default
Priority:     0
Node:         k3d-k3d-122-agent-1/172.21.0.6
Start Time:   Fri, 10 Dec 2021 14:00:24 +0100
# ... truncated ...
Containers:
  ephemeral-demo:
    Container ID:   containerd://bf9ea56190c16edf397cc01cffd566fcf3caa2c17e1bb44c8898a40be4885d73
    Image:          k8s.gcr.io/pause:3.1
    Image ID:       docker.io/rancher/mirrored-pause@sha256:82f4f58a1877a3a819de055a111b33fc63127b896b928b0bd7419da549f88d65
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Fri, 10 Dec 2021 14:00:25 +0100
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fzcg4 (ro)
Ephemeral Containers:
  debugger-vn2zg:
    Container ID:   containerd://a68b4bde3ef8319e53538f1a3c177a9e02647238370e26ced3296a05aa5dbeb8
    Image:          busybox
    Image ID:       docker.io/library/busybox@sha256:b5cfd4befc119a590ca1a81d6bb0fa1fb19f1fbebd0397f25fae164abe1e8a6a
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Fri, 10 Dec 2021 14:00:38 +0100
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:         <none>
# ... truncated ...
Node-Selectors:              kubernetes.io/hostname=k3d-k3d-122-agent-1
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  2m40s  default-scheduler  Successfully assigned default/ephemeral-demo to k3d-k3d-122-agent-1
  Normal  Pulling    2m41s  kubelet            Pulling image "k8s.gcr.io/pause:3.1"
  Normal  Pulled     2m40s  kubelet            Successfully pulled image "k8s.gcr.io/pause:3.1" in 702.025356ms
  Normal  Created    2m40s  kubelet            Created container ephemeral-demo
  Normal  Started    2m40s  kubelet            Started container ephemeral-demo
  Normal  Pulling    2m30s  kubelet            Pulling image "busybox"
  Normal  Pulled     2m27s  kubelet            Successfully pulled image "busybox" in 3.054293788s
  Normal  Created    2m27s  kubelet            Created container debugger-vn2zg
  Normal  Started    2m27s  kubelet            Started container debugger-vn2zg

Update: I also updated the FAQ entry containing this use case as an example to reflect all three args: https://k3d.io/v5.2.1/faq/faq/#passing-additional-argumentsflags-to-k3s-and-on-to-eg-the-kube-apiserver

[ffatghub]

Hi @iwilltry42, nice, thank you! Enabling the feature gate in all CO components (API-Server, kube-scheduler and kubelets) makes the e phemeral container up and running :-)

Below the config file version of your cli-args example.

ciao

options:
  k3s:
    extraArgs:
      - arg: --kube-apiserver-arg=feature-gates=EphemeralContainers=true
        nodeFilters:
          - server:*
      - arg: --kube-scheduler-arg=feature-gates=EphemeralContainers=true
        nodeFilters:
          - server:*
      - arg: --kubelet-arg=feature-gates=EphemeralContainers=true
        nodeFilters:
          - agent:*