[QUESTION/HELP] Unable to kubectl exec the container

[wymli]

Question / Where do you need Help?

I want to go into the container, but failed to run kubectl exec in posting request to api server.
However crictl exec is ok to go into the container.

Detail

liwm29@wymli-NB1:~/bc_sns$ kubectl get pod
NAME                     READY   STATUS    RESTARTS   AGE
mysql-6778bd6487-lqc8l   1/1     Running   1          44m

liwm29@wymli-NB1:~/bc_sns$ kubectl exec -it mysql-6778bd6487-lqc8l -- bash
error: error sending request: Post "https://0.0.0.0:32879/api/v1/namespaces/default/pods/mysql-6778bd6487-lqc8l/exec?command=bash&container=mysql&stdin=true&stdout=true&tty=true": EOF

node log

There seems to be some issues with nodes simulated with docker

liwm29@wymli-NB1:~/bc_sns$ docker ps
CONTAINER ID   IMAGE                      COMMAND                  CREATED          STATUS          PORTS                             NAMES
a4619e154c38   rancher/k3d-proxy:5.2.2    "/bin/sh -c nginx-pr…"   47 minutes ago   Up 47 minutes   80/tcp, 0.0.0.0:32879->6443/tcp   k3d-bcsns-serverlb
041f9b0c5724   rancher/k3s:v1.21.7-k3s1   "/bin/k3s server --t…"   47 minutes ago   Up 47 minutes                                     k3d-bcsns-server-0

liwm29@wymli-NB1:~/bc_sns$ docker logs 041f9b0c5724   | tail
...
E0210 06:44:06.663346       7 summary_sys_containers.go:47] "Failed to get system container stats" err="failed to get cgroup stats for \"/docker/041f9b0c57249c8bef04db5191fb584c751abfded78a1a588f5fac81f689dd07\": failed to get container info for \"/docker/041f9b0c57249c8bef04db5191fb584c751abfded78a1a588f5fac81f689dd07\": unknown container \"/docker/041f9b0c57249c8bef04db5191fb584c751abfded78a1a588f5fac81f689dd07\"" containerName="/docker/041f9b0c57249c8bef04db5191fb584c751abfded78a1a588f5fac81f689dd07"
E0210 06:44:14.235675       7 summary_sys_containers.go:82] "Failed to get system container stats" err="failed to get cgroup stats for \"/docker/041f9b0c57249c8bef04db5191fb584c751abfded78a1a588f5fac81f689dd07\": failed to get container info for \"/docker/041f9b0c57249c8bef04db5191fb584c751abfded78a1a588f5fac81f689dd07\": unknown container \"/docker/041f9b0c57249c8bef04db5191fb584c751abfded78a1a588f5fac81f689dd07\"" containerName="/docker/041f9b0c57249c8bef04db5191fb584c751abfded78a1a588f5fac81f689dd07"
E0210 06:44:16.700775       7 summary_sys_containers.go:47] "Failed to get system container stats" err="failed to get cgroup stats for \"/docker/041f9b0c57249c8bef04db5191fb584c751abfded78a1a588f5fac81f689dd07\": failed to get container info for \"/docker/041f9b0c57249c8bef04db5191fb584c751abfded78a1a588f5fac81f689dd07\": unknown container \"/docker/041f9b0c57249c8bef04db5191fb584c751abfded78a1a588f5fac81f689dd07\"" containerName="/docker/041f9b0c57249c8bef04db5191fb584c751abfded78a1a588f5fac81f689dd07"
E0210 06:44:26.742367       7 summary_sys_containers.go:47] "Failed to get system container stats" err="failed to get cgroup stats for \"/docker/041f9b0c57249c8bef04db5191fb584c751abfded78a1a588f5fac81f689dd07\": failed to get container info for \"/docker/041f9b0c57249c8bef04db5191fb584c751abfded78a1a588f5fac81f689dd07\": unknown container \"/docker/041f9b0c57249c8bef04db5191fb584c751abfded78a1a588f5fac81f689dd07\"" containerName="/docker/041f9b0c57249c8bef04db5191fb584c751abfded78a1a588f5fac81f689dd07"
E0210 06:44:32.509863       7 proxier.go:1615] "Failed to execute iptables-restore" err="exit status 2 (ip6tables-restore v1.8.6 (legacy): unknown option \"--random-fully\"\nError occurred at line: 18\nTry `ip6tables-restore -h' or 'ip6tables-restore --help' for more information.\n)"
I0210 06:44:32.509905       7 proxier.go:862] "Sync failed" retryingTime="30s"

Crictl

I can access the mysql container using crictl exec after entering the docker node.

liwm29@wymli-NB1:~/bc_sns$ kubectl exec -it mysql-59588fd84c-67cn5 -- bash
error: error sending request: Post "https://0.0.0.0:32879/api/v1/namespaces/default/pods/mysql-59588fd84c-67cn5/exec?command=bash&container=mysql&stdin=true&stdout=true&tty=true": EOF

liwm29@wymli-NB1:~/bc_sns$ docker ps
CONTAINER ID   IMAGE                      COMMAND                  CREATED       STATUS       PORTS                             NAMES
a4619e154c38   rancher/k3d-proxy:5.2.2    "/bin/sh -c nginx-pr…"   6 hours ago   Up 6 hours   80/tcp, 0.0.0.0:32879->6443/tcp   k3d-bcsns-serverlb
041f9b0c5724   rancher/k3s:v1.21.7-k3s1   "/bin/k3s server --t…"   6 hours ago   Up 6 hours                                     k3d-bcsns-server-0

liwm29@wymli-NB1:~/bc_sns$ docker exec -it 041f9b0c5724 sh

/ # crictl ps
CONTAINER           IMAGE               CREATED             STATE               NAME                     ATTEMPT             POD ID
26622e350a9b5       dd3b2a5dcb48f       55 seconds ago      Running             mysql                    0                   d6efeae7f0cab
12935fe085613       deaf4b1027ed4       6 hours ago         Running             traefik                  0                   d512f1d189280
bb16edc748aeb       746788bcc27e2       6 hours ago         Running             lb-port-443              0                   bf63ab913c7e3
b8cb93830828b       746788bcc27e2       6 hours ago         Running             lb-port-80               0                   bf63ab913c7e3
2d78c86a87cc7       3885a5b7f138c       6 hours ago         Running             coredns                  0                   7ae9a68976c11
ed6ef67202eea       148c192562719       6 hours ago         Running             local-path-provisioner   0                   63058d35a7ab9
84c26e9f730fe       9dd718864ce61       6 hours ago         Running             metrics-server           0                   0ff00c704beda

/ # crictl exec -it 26622e350a9b5 bash

root@mysql-59588fd84c-67cn5:/# 

What have I tried

I tried to run k3d cluster delete and k3d cluster create to recreate a new cluster, but failed.

Scope of your Question

liwm29@wymli-NB1:~/bc_sns$ k3d version
k3d version v5.2.2
k3s version v1.21.7-k3s1 (default)

Thanks for any help!!!

[iwilltry42]

Hi @wymli , thanks for opening this issue!
This looks strange 🤔
Can you please give some more info on your environment? OS, docker version and kubectl version?