pod cycle restart,Pod sandbox changed, it will be killed and re-created. #4661
Description
Summary
I installed MicroK8s v1.30.4 revision 7167 on Ubuntu 24.04. Using microK8s status to check the status, MicroK8s is running, but when checking the status of pods, only the Calico node is running, and all other pods are ContainerCreating The ContainerCreating status has been going on for a long time. I have checked the detailed logs and the error reason is: Pod sandbox changed, it will be killed and re created. I used #1775 The method in the system involves stopping snap.microk8s.daemon-apiserver picker to stop the apiserver picker service, but the pod still restarts repeatedly. What should I do? thank you.
System:
Ubuntu 24.04
MicroK8s Version:
1.30.4
What Should Happen Instead?
All pods are in running state
Reproduction Steps
- sudo snap install microk8s --classic
- microk8s status --wait-ready
- microk8s enable dashboard
- microk8s enable dns
- microk8s kubectl get all --all-namespaces
- microk8s kubectl get pods --all-namespaces
Detail
> microk8s kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-796fb75cc-8h6m2 0/1 ContainerCreating 0 17h
kube-system calico-node-cf48j 1/1 Running 1 (16h ago) 17h
kube-system coredns-5986966c54-65tdp 0/1 ContainerCreating 0 17h
kube-system dashboard-metrics-scraper-795895d745-72jrs 0/1 ContainerCreating 0 17h
kube-system hostpath-provisioner-7c8bdf94b8-lgszw 0/1 ContainerCreating 0 17h
kube-system kubernetes-dashboard-6796797fb5-bdrsj 0/1 ContainerCreating 0 17h
kube-system metrics-server-7cff7889bd-hx2q6 0/1 ContainerCreating 0 17h
> microk8s version
MicroK8s v1.30.4 revision 7167
> microk8s kubectl describe pods kubernetes-dashboard-6796797fb5-bdrsj -n kube-system
Name: kubernetes-dashboard-6796797fb5-bdrsj
Namespace: kube-system
Priority: 0
Service Account: kubernetes-dashboard
Node: xxxx-pc/192.168.254.131
Start Time: Tue, 03 Sep 2024 22:17:25 +0800
Labels: k8s-app=kubernetes-dashboard
pod-template-hash=6796797fb5
Annotations: <none>
Status: Pending
SeccompProfile: RuntimeDefault
IP:
IPs: <none>
Controlled By: ReplicaSet/kubernetes-dashboard-6796797fb5
Containers:
kubernetes-dashboard:
Container ID:
Image: kubernetesui/dashboard:v2.7.0
Image ID:
Port: 8443/TCP
Host Port: 0/TCP
Args:
--auto-generate-certificates
--namespace=kube-system
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Liveness: http-get https://:8443/ delay=30s timeout=30s period=10s #success=1 #failure=3
Environment: <none>
Mounts:
/certs from kubernetes-dashboard-certs (rw)
/tmp from tmp-volume (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-2gs6r (ro)
Conditions:
Type Status
PodReadyToStartContainers False
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
kubernetes-dashboard-certs:
Type: Secret (a volume populated by a Secret)
SecretName: kubernetes-dashboard-certs
Optional: false
tmp-volume:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
kube-api-access-2gs6r:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: kubernetes.io/os=linux
Tolerations: node-role.kubernetes.io/master:NoSchedule
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SandboxChanged 63s (x224 over 16h) kubelet Pod sandbox changed, it will be killed and re-created.
> microk8s inspect
Inspecting system
Inspecting Certificates
Inspecting services
Service snap.microk8s.daemon-cluster-agent is running
Service snap.microk8s.daemon-containerd is running
Service snap.microk8s.daemon-kubelite is running
Service snap.microk8s.daemon-k8s-dqlite is running
FAIL: Service snap.microk8s.daemon-apiserver-kicker is not running
For more details look at: sudo journalctl -u snap.microk8s.daemon-apiserver-kicker
Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
Copy processes list to the final report tarball
Copy disk usage information to the final report tarball
Copy memory usage information to the final report tarball
Copy server uptime to the final report tarball
Copy openSSL information to the final report tarball
Copy snap list to the final report tarball
Copy VM name (or none) to the final report tarball
Copy current linux distribution to the final report tarball
Copy asnycio usage and limits to the final report tarball
Copy inotify max_user_instances and max_user_watches to the final report tarball
Copy network configuration to the final report tarball
Inspecting kubernetes cluster
Inspect kubernetes cluster
Inspecting dqlite
Inspect dqlite
cp: cannot stat '/var/snap/microk8s/7167/var/kubernetes/backend/localnode.yaml': No such file or directory
Building the report tarball
Report tarball is at /var/snap/microk8s/7167/inspection-report-20240904_155434.tar.gz