This repository was archived by the owner on Feb 12, 2021. It is now read-only.
This repository was archived by the owner on Feb 12, 2021. It is now read-only.
kubernetes network debugging guide #802
Open
Description
- toolbox and attaching to shell-less containers ( docs: Add description of how toolbox runs #803, kubernetes/: Add network troubleshooting doc #810)
- k8s-dns troubles (kubernetes/: Add network troubleshooting doc #810)
- iptables troubleshooting
- NAT troubleshooting
A common issue when getting started with Kubernetes is debugging networking. We need to provide a debugging guide for Kubernetes networking that covers the following topics:
DNS Debugging
Untested:
kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o template --template="{{range.items}}{{.metadata.name}}{{end}}" | xargs -I{} kubectl port-forward --namespace=kube-system {} 5300:53
dig something something
Bridge Debugging
ssh into a host and use toolbox + tcpdump to dump the flannel0/cbr0 bridge
Pod Debugging
Bash in the container
cd k8s.io/kubernetes/examples/guestbook
for i in *.yaml; do kubectl create -f ${i}; done
Find a pod you want to debug
kubectl get pods
$ kubectl exec -ti frontend-r1lq4 /bin/bash
root@frontend-r1lq4:/var/www/html# ping yahoo.com
PING yahoo.com (98.138.253.109): 56 data bytes
64 bytes from 98.138.253.109: icmp_seq=0 ttl=127 time=50.670 ms
64 bytes from 98.138.253.109: icmp_seq=1 ttl=127 time=43.191 ms
^C--- yahoo.com ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 43.191/46.931/50.670/3.740 ms
No bash
You will need to tutorial a busybox sleep container added to the pod. This is not exactly that but you get the idea:
$ kubectl run --image busybox tester -- /bin/sleep 5000
Error from server: deployments.extensions "tester" already exists
$ kubectl delete deployment tester
deployment "tester" deleted
$ kubectl run --image busybox tester -- /bin/sleep 5000
deployment "tester" created
$ kubectl get pod
NAME READY STATUS RESTARTS AGE
bash-3819658126-4r2r3 0/1 CrashLoopBackOff 5 4m
frontend-du0jv 1/1 Running 0 5m
frontend-k0ykp 1/1 Running 0 5m
frontend-r1lq4 1/1 Running 0 5m
redis-master-3djov 1/1 Running 0 5m
redis-slave-aran5 0/1 Pending 0 5m
redis-slave-u2hdj 1/1 Running 0 5m
tester-3286786242-7xidb 1/1 Running 0 4s
$ kubectl exec -ti tester-3286786242-7xidb /bin/bash
exec: "/bin/bash": stat /bin/bash: no such file or directory
error: error executing remote command: Error executing command in container: Error executing in Docker Container: -1
$ kubectl exec -ti tester-3286786242-7xidb /bin/sh
/ # ping yahoo.com
PING yahoo.com (206.190.36.45): 56 data bytes
64 bytes from 206.190.36.45: seq=0 ttl=127 time=35.360 ms
^C
--- yahoo.com ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 35.360/35.360/35.360 ms