Kube-dns not happy (kubernetes v1.10)

[phagunbaya]

Using kubeadm-1.10.0-0 to install kubernetes.

Kube-dns not able to talk to apiserver.

I0427 11:56:26.744252       1 dns.go:48] version: 1.14.6-3-gc36cb11
I0427 11:56:26.754567       1 server.go:69] Using configuration read from directory: /kube-dns-config with period 10s
I0427 11:56:26.754693       1 server.go:112] FLAG: --alsologtostderr="false"
I0427 11:56:26.754719       1 server.go:112] FLAG: --config-dir="/kube-dns-config"
I0427 11:56:26.754736       1 server.go:112] FLAG: --config-map=""
I0427 11:56:26.754752       1 server.go:112] FLAG: --config-map-namespace="kube-system"
I0427 11:56:26.754772       1 server.go:112] FLAG: --config-period="10s"
I0427 11:56:26.754800       1 server.go:112] FLAG: --dns-bind-address="0.0.0.0"
I0427 11:56:26.754813       1 server.go:112] FLAG: --dns-port="10053"
I0427 11:56:26.754831       1 server.go:112] FLAG: --domain="cluster.local."
I0427 11:56:26.754859       1 server.go:112] FLAG: --federations=""
I0427 11:56:26.754875       1 server.go:112] FLAG: --healthz-port="8081"
I0427 11:56:26.754930       1 server.go:112] FLAG: --initial-sync-timeout="1m0s"
I0427 11:56:26.754944       1 server.go:112] FLAG: --kube-master-url=""
I0427 11:56:26.754964       1 server.go:112] FLAG: --kubecfg-file=""
I0427 11:56:26.754975       1 server.go:112] FLAG: --log-backtrace-at=":0"
I0427 11:56:26.755007       1 server.go:112] FLAG: --log-dir=""
I0427 11:56:26.755020       1 server.go:112] FLAG: --log-flush-frequency="5s"
I0427 11:56:26.755033       1 server.go:112] FLAG: --logtostderr="true"
I0427 11:56:26.755044       1 server.go:112] FLAG: --nameservers=""
I0427 11:56:26.755055       1 server.go:112] FLAG: --stderrthreshold="2"
I0427 11:56:26.755085       1 server.go:112] FLAG: --v="2"
I0427 11:56:26.755096       1 server.go:112] FLAG: --version="false"
I0427 11:56:26.755138       1 server.go:112] FLAG: --vmodule=""
I0427 11:56:26.755438       1 server.go:194] Starting SkyDNS server (0.0.0.0:10053)
I0427 11:56:26.756504       1 server.go:213] Skydns metrics enabled (/metrics:10055)
I0427 11:56:26.756554       1 dns.go:146] Starting endpointsController
I0427 11:56:26.756568       1 dns.go:149] Starting serviceController
I0427 11:56:26.757954       1 logs.go:41] skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053 [rcache 0]
I0427 11:56:26.758043       1 logs.go:41] skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053 [rcache 0]
I0427 11:56:27.260334       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0427 11:56:27.760223       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...

weave-net/weave logs

WARN: 2018/04/27 11:57:02.061581 Vetoed installation of hairpin flow FlowSpec{keys: [EthernetFlowKey{src: ba:0e:db:f2:f0:83, dst: 56:fa:1f:ea:49:46} InPortFlowKey{vport: 1}], actions: [OutputAction{vport: 1}]}

$ docker version

Client:
 Version:      17.03.1-ce
 API version:  1.27
 Go version:   go1.7.5
 Git commit:   c6d412e
 Built:        Mon Mar 27 17:05:44 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.03.1-ce
 API version:  1.27 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   c6d412e
 Built:        Mon Mar 27 17:05:44 2017
 OS/Arch:      linux/amd64
 Experimental: false

$ uname -a

Linux termxanltcapp01 3.10.0-327.el7.x86_64 #1 SMP Thu Oct 29 17:29:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux

$ kubectl version 1.10.0

Network:

$ ip route

default via 10.190.159.1 dev ens192  proto static  metric 100
10.32.0.0/12 dev weave  proto kernel  scope link  src 10.32.0.1
10.190.159.0/24 dev ens192  proto kernel  scope link  src 10.190.159.226  metric 100
172.17.0.0/16 dev docker0  proto kernel  scope link  src 172.17.0.1
192.168.122.0/24 dev virbr0  proto kernel  scope link  src 192.168.122.1

$ ip -4 -o addr

1: lo    inet 127.0.0.1/8 scope host lo\       valid_lft forever preferred_lft forever
2: ens192    inet 10.190.159.226/24 brd 10.190.159.255 scope global ens192\       valid_lft forever preferred_lft forever
3: virbr0    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0\       valid_lft forever preferred_lft forever
5: docker0    inet 172.17.0.1/16 scope global docker0\       valid_lft forever preferred_lft forever
7: weave    inet 10.32.0.1/12 brd 10.47.255.255 scope global weave\       valid_lft forever preferred_lft forever

$ sudo iptables-save

# Completed on Fri Apr 27 07:11:17 2018
# Generated by iptables-save v1.4.21 on Fri Apr 27 07:11:17 2018
*nat
:PREROUTING ACCEPT [635:33218]
:INPUT ACCEPT [635:33218]
:OUTPUT ACCEPT [13:806]
:POSTROUTING ACCEPT [13:806]
:DOCKER - [0:0]
:KUBE-MARK-DROP - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-NODEPORTS - [0:0]
:KUBE-POSTROUTING - [0:0]
:KUBE-SEP-EW3FHQXL3IP6YAQP - [0:0]
:KUBE-SERVICES - [0:0]
:KUBE-SVC-ERIFXISQEP7F7OF4 - [0:0]
:KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]
:KUBE-SVC-TCOU7JCQXEZGVUNU - [0:0]
:WEAVE - [0:0]
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 192.168.122.0/24 -d 224.0.0.0/24 -j RETURN
-A POSTROUTING -s 192.168.122.0/24 -d 255.255.255.255/32 -j RETURN
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p udp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -j MASQUERADE
-A POSTROUTING -j WEAVE
-A DOCKER -i docker0 -j RETURN
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE
-A KUBE-SEP-EW3FHQXL3IP6YAQP -s 10.190.159.226/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ
-A KUBE-SEP-EW3FHQXL3IP6YAQP -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-EW3FHQXL3IP6YAQP --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 10.190.159.226:6443
-A KUBE-SERVICES -d 10.96.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-SVC-TCOU7JCQXEZGVUNU
-A KUBE-SERVICES -d 10.96.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-SVC-ERIFXISQEP7F7OF4
-A KUBE-SERVICES -d 10.96.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-EW3FHQXL3IP6YAQP --mask 255.255.255.255 --rsource -j KUBE-SEP-EW3FHQXL3IP6YAQP
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -j KUBE-SEP-EW3FHQXL3IP6YAQP
-A WEAVE -s 10.32.0.0/12 -d 224.0.0.0/4 -j RETURN
-A WEAVE ! -s 10.32.0.0/12 -d 10.32.0.0/12 -j MASQUERADE
-A WEAVE -s 10.32.0.0/12 ! -d 10.32.0.0/12 -j MASQUERADE
COMMIT
# Completed on Fri Apr 27 07:11:17 2018
# Generated by iptables-save v1.4.21 on Fri Apr 27 07:11:17 2018
*filter
:INPUT ACCEPT [2351:475783]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [2105:402511]
:DOCKER - [0:0]
:DOCKER-ISOLATION - [0:0]
:KUBE-FIREWALL - [0:0]
:KUBE-FORWARD - [0:0]
:KUBE-SERVICES - [0:0]
:WEAVE-NPC - [0:0]
:WEAVE-NPC-DEFAULT - [0:0]
:WEAVE-NPC-INGRESS - [0:0]
-A INPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A INPUT -j KUBE-FIREWALL
-A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT
-A FORWARD -o weave -m comment --comment "NOTE: this must go before \'-j KUBE-FORWARD\'" -j WEAVE-NPC
-A FORWARD -o weave -m state --state NEW -j NFLOG --nflog-group 86
-A FORWARD -o weave -j DROP
-A FORWARD -i weave ! -o weave -j ACCEPT
-A FORWARD -o weave -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -m comment --comment "kubernetes forward rules" -j KUBE-FORWARD
-A FORWARD -j DOCKER-ISOLATION
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -d 192.168.122.0/24 -o virbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 192.168.122.0/24 -i virbr0 -j ACCEPT
-A FORWARD -i virbr0 -o virbr0 -j ACCEPT
-A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -j KUBE-FIREWALL
-A OUTPUT -o virbr0 -p udp -m udp --dport 68 -j ACCEPT
-A DOCKER-ISOLATION -j RETURN
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding rules" -m mark --mark 0x4000/0x4000 -j ACCEPT
-A KUBE-SERVICES -d 10.96.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns has no endpoints" -m udp --dport 53 -j REJECT --reject-with icmp-port-unreachable
-A KUBE-SERVICES -d 10.96.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp has no endpoints" -m tcp --dport 53 -j REJECT --reject-with icmp-port-unreachable
-A WEAVE-NPC -m state --state RELATED,ESTABLISHED -j ACCEPT
-A WEAVE-NPC -d 224.0.0.0/4 -j ACCEPT
-A WEAVE-NPC -m state --state NEW -j WEAVE-NPC-DEFAULT
-A WEAVE-NPC -m state --state NEW -j WEAVE-NPC-INGRESS
-A WEAVE-NPC -m set ! --match-set weave-local-pods dst -j ACCEPT
-A WEAVE-NPC-DEFAULT -m set --match-set weave-E.1.0W^NGSp]0_t5WwH/]gX@L dst -m comment --comment "DefaultAllow isolation for namespace: default" -j ACCEPT
-A WEAVE-NPC-DEFAULT -m set --match-set weave-?b%zl9GIe0AET1(QI^7NWe*fO dst -m comment --comment "DefaultAllow isolation for namespace: kube-system" -j ACCEPT
-A WEAVE-NPC-DEFAULT -m set --match-set weave-0EHD/vdN#O4]V?o4Tx7kS;APH dst -m comment --comment "DefaultAllow isolation for namespace: kube-public" -j ACCEPT

[phagunbaya]

kubeadm just warning about crictl missing. Otherwise, there is no warning for IP overlaps.

[phagunbaya]

It's even happening with kubernetes 1.7.5. Any help would be much appreciated. The whole setup works fine of other machine and not just this one. I want to know what's the case.

[bboreham]

Please give the logs from the Weave container.

[bboreham]

What exactly is the basis for your assertion in the issue title? I don’t see any error messages.

Can you check if ip forwarding is turned on - sysctl net.ipv4.ip_forward

Is there some background to using kernel 3.10? On the face of it that’s really old, unless it’s a redhat variant.

[phagunbaya]

kube-dns pod goes under ContainerCrashing state. IP forwarding is ON. Kestrel is 3.10 and it is RHeL 7.4. Another thing I noticed is that flannel cni plugin works fine for this case.

[bboreham]

Cannot help you further if you do not provide the requested logs.

[brb]

Closing due to missing logs.