weavenet daemonset failed to start: address already in use

[yongzhang]

Hi,

I have Kubernetes cluster 1.11.3 with weavenet 2.4.1, sometimes when I create a new node, weavenet failed to start with errors below:

DEBU: 2018/11/23 14:50:50.126789 [kube-peers] Checking peer "56:eb:5d:e6:5b:89" against list &{[{ce:90:a9:dd:38:7d 10.10.100.12} {8e:cb:54:8a:36:c5 10.10.100.10} {4e:72:ac:57:c3:fd 10.10.100.11} {1e:61:b6:f7:af:00 10.10.100.7} {5e:95:3c:c2:28:94 10.10.100.9} {2a:f3:62:05:39:96 10.10.102.9} {ca:25:73:9b:ff:4c 10.10.101.5}]}
Peer not in list; removing persisted data
INFO: 2018/11/23 14:50:50.248345 Command line options: map[conn-limit:100 host-root:/host ipalloc-init:consensus=8 no-dns:true port:6783 name:56:eb:5d:e6:5b:89 expect-npc:true http-addr:127.0.0.1:6784 ipalloc-range:10.42.0.0/16 metrics-addr:0.0.0.0:6782 nickname:10.10.102.3 datapath:datapath db-prefix:/weavedb/weave-net docker-api:]
INFO: 2018/11/23 14:50:50.248409 weave  2.4.1
INFO: 2018/11/23 14:50:50.324735 Bridge type is bridged_fastdp
INFO: 2018/11/23 14:50:50.324755 Communication between peers via untrusted networks is encrypted.
INFO: 2018/11/23 14:50:50.364984 Our name is 56:eb:5d:e6:5b:89(10.10.102.3)
INFO: 2018/11/23 14:50:50.365020 Launch detected - using supplied peer list: [10.10.100.10 10.10.100.11 10.10.100.12 10.10.100.7 10.10.100.9 10.10.101.5 10.10.102.3 10.10.102.9]
INFO: 2018/11/23 14:50:50.365043 Checking for pre-existing addresses on weave bridge
INFO: 2018/11/23 14:50:50.381203 [allocator 56:eb:5d:e6:5b:89] No valid persisted data
INFO: 2018/11/23 14:50:50.398851 [allocator 56:eb:5d:e6:5b:89] Initialising via deferred consensus
INFO: 2018/11/23 14:50:50.398903 Sniffing traffic on datapath (via ODP)
INFO: 2018/11/23 14:50:50.399154 ->[10.10.102.9:6783] attempting connection
INFO: 2018/11/23 14:50:50.399980 ->[10.10.101.5:6783] attempting connection
INFO: 2018/11/23 14:50:50.400071 ->[10.10.102.3:6783] attempting connection
INFO: 2018/11/23 14:50:50.400135 ->[10.10.100.12:6783] attempting connection
INFO: 2018/11/23 14:50:50.400163 ->[10.10.100.10:6783] attempting connection
INFO: 2018/11/23 14:50:50.400192 ->[10.10.100.11:6783] attempting connection
INFO: 2018/11/23 14:50:50.400316 ->[10.10.102.3:41802] connection accepted
INFO: 2018/11/23 14:50:50.400824 ->[10.10.102.3:41802|56:eb:5d:e6:5b:89(10.10.102.3)]: connection shutting down due to error: cannot connect to ourself
INFO: 2018/11/23 14:50:50.400940 ->[10.10.102.3:6783|56:eb:5d:e6:5b:89(10.10.102.3)]: connection shutting down due to error: cannot connect to ourself
INFO: 2018/11/23 14:50:50.401006 ->[10.10.100.7:6783] attempting connection
INFO: 2018/11/23 14:50:50.401044 ->[10.10.100.9:6783] attempting connection
INFO: 2018/11/23 14:50:50.401881 ->[10.10.102.9:6783|2a:f3:62:05:39:96(10.10.102.9)]: connection ready; using protocol version 2
INFO: 2018/11/23 14:50:50.402002 overlay_switch ->[2a:f3:62:05:39:96(10.10.102.9)] using fastdp
INFO: 2018/11/23 14:50:50.402032 ->[10.10.102.9:6783|2a:f3:62:05:39:96(10.10.102.9)]: connection added (new peer)
INFO: 2018/11/23 14:50:50.402106 Setting up IPsec between 56:eb:5d:e6:5b:89(10.10.102.3) and 2a:f3:62:05:39:96(10.10.102.9)
INFO: 2018/11/23 14:50:50.402190 ipsec: InitSALocal: 10.10.102.9 -> 10.10.102.3 :6784 0x231004a0
INFO: 2018/11/23 14:50:50.405373 Listening for HTTP control messages on 127.0.0.1:6784
INFO: 2018/11/23 14:50:50.405506 Listening for metrics requests on 0.0.0.0:6782
FATA: 2018/11/23 14:50:50.405822 Unable to create http listener socket: listen tcp 0.0.0.0:6782: bind: address already in use

CONTAINER ID        IMAGE                                                                                           COMMAND                  CREATED             STATUS                     PORTS               NAMES
5bedbdaab5d3        weaveworks/weave-kube@sha256:e061cf1bec08c985bf235c4de88acaa3f32d80658bc1a17cab76d4d0663cce57   "/home/weave/launc..."   4 minutes ago       Exited (1) 4 minutes ago                       k8s_weave_weave-net-flv5x_kube-system_9913f24f-ef2d-11e8-9f5a-525400b80858_6
1a0df7b620ba        weaveworks/weave-npc@sha256:a11fb31e3a3a9b62d96dbf96bcf379abb9caf3d745134c926972197159c3b861    "/usr/bin/weave-npc"     10 minutes ago      Up 10 minutes       

After server reboot, it can back to work:

55f6e7bcd4dd        weaveworks/weave-npc@sha256:a11fb31e3a3a9b62d96dbf96bcf379abb9caf3d745134c926972197159c3b861    "/usr/bin/weave-npc"     48 seconds ago      Up 47 seconds                                       k8s_weave-npc_weave-net-flv5x_kube-system_9913f24f-ef2d-11e8-9f5a-525400b80858_1
732f96ac3324        weaveworks/weave-kube@sha256:e061cf1bec08c985bf235c4de88acaa3f32d80658bc1a17cab76d4d0663cce57   "/home/weave/launc..."   50 seconds ago      Up 49 seconds                                       k8s_weave_weave-net-flv5x_kube-system_9913f24f-ef2d-11e8-9f5a-525400b80858_11

Please help, thanks.

[murali-reddy]

sometimes when I create a new node, weavenet failed to start with errors below:

Are you running into this issue often?

FATA: 2018/11/23 14:50:50.405822 Unable to create http listener socket: listen tcp 0.0.0.0:6782: bind: address already in use

As you noted this is because port is already is use. If you encounter this issue again please check sudo netstat -pna | grep 6782 and see if there is a process listening on the port.

[yongzhang]

yes, there's indeed a port listening there, but this is a clean node, I don't think there's any other application listening on that port.

[murali-reddy]

Sure there may not other application listening on that port. But its likely weave process is the one holding the port. When you run into this issue again please check if its weave process that is holding the port and also check if for any reason pod failed to start and it went into crash loop etc.