Nodes are attempting to claim same IP range

[zacblazic]

What you expected to happen?

Nodes should claim their own distinct IP ranges.

What happened?

It seems that two nodes have (or are attempting to) claim the same IP address range.

The node ip-10-83-42-111.ec2.internal claimed the 100.105.68.0 IP range:

...
{
    "Token": "100.105.68.0",
    "Size": 1024,
    "Peer": "12:84:ba:e1:47:79",
    "Nickname": "ip-10-83-42-111.ec2.internal",
    "IsKnownPeer": true,
    "Version": 1451
}
...

However, the node ip-10-83-54-199.ec2.internal is also attempting to claim the 100.105.68.0 IP range:

$ ./weave --local status connections
-> 10.83.80.134:6783     failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:15:31.000788717 +0000 UTC m=+1034.303622109
-> 10.83.98.46:6783      failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:11:04.422577727 +0000 UTC m=+767.725411089
-> 10.83.126.221:6783    failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:11:33.742999475 +0000 UTC m=+797.045832929
-> 10.83.85.183:6783     failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:08:47.877010584 +0000 UTC m=+631.179843953
-> 10.83.60.42:6783      failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:09:09.444156631 +0000 UTC m=+652.746990055
-> 10.83.78.215:6783     failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:11:40.747663238 +0000 UTC m=+804.050496608
-> 10.83.124.188:6783    failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:10:38.016883177 +0000 UTC m=+741.319716603
-> 10.83.42.111:6783     failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:14:14.702258004 +0000 UTC m=+958.005091377
-> 10.83.109.188:6783    failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:11:40.776242153 +0000 UTC m=+804.079075513
-> 10.83.36.17:6783      failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:10:19.676858864 +0000 UTC m=+722.979692293
-> 10.83.59.237:6783     failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:08:27.22810513 +0000 UTC m=+610.530938541
-> 10.83.41.74:6783      failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:11:44.830033868 +0000 UTC m=+808.132867307
-> 10.83.54.199:6783     failed      cannot connect to ourself, retry: never
-> 10.83.87.92:6783      failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:10:32.774426814 +0000 UTC m=+736.077260261
-> 10.83.76.130:6783     failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:10:48.434728192 +0000 UTC m=+751.737561589

Restarting the weave pods on either ip-10-83-42-111.ec2.internal or ip-10-83-54-199.ec2.internal does not change the situation, likely due to the the fact that state is persisted on disk at /var/lib/weave.

---

During the same time period, there was another issue with weave I noticed. I'm not sure if it is related but some nodes appeared to be attempting to remove a non-existent peer:

DEBU: 2018/06/05 11:20:29.571705 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:29.571731 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:29.571739 [kube-peers] Existing annotation 46:f4:4b:41:dd:11

According to our logs, this error started on 2018/05/16 (~21 days ago). Around this time we were having issues with weave consuming a large amount of memory.

How to reproduce it?

Not sure if it is easily reproducible, but it occurred while restarting all weave pods in the cluster, with a 15 second sleep between each restart.

There is a possibility that two pods started at the same time due to scheduling delays.

Anything else we need to know?

Cloud provider: AWS, managed by Kops

Versions:

$ ./weave --local version
weave 2.3.0

$ docker version
Client:
 Version:      17.03.2-ce
 API version:  1.27
 Go version:   go1.7.5
 Git commit:   f5ec1e2
 Built:        Tue Jun 27 02:09:56 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.03.2-ce
 API version:  1.27 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   f5ec1e2
 Built:        Tue Jun 27 02:09:56 2017
 OS/Arch:      linux/amd64
 Experimental: false

$ uname -a
Linux ip-10-83-54-199 4.4.115-k8s #1 SMP Thu Feb 8 15:37:40 UTC 2018 x86_64 GNU/Linux

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.13", GitCommit:"290fb182489a396dce5d136451388f9b12f29c94", GitTreeState:"clean", BuildDate:"2018-05-15T18:16:09Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.13", GitCommit:"290fb182489a396dce5d136451388f9b12f29c94", GitTreeState:"clean", BuildDate:"2018-05-15T18:08:47Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

$ kops version
Version 1.9.1

Logs:

Click the black arrows to reveal the details!

From the node with the issue (ip-10-83-54-199.ec2.internal):

Weave IPAM

$ ./weave --local status ipam
6a:5c:7c:af:e3:2e(ip-10-83-54-199.ec2.internal)   297040 IPs (14.2% of total) (4 active)
46:f4:4b:41:dd:11(ip-10-83-124-112.ec2.internal)    65536 IPs (03.1% of total) - unreachable!
fe:6a:75:3b:98:1f(ip-10-83-123-141.ec2.internal)     8192 IPs (00.4% of total) - unreachable!
7a:54:95:96:77:8e(ip-10-83-98-15.ec2.internal)    65536 IPs (03.1% of total) - unreachable!
c2:35:ec:05:d8:a5()                      65536 IPs (03.1% of total) - unreachable!
1a:cd:c1:53:24:bf(ip-10-83-36-17.ec2.internal)    16384 IPs (00.8% of total) - unreachable!
1e:dd:1b:3a:d3:11(ip-10-83-109-170.ec2.internal)    65536 IPs (03.1% of total) - unreachable!
3e:06:82:a1:9c:ec(ip-10-83-103-30.ec2.internal)    66048 IPs (03.1% of total) - unreachable!
1e:1c:57:4a:9e:66(ip-10-83-52-111.ec2.internal)     8192 IPs (00.4% of total) - unreachable!
fe:9a:a2:ec:04:20(ip-10-83-117-100.ec2.internal)    65536 IPs (03.1% of total) - unreachable!
2e:60:25:ef:21:f3(ip-10-83-125-194.ec2.internal)    32768 IPs (01.6% of total) - unreachable!
22:e1:23:b2:2f:8c(ip-10-83-59-237.ec2.internal)     2048 IPs (00.1% of total) - unreachable!
92:6b:5f:82:d0:24(ip-10-83-41-63.ec2.internal)    98304 IPs (04.7% of total) - unreachable!
6e:89:89:29:40:32(ip-10-83-98-46.ec2.internal)    16384 IPs (00.8% of total) - unreachable!
fa:ea:3e:94:41:e0(ip-10-83-85-183.ec2.internal)    16384 IPs (00.8% of total) - unreachable!
16:67:3c:ec:79:79(ip-10-83-47-111.ec2.internal)    16384 IPs (00.8% of total) - unreachable!
ba:f1:81:93:ab:b4(ip-10-83-32-140.ec2.internal)    32768 IPs (01.6% of total) - unreachable!
8a:9b:87:49:9d:5b(ip-10-83-67-78.ec2.internal)     8192 IPs (00.4% of total) - unreachable!
9a:55:19:b7:97:7c(ip-10-83-76-130.ec2.internal)     2048 IPs (00.1% of total) - unreachable!
f2:39:2f:d4:16:d7(ip-10-83-72-203.ec2.internal)    30720 IPs (01.5% of total) - unreachable!
ee:c9:d9:9b:bd:ec(ip-10-83-78-215.ec2.internal)     4608 IPs (00.2% of total) - unreachable!
9e:d3:fb:04:82:af(ip-10-83-109-104.ec2.internal)    16384 IPs (00.8% of total) - unreachable!
7e:44:ff:00:f5:f9(ip-10-83-60-42.ec2.internal)     4096 IPs (00.2% of total) - unreachable!
8e:71:01:6f:cd:9f(ip-10-83-73-41.ec2.internal)    16384 IPs (00.8% of total) - unreachable!
86:a9:66:cb:4a:c6(ip-10-83-85-200.ec2.internal)    16384 IPs (00.8% of total) - unreachable!
f6:b4:71:ad:ba:22(ip-10-83-108-91.ec2.internal)    65536 IPs (03.1% of total) - unreachable!
f2:11:ce:93:da:0b(ip-10-83-34-95.ec2.internal)    32768 IPs (01.6% of total) - unreachable!
0e:8c:f8:33:67:7f(ip-10-83-87-92.ec2.internal)    16384 IPs (00.8% of total) - unreachable!
6e:1b:67:ca:e9:72(ip-10-83-101-85.ec2.internal)    65536 IPs (03.1% of total) - unreachable!
7a:fb:6b:30:aa:2a(ip-10-83-106-172.ec2.internal)    32768 IPs (01.6% of total) - unreachable!
72:bf:6c:d7:7a:03(ip-10-83-85-193.ec2.internal)    32768 IPs (01.6% of total) - unreachable!
3a:47:b8:a9:34:ef(ip-10-83-126-221.ec2.internal)    19456 IPs (00.9% of total) - unreachable!
3a:5f:b6:6e:dd:bb(ip-10-83-80-134.ec2.internal)    15360 IPs (00.7% of total) - unreachable!
fe:f3:20:f4:5a:52(ip-10-83-60-125.ec2.internal)    65536 IPs (03.1% of total) - unreachable!
12:ea:66:1b:70:3d(ip-10-83-38-5.ec2.internal)     8192 IPs (00.4% of total) - unreachable!
ae:9e:35:14:6c:0c(ip-10-83-109-171.ec2.internal)    32768 IPs (01.6% of total) - unreachable!
86:d8:10:87:a0:b1(ip-10-83-126-77.ec2.internal)    32768 IPs (01.6% of total) - unreachable!
26:00:36:0b:01:55(ip-10-83-51-193.ec2.internal)    32768 IPs (01.6% of total) - unreachable!
92:55:13:0d:72:f7(ip-10-83-96-158.ec2.internal)     2048 IPs (00.1% of total) - unreachable!
66:15:ce:c7:af:ab(ip-10-83-96-156.ec2.internal)   131072 IPs (06.2% of total) - unreachable!
9a:5d:d4:e9:78:29(ip-10-83-109-188.ec2.internal)     1024 IPs (00.0% of total) - unreachable!
8e:02:27:40:f8:fd(ip-10-83-106-134.ec2.internal)    15872 IPs (00.8% of total) - unreachable!
36:35:70:69:df:eb(ip-10-83-50-197.ec2.internal)    32768 IPs (01.6% of total) - unreachable!
4a:d5:9f:aa:e3:99(ip-10-83-124-188.ec2.internal)     8192 IPs (00.4% of total) - unreachable!
02:77:dd:50:7b:3a(ip-10-83-63-111.ec2.internal)    32768 IPs (01.6% of total) - unreachable!
16:d8:40:86:d6:bf(ip-10-83-41-74.ec2.internal)     6576 IPs (00.3% of total) - unreachable!
16:e5:a9:4e:45:a4(ip-10-83-115-236.ec2.internal)   327680 IPs (15.6% of total) - unreachable!
9a:8b:af:32:3b:0d(ip-10-83-81-239.ec2.internal)    16384 IPs (00.8% of total) - unreachable!
62:03:63:9c:81:8c(ip-10-83-124-69.ec2.internal)    32768 IPs (01.6% of total) - unreachable!

Weave connections

$ ./weave --local status connections
-> 10.83.80.134:6783     failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:15:31.000788717 +0000 UTC m=+1034.303622109
-> 10.83.98.46:6783      failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:11:04.422577727 +0000 UTC m=+767.725411089
-> 10.83.126.221:6783    failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:11:33.742999475 +0000 UTC m=+797.045832929
-> 10.83.85.183:6783     failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:08:47.877010584 +0000 UTC m=+631.179843953
-> 10.83.60.42:6783      failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:09:09.444156631 +0000 UTC m=+652.746990055
-> 10.83.78.215:6783     failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:11:40.747663238 +0000 UTC m=+804.050496608
-> 10.83.124.188:6783    failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:10:38.016883177 +0000 UTC m=+741.319716603
-> 10.83.42.111:6783     failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:14:14.702258004 +0000 UTC m=+958.005091377
-> 10.83.109.188:6783    failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:11:40.776242153 +0000 UTC m=+804.079075513
-> 10.83.36.17:6783      failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:10:19.676858864 +0000 UTC m=+722.979692293
-> 10.83.59.237:6783     failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:08:27.22810513 +0000 UTC m=+610.530938541
-> 10.83.41.74:6783      failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:11:44.830033868 +0000 UTC m=+808.132867307
-> 10.83.54.199:6783     failed      cannot connect to ourself, retry: never
-> 10.83.87.92:6783      failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:10:32.774426814 +0000 UTC m=+736.077260261
-> 10.83.76.130:6783     failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:10:48.434728192 +0000 UTC m=+751.737561589

Weave logs

$ kubectl logs -f -n=kube-system weave-net-5qbw9  -c=weave
DEBU: 2018/06/05 11:15:30.215989 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:30.215999 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:30.415837 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:30.415863 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:30.415872 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:30.616648 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:30.616686 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:30.616699 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:30.816246 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:30.816273 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:30.816284 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
INFO: 2018/06/05 11:15:31.001044 ->[10.83.80.134:6783] attempting connection
INFO: 2018/06/05 11:15:31.002845 ->[10.83.80.134:6783|3a:5f:b6:6e:dd:bb(ip-10-83-80-134.ec2.internal)]: connection ready; using protocol version 2
INFO: 2018/06/05 11:15:31.002919 overlay_switch ->[3a:5f:b6:6e:dd:bb(ip-10-83-80-134.ec2.internal)] using fastdp
INFO: 2018/06/05 11:15:31.002973 ->[10.83.80.134:6783|3a:5f:b6:6e:dd:bb(ip-10-83-80-134.ec2.internal)]: connection added (new peer)
INFO: 2018/06/05 11:15:31.013028 ->[10.83.80.134:6783|3a:5f:b6:6e:dd:bb(ip-10-83-80-134.ec2.internal)]: connection shutting down due to error: Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460
INFO: 2018/06/05 11:15:31.013113 ->[10.83.80.134:6783|3a:5f:b6:6e:dd:bb(ip-10-83-80-134.ec2.internal)]: connection deleted
INFO: 2018/06/05 11:15:31.013140 Removed unreachable peer 12:84:ba:e1:47:79(ip-10-83-42-111.ec2.internal)
INFO: 2018/06/05 11:15:31.013151 Removed unreachable peer 7e:44:ff:00:f5:f9(ip-10-83-60-42.ec2.internal)
INFO: 2018/06/05 11:15:31.013160 Removed unreachable peer 9a:55:19:b7:97:7c(ip-10-83-76-130.ec2.internal)
INFO: 2018/06/05 11:15:31.013168 Removed unreachable peer 3a:5f:b6:6e:dd:bb(ip-10-83-80-134.ec2.internal)
INFO: 2018/06/05 11:15:31.013176 Removed unreachable peer 22:e1:23:b2:2f:8c(ip-10-83-59-237.ec2.internal)
INFO: 2018/06/05 11:15:31.013185 Removed unreachable peer 0e:8c:f8:33:67:7f(ip-10-83-87-92.ec2.internal)
INFO: 2018/06/05 11:15:31.013192 Removed unreachable peer ee:c9:d9:9b:bd:ec(ip-10-83-78-215.ec2.internal)
INFO: 2018/06/05 11:15:31.013198 Removed unreachable peer 1a:cd:c1:53:24:bf(ip-10-83-36-17.ec2.internal)
INFO: 2018/06/05 11:15:31.013204 Removed unreachable peer 9a:5d:d4:e9:78:29(ip-10-83-109-188.ec2.internal)
INFO: 2018/06/05 11:15:31.013209 Removed unreachable peer 16:d8:40:86:d6:bf(ip-10-83-41-74.ec2.internal)
INFO: 2018/06/05 11:15:31.013215 Removed unreachable peer 4a:d5:9f:aa:e3:99(ip-10-83-124-188.ec2.internal)
INFO: 2018/06/05 11:15:31.013221 Removed unreachable peer fa:ea:3e:94:41:e0(ip-10-83-85-183.ec2.internal)
INFO: 2018/06/05 11:15:31.013226 Removed unreachable peer 3a:47:b8:a9:34:ef(ip-10-83-126-221.ec2.internal)
INFO: 2018/06/05 11:15:31.013232 Removed unreachable peer 6e:89:89:29:40:32(ip-10-83-98-46.ec2.internal)
DEBU: 2018/06/05 11:15:31.016697 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:31.016753 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:31.016777 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:31.215970 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:31.215996 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:31.216007 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:31.416197 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:31.416222 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:31.416231 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:31.616070 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:31.616096 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:31.616105 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:31.815906 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:31.815931 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:31.815941 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:32.015766 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:32.015791 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:32.015801 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:32.216196 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:32.216221 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:32.216231 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:32.415785 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:32.415811 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:32.415821 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:32.615892 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:32.615919 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:32.615928 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:32.815841 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:32.815867 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:32.815877 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:33.016143 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:33.016172 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:33.016182 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:33.216024 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:33.216049 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:33.216060 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:33.416010 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:33.416037 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:33.416046 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:33.616145 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:33.616173 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:33.616182 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:33.815812 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:33.815836 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:33.815847 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:34.017516 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:34.017542 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:34.017552 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:34.216132 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:34.216156 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:34.216167 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:34.415847 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:34.415872 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:34.415882 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:34.615669 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:34.615696 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:34.615706 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:34.815762 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:34.815788 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:34.815798 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:35.015882 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:35.015909 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:35.015919 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:35.215745 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:35.215770 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:35.215780 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:35.415768 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:35.415793 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:35.415803 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:35.615818 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:35.615843 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:35.615853 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:35.815760 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:35.815785 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:35.815796 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:36.015913 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:36.015940 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:36.015950 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:36.215677 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:36.215705 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:36.215715 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:36.415952 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:36.415978 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:36.415988 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:36.616091 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:36.616116 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:36.616127 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:36.816059 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:36.816086 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:36.816099 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:37.016210 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:37.016237 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:37.016246 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:37.216211 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:37.216237 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:37.216247 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:37.415755 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:37.415781 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:37.415791 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:37.615952 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:37.615978 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:37.615988 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:37.815813 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:37.815838 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:37.815848 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:38.016128 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:38.016157 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:38.016167 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:38.216355 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:38.216380 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:38.216391 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:38.415765 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:38.415788 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:38.415798 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:38.616028 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:38.616055 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:38.616065 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:38.818819 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:38.818845 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:38.818854 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:39.016368 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:39.016394 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:39.016404 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:39.215962 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:39.215986 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:39.215995 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:39.415718 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:39.415745 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:39.415754 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:39.616077 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:39.616102 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:39.616117 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:39.815972 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:39.815998 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:39.816008 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:40.016211 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:40.016238 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:40.016248 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:40.216860 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:40.216886 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:40.216897 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:40.416162 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:40.416187 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:40.416197 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:40.616059 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:40.616083 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:40.616093 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:40.816285 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:40.816312 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:40.816322 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:41.016070 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:41.016098 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:41.016109 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:41.216637 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:41.216663 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:41.216673 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:41.416124 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:41.416188 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:41.416220 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:41.616358 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:41.616383 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:41.616393 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:41.815816 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:41.815841 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:41.815850 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:42.016532 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:42.016558 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:42.016568 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:42.217878 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:42.217905 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:42.217919 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:42.415987 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:42.416019 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:42.416033 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:42.616205 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:42.616230 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:42.616240 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:42.816255 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:42.816280 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:42.816290 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:43.016187 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:43.016212 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:43.016223 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:43.216480 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:43.216503 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:43.216513 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:43.416036 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:43.416063 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:43.416074 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:43.615932 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:43.615960 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:43.615971 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:43.815733 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]

From a node that has already claimed the IP range (ip-10-83-42-111.ec2.internal):

Weave IPAM

$ ./weave --local status ipam
12:84:ba:e1:47:79(ip-10-83-42-111.ec2.internal)     1024 IPs (00.0% of total) (61 active)
16:d8:40:86:d6:bf(ip-10-83-41-74.ec2.internal)   334256 IPs (15.9% of total)
3a:47:b8:a9:34:ef(ip-10-83-126-221.ec2.internal)    19504 IPs (00.9% of total)
1a:cd:c1:53:24:bf(ip-10-83-36-17.ec2.internal)   507856 IPs (24.2% of total)
22:e1:23:b2:2f:8c(ip-10-83-59-237.ec2.internal)   190464 IPs (09.1% of total)
fa:ea:3e:94:41:e0(ip-10-83-85-183.ec2.internal)    16384 IPs (00.8% of total)
6a:5c:7c:af:e3:2e(ip-10-83-54-199.ec2.internal)   296016 IPs (14.1% of total) - unreachable!
6e:89:89:29:40:32(ip-10-83-98-46.ec2.internal)    16388 IPs (00.8% of total)
ee:c9:d9:9b:bd:ec(ip-10-83-78-215.ec2.internal)     4608 IPs (00.2% of total)
4a:d5:9f:aa:e3:99(ip-10-83-124-188.ec2.internal)     8192 IPs (00.4% of total)
9a:55:19:b7:97:7c(ip-10-83-76-130.ec2.internal)   313344 IPs (14.9% of total)
3a:5f:b6:6e:dd:bb(ip-10-83-80-134.ec2.internal)   367612 IPs (17.5% of total)
0e:8c:f8:33:67:7f(ip-10-83-87-92.ec2.internal)    16384 IPs (00.8% of total)
9a:5d:d4:e9:78:29(ip-10-83-109-188.ec2.internal)     1024 IPs (00.0% of total)
7e:44:ff:00:f5:f9(ip-10-83-60-42.ec2.internal)     4096 IPs (00.2% of total)

Weave connections

$ ./weave --local status connections
-> 10.83.41.74:6783      established fastdp 16:d8:40:86:d6:bf(ip-10-83-41-74.ec2.internal) mtu=1376
-> 10.83.126.221:6783    established fastdp 3a:47:b8:a9:34:ef(ip-10-83-126-221.ec2.internal) mtu=1376
-> 10.83.76.130:6783     established fastdp 9a:55:19:b7:97:7c(ip-10-83-76-130.ec2.internal) mtu=1376
-> 10.83.109.188:6783    established fastdp 9a:5d:d4:e9:78:29(ip-10-83-109-188.ec2.internal) mtu=1376
-> 10.83.60.42:6783      established fastdp 7e:44:ff:00:f5:f9(ip-10-83-60-42.ec2.internal) mtu=1376
-> 10.83.98.46:6783      established fastdp 6e:89:89:29:40:32(ip-10-83-98-46.ec2.internal) mtu=1376
-> 10.83.85.183:6783     established fastdp fa:ea:3e:94:41:e0(ip-10-83-85-183.ec2.internal) mtu=1376
-> 10.83.124.188:6783    established fastdp 4a:d5:9f:aa:e3:99(ip-10-83-124-188.ec2.internal) mtu=1376
-> 10.83.36.17:6783      established fastdp 1a:cd:c1:53:24:bf(ip-10-83-36-17.ec2.internal) mtu=1376
-> 10.83.78.215:6783     established fastdp ee:c9:d9:9b:bd:ec(ip-10-83-78-215.ec2.internal) mtu=1376
-> 10.83.80.134:6783     established fastdp 3a:5f:b6:6e:dd:bb(ip-10-83-80-134.ec2.internal) mtu=1376
-> 10.83.59.237:6783     established fastdp 22:e1:23:b2:2f:8c(ip-10-83-59-237.ec2.internal) mtu=1376
-> 10.83.87.92:6783      established fastdp 0e:8c:f8:33:67:7f(ip-10-83-87-92.ec2.internal) mtu=1376
-> 10.83.42.111:6783     failed      cannot connect to ourself, retry: never
-> 10.83.54.199:6783     failed      read tcp4 10.83.42.111:48865->10.83.54.199:6783: read: connection reset by peer, retry: 2018-06-05 11:15:00.712417081 +0000 UTC m=+836.618070318

Weave logs

$ kubectl logs -f -n=kube-system weave-net-tk4np -c=weave
DEBU: 2018/06/05 11:20:20.171724 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:20.373726 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:20.373752 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:20.373759 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:20.571557 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:20.571584 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:20.571591 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
INFO: 2018/06/05 11:20:20.701216 Removed unreachable peer 6a:5c:7c:af:e3:2e(ip-10-83-54-199.ec2.internal)
DEBU: 2018/06/05 11:20:20.771492 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:20.771516 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:20.771525 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:20.971641 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:20.971671 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:20.971684 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:21.171472 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:21.171499 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:21.171506 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:21.371635 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:21.371660 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:21.371668 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:21.571635 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:21.571663 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:21.571673 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:21.771057 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:21.771085 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:21.771092 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:21.971717 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:21.971743 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:21.971751 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:22.171426 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:22.171453 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:22.171463 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:22.371484 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:22.371510 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:22.371520 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:22.571555 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:22.571581 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:22.571589 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
INFO: 2018/06/05 11:20:22.743656 Discovered remote MAC 5a:d8:6a:42:96:2e at 9a:5d:d4:e9:78:29(ip-10-83-109-188.ec2.internal)
DEBU: 2018/06/05 11:20:22.771318 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:22.771341 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:22.771352 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:22.971001 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:22.971027 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:22.971035 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:23.171592 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:23.171617 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:23.171625 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:23.371614 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:23.371644 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:23.371652 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:23.574286 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:23.574312 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:23.574321 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:23.771439 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:23.771463 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:23.771471 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:23.971544 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:23.971571 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:23.971579 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:24.171478 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:24.171503 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:24.171512 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:24.371619 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:24.371645 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:24.371653 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:24.571288 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:24.571313 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:24.571321 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:24.771667 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:24.771712 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:24.771726 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:24.971541 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:24.971567 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:24.971575 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:25.171645 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:25.171675 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:25.171690 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:25.371589 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:25.371618 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:25.371629 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:25.571137 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:25.571170 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:25.571181 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:25.771402 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:25.771430 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:25.771438 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:25.971431 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:25.971459 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:25.971468 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:26.171483 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:26.171511 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:26.171521 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:26.371503 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:26.371531 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:26.371542 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:26.571542 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:26.571571 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:26.571581 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:26.771073 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:26.771102 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:26.771113 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:26.971435 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:26.971462 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:26.971469 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
INFO: 2018/06/05 11:20:27.070323 Removed unreachable peer 6a:5c:7c:af:e3:2e(ip-10-83-54-199.ec2.internal)
DEBU: 2018/06/05 11:20:27.171473 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:27.171498 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:27.171506 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:27.371505 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:27.371530 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:27.371538 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:27.574060 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:27.574088 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:27.574097 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:27.771457 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:27.771483 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:27.771492 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:27.972061 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:27.972087 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:27.972095 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:28.171557 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:28.171583 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:28.171592 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:28.371123 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:28.371149 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:28.371157 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:28.571669 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:28.571695 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:28.571703 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
INFO: 2018/06/05 11:20:28.677615 Discovered remote MAC ea:2d:28:ff:48:cf at ee:c9:d9:9b:bd:ec(ip-10-83-78-215.ec2.internal)
DEBU: 2018/06/05 11:20:28.771496 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:28.771524 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:28.771533 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:28.971517 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:28.971545 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:28.971552 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:29.171549 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:29.171575 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:29.171584 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:29.371325 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:29.371351 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:29.371413 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:29.571705 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:29.571731 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:29.571739 [kube-peers] Existing annotation 46:f4:4b:41:dd:11

[brb]

@zacblazic Thanks for the issue.

Regarding the first issue, do you have by any chance full Weave Net logs from ip-10-83-42-111.ec2.internal and ip-10-83-54-199.ec2.internal before the restart?

The second issue is going to be fixed in #3317.

[zacblazic]

@brb I'll try get them for you, no guarantees though.

We're seeing this happen more and more often now, seemingly random but cannot confirm the catalyst at this time.

Regarding the second issue, is there a way to manually remove the annotation or clear this state? It's clearly something that persists across the weave cluster as fresh nodes are having the same problem.

[itskingori]

@brb I work with Zac, and our cluster is getting unstable at least once a day. We fix it by finding out the bad weave pod and clearing it's state. This only started happening after upgrading to 2.3.0. We never had this issue with 2.2.0.

This is the result of connection status on all weave pods 👇 ... you can see it's pretty bad (although better than what we've seen recently).

$ ./weave.sh sandbox "status connections"
Running 'weave --local status connections' on weave-net-2md4g
----------------------------------------------
Defaulting container name to weave.
Use 'kubectl describe pod/weave-net-2md4g -n kube-system' to see all of the containers in this pod.
<- 10.83.109.188:64322   established fastdp 9a:5d:d4:e9:78:29(ip-10-83-109-188.ec2.internal) mtu=1376
<- 10.83.59.237:25989    established fastdp 22:e1:23:b2:2f:8c(ip-10-83-59-237.ec2.internal) mtu=1376
-> 10.83.42.111:6783     established fastdp 12:84:ba:e1:47:79(ip-10-83-42-111.ec2.internal) mtu=1376
<- 10.83.108.123:44036   established fastdp fa:17:fc:80:d1:6c(ip-10-83-108-123.ec2.internal) mtu=1376
-> 10.83.41.74:6783      established fastdp 16:d8:40:86:d6:bf(ip-10-83-41-74.ec2.internal) mtu=1376
<- 10.83.80.99:44626     established fastdp 2a:d6:a8:e7:2e:de(ip-10-83-80-99.ec2.internal) mtu=1376
<- 10.83.87.92:14742     established fastdp 0e:8c:f8:33:67:7f(ip-10-83-87-92.ec2.internal) mtu=1376
-> 10.83.78.215:6783     established fastdp ee:c9:d9:9b:bd:ec(ip-10-83-78-215.ec2.internal) mtu=1376
<- 10.83.125.214:18042   established fastdp 16:fb:51:d2:5c:25(ip-10-83-125-214.ec2.internal) mtu=1376
-> 10.83.60.42:6783      established fastdp 7e:44:ff:00:f5:f9(ip-10-83-60-42.ec2.internal) mtu=1376
<- 10.83.76.130:18782    established fastdp 9a:55:19:b7:97:7c(ip-10-83-76-130.ec2.internal) mtu=1376
<- 10.83.61.125:27523    established fastdp 46:1b:b4:25:cf:87(ip-10-83-61-125.ec2.internal) mtu=1376
-> 10.83.45.86:6783      established fastdp b2:8f:dd:99:f4:ae(ip-10-83-45-86.ec2.internal) mtu=1376
-> 10.83.114.127:6783    established fastdp d6:4a:51:67:64:13(ip-10-83-114-127.ec2.internal) mtu=1376
-> 10.83.88.180:6783     failed      cannot connect to ourself, retry: never
-> 10.83.98.46:6783      failed      Inconsistent entries for 100.97.0.36: owned by 3a:47:b8:a9:34:ef but incoming message says 2a:20:75:a8:38:e9, retry: 2018-06-14 11:31:33.965456343 +0000 UTC m=+81967.687532890
-> 10.83.126.221:6783    retrying    dial tcp4 :0->10.83.126.221:6783: connect: connection timed out
-> 10.83.58.17:6783      failed      Received update for IP range I own at 100.126.32.0 v31: incoming message says owner b2:30:8c:a3:99:1d v32, retry: 2018-06-14 11:31:05.706477947 +0000 UTC m=+81939.428554426
Running 'weave --local status connections' on weave-net-5kvfv
----------------------------------------------
Defaulting container name to weave.
Use 'kubectl describe pod/weave-net-5kvfv -n kube-system' to see all of the containers in this pod.
<- 10.83.125.214:48907   established fastdp 16:fb:51:d2:5c:25(ip-10-83-125-214.ec2.internal) mtu=1376
-> 10.83.80.99:6783      established fastdp 2a:d6:a8:e7:2e:de(ip-10-83-80-99.ec2.internal) mtu=1376
-> 10.83.45.86:6783      established fastdp b2:8f:dd:99:f4:ae(ip-10-83-45-86.ec2.internal) mtu=1376
-> 10.83.88.180:6783     established fastdp ae:d0:02:a5:ff:b4(ip-10-83-88-180.ec2.internal) mtu=1376
<- 10.83.60.42:37129     established fastdp 7e:44:ff:00:f5:f9(ip-10-83-60-42.ec2.internal) mtu=1376
<- 10.83.78.215:51406    established fastdp ee:c9:d9:9b:bd:ec(ip-10-83-78-215.ec2.internal) mtu=1376
<- 10.83.76.130:52366    established fastdp 9a:55:19:b7:97:7c(ip-10-83-76-130.ec2.internal) mtu=1376
<- 10.83.109.188:11699   established fastdp 9a:5d:d4:e9:78:29(ip-10-83-109-188.ec2.internal) mtu=1376
<- 10.83.59.237:62120    established fastdp 22:e1:23:b2:2f:8c(ip-10-83-59-237.ec2.internal) mtu=1376
-> 10.83.114.127:6783    established fastdp d6:4a:51:67:64:13(ip-10-83-114-127.ec2.internal) mtu=1376
<- 10.83.41.74:32446     established fastdp 16:d8:40:86:d6:bf(ip-10-83-41-74.ec2.internal) mtu=1376
-> 10.83.42.111:6783     established fastdp 12:84:ba:e1:47:79(ip-10-83-42-111.ec2.internal) mtu=1376
-> 10.83.108.123:6783    established fastdp fa:17:fc:80:d1:6c(ip-10-83-108-123.ec2.internal) mtu=1376
-> 10.83.61.125:6783     established fastdp 46:1b:b4:25:cf:87(ip-10-83-61-125.ec2.internal) mtu=1376
-> 10.83.58.17:6783      failed      Inconsistent entries for 100.96.36.0: owned by fa:17:fc:80:d1:6c but incoming message says 6e:89:89:29:40:32, retry: 2018-06-14 11:27:33.772854715 +0000 UTC m=+83875.969683927
-> 10.83.126.221:6783    failed      dial tcp4 :0->10.83.126.221:6783: connect: connection timed out, retry: 2018-06-14 11:29:24.549128843 +0000 UTC m=+83986.745958356
-> 10.83.87.92:6783      failed      cannot connect to ourself, retry: never
-> 10.83.98.46:6783      failed      Inconsistent entries for 100.105.64.0: owned by fa:17:fc:80:d1:6c but incoming message says 6e:89:89:29:40:32, retry: 2018-06-14 11:25:42.160799739 +0000 UTC m=+83764.357628911
Running 'weave --local status connections' on weave-net-5l8f9
----------------------------------------------
Defaulting container name to weave.
Use 'kubectl describe pod/weave-net-5l8f9 -n kube-system' to see all of the containers in this pod.
-> 10.83.87.92:6783      established fastdp 0e:8c:f8:33:67:7f(ip-10-83-87-92.ec2.internal) mtu=1376
<- 10.83.41.74:26540     established fastdp 16:d8:40:86:d6:bf(ip-10-83-41-74.ec2.internal) mtu=1376
<- 10.83.88.180:44552    established fastdp ae:d0:02:a5:ff:b4(ip-10-83-88-180.ec2.internal) mtu=1376
-> 10.83.80.99:6783      established fastdp 2a:d6:a8:e7:2e:de(ip-10-83-80-99.ec2.internal) mtu=1376
-> 10.83.45.86:6783      established fastdp b2:8f:dd:99:f4:ae(ip-10-83-45-86.ec2.internal) mtu=1376
-> 10.83.61.125:6783     established fastdp 46:1b:b4:25:cf:87(ip-10-83-61-125.ec2.internal) mtu=1376
<- 10.83.109.188:11537   established fastdp 9a:5d:d4:e9:78:29(ip-10-83-109-188.ec2.internal) mtu=1376
<- 10.83.125.214:19043   established fastdp 16:fb:51:d2:5c:25(ip-10-83-125-214.ec2.internal) mtu=1376
-> 10.83.114.127:6783    established fastdp d6:4a:51:67:64:13(ip-10-83-114-127.ec2.internal) mtu=1376
-> 10.83.78.215:6783     established fastdp ee:c9:d9:9b:bd:ec(ip-10-83-78-215.ec2.internal) mtu=1376
-> 10.83.42.111:6783     established fastdp 12:84:ba:e1:47:79(ip-10-83-42-111.ec2.internal) mtu=1376
<- 10.83.59.237:17338    established fastdp 22:e1:23:b2:2f:8c(ip-10-83-59-237.ec2.internal) mtu=1376
<- 10.83.76.130:31475    established fastdp 9a:55:19:b7:97:7c(ip-10-83-76-130.ec2.internal) mtu=1376
<- 10.83.108.123:33698   established fastdp fa:17:fc:80:d1:6c(ip-10-83-108-123.ec2.internal) mtu=1376
-> 10.83.98.46:6783      failed      Inconsistent entries for 100.105.64.0: owned by fa:17:fc:80:d1:6c but incoming message says 6e:89:89:29:40:32, retry: 2018-06-14 11:28:21.515968018 +0000 UTC m=+83917.194931761
-> 10.83.126.221:6783    failed      dial tcp4 :0->10.83.126.221:6783: connect: connection timed out, retry: 2018-06-14 11:31:59.350542407 +0000 UTC m=+84135.029506466
-> 10.83.58.17:6783      failed      Inconsistent entries for 100.96.36.0: owned by fa:17:fc:80:d1:6c but incoming message says 6e:89:89:29:40:32, retry: 2018-06-14 11:25:51.38209743 +0000 UTC m=+83767.061061136
-> 10.83.60.42:6783      failed      cannot connect to ourself, retry: never
Running 'weave --local status connections' on weave-net-62x5t
----------------------------------------------
Defaulting container name to weave.
Use 'kubectl describe pod/weave-net-62x5t -n kube-system' to see all of the containers in this pod.
-> 10.83.42.111:6783     established fastdp 12:84:ba:e1:47:79(ip-10-83-42-111.ec2.internal) mtu=1376
-> 10.83.59.237:6783     established fastdp 22:e1:23:b2:2f:8c(ip-10-83-59-237.ec2.internal) mtu=1376
-> 10.83.109.188:6783    established fastdp 9a:5d:d4:e9:78:29(ip-10-83-109-188.ec2.internal) mtu=1376
<- 10.83.76.130:52288    established fastdp 9a:55:19:b7:97:7c(ip-10-83-76-130.ec2.internal) mtu=1376
<- 10.83.78.215:29365    established fastdp ee:c9:d9:9b:bd:ec(ip-10-83-78-215.ec2.internal) mtu=1376
-> 10.83.60.42:6783      established fastdp 7e:44:ff:00:f5:f9(ip-10-83-60-42.ec2.internal) mtu=1376
-> 10.83.61.125:6783     established fastdp 46:1b:b4:25:cf:87(ip-10-83-61-125.ec2.internal) mtu=1376
-> 10.83.114.127:6783    established fastdp d6:4a:51:67:64:13(ip-10-83-114-127.ec2.internal) mtu=1376
-> 10.83.125.214:6783    established fastdp 16:fb:51:d2:5c:25(ip-10-83-125-214.ec2.internal) mtu=1376
-> 10.83.45.86:6783      established fastdp b2:8f:dd:99:f4:ae(ip-10-83-45-86.ec2.internal) mtu=1376
-> 10.83.108.123:6783    established fastdp fa:17:fc:80:d1:6c(ip-10-83-108-123.ec2.internal) mtu=1376
-> 10.83.87.92:6783      established fastdp 0e:8c:f8:33:67:7f(ip-10-83-87-92.ec2.internal) mtu=1376
-> 10.83.80.99:6783      established fastdp 2a:d6:a8:e7:2e:de(ip-10-83-80-99.ec2.internal) mtu=1376
<- 10.83.88.180:54210    established fastdp ae:d0:02:a5:ff:b4(ip-10-83-88-180.ec2.internal) mtu=1376
-> 10.83.41.74:6783      failed      cannot connect to ourself, retry: never
-> 10.83.58.17:6783      failed      Inconsistent entries for 100.96.36.0: owned by fa:17:fc:80:d1:6c but incoming message says 6e:89:89:29:40:32, retry: 2018-06-14 11:29:29.912737519 +0000 UTC m=+78654.224234518
-> 10.83.108.81:6783     failed      dial tcp4 :0->10.83.108.81:6783: connect: connection timed out, retry: 2018-06-14 11:26:12.325025858 +0000 UTC m=+78456.636522850
-> 10.83.98.46:6783      failed      Inconsistent entries for 100.105.64.0: owned by fa:17:fc:80:d1:6c but incoming message says 6e:89:89:29:40:32, retry: 2018-06-14 11:26:29.340993969 +0000 UTC m=+78473.652490924
-> 10.83.89.115:6783     failed      dial tcp4 :0->10.83.89.115:6783: connect: connection timed out, retry: 2018-06-14 11:28:42.789700924 +0000 UTC m=+78607.101198116
Running 'weave --local status connections' on weave-net-88667
----------------------------------------------
Defaulting container name to weave.
Use 'kubectl describe pod/weave-net-88667 -n kube-system' to see all of the containers in this pod.
-> 10.83.45.86:6783      established fastdp b2:8f:dd:99:f4:ae(ip-10-83-45-86.ec2.internal) mtu=1376
-> 10.83.114.127:6783    established fastdp d6:4a:51:67:64:13(ip-10-83-114-127.ec2.internal) mtu=1376
-> 10.83.108.123:6783    established fastdp fa:17:fc:80:d1:6c(ip-10-83-108-123.ec2.internal) mtu=1376
<- 10.83.60.42:17711     established fastdp 7e:44:ff:00:f5:f9(ip-10-83-60-42.ec2.internal) mtu=1376
-> 10.83.61.125:6783     established fastdp 46:1b:b4:25:cf:87(ip-10-83-61-125.ec2.internal) mtu=1376
<- 10.83.78.215:19858    established fastdp ee:c9:d9:9b:bd:ec(ip-10-83-78-215.ec2.internal) mtu=1376
<- 10.83.109.188:65431   established fastdp 9a:5d:d4:e9:78:29(ip-10-83-109-188.ec2.internal) mtu=1376
<- 10.83.41.74:17723     established fastdp 16:d8:40:86:d6:bf(ip-10-83-41-74.ec2.internal) mtu=1376
<- 10.83.125.214:39423   established fastdp 16:fb:51:d2:5c:25(ip-10-83-125-214.ec2.internal) mtu=1376
-> 10.83.76.130:6783     established fastdp 9a:55:19:b7:97:7c(ip-10-83-76-130.ec2.internal) mtu=1376
-> 10.83.88.180:6783     established fastdp ae:d0:02:a5:ff:b4(ip-10-83-88-180.ec2.internal) mtu=1376
<- 10.83.87.92:34053     established fastdp 0e:8c:f8:33:67:7f(ip-10-83-87-92.ec2.internal) mtu=1376
<- 10.83.80.99:28473     established fastdp 2a:d6:a8:e7:2e:de(ip-10-83-80-99.ec2.internal) mtu=1376
<- 10.83.59.237:35314    established fastdp 22:e1:23:b2:2f:8c(ip-10-83-59-237.ec2.internal) mtu=1376
-> 10.83.42.111:6783     failed      cannot connect to ourself, retry: never
-> 10.83.98.46:6783      failed      Inconsistent entries for 100.105.64.0: owned by fa:17:fc:80:d1:6c but incoming message says 6e:89:89:29:40:32, retry: 2018-06-14 11:29:55.896806219 +0000 UTC m=+84130.047212383
-> 10.83.58.17:6783      failed      Inconsistent entries for 100.96.36.0: owned by fa:17:fc:80:d1:6c but incoming message says 6e:89:89:29:40:32, retry: 2018-06-14 11:25:49.94123336 +0000 UTC m=+83884.091639521
-> 10.83.126.221:6783    failed      dial tcp4 :0->10.83.126.221:6783: connect: connection timed out, retry: 2018-06-14 11:29:06.362327232 +0000 UTC m=+84080.512733531
Running 'weave --local status connections' on weave-net-8wdpt
----------------------------------------------
Defaulting container name to weave.
Use 'kubectl describe pod/weave-net-8wdpt -n kube-system' to see all of the containers in this pod.
-> 10.83.108.123:6783    established fastdp fa:17:fc:80:d1:6c(ip-10-83-108-123.ec2.internal) mtu=1376
-> 10.83.42.111:6783     established fastdp 12:84:ba:e1:47:79(ip-10-83-42-111.ec2.internal) mtu=1376
<- 10.83.125.214:28764   established fastdp 16:fb:51:d2:5c:25(ip-10-83-125-214.ec2.internal) mtu=1376
-> 10.83.114.127:6783    established fastdp d6:4a:51:67:64:13(ip-10-83-114-127.ec2.internal) mtu=1376
<- 10.83.76.130:15202    established fastdp 9a:55:19:b7:97:7c(ip-10-83-76-130.ec2.internal) mtu=1376
-> 10.83.109.188:6783    established fastdp 9a:5d:d4:e9:78:29(ip-10-83-109-188.ec2.internal) mtu=1376
<- 10.83.88.180:54577    established fastdp ae:d0:02:a5:ff:b4(ip-10-83-88-180.ec2.internal) mtu=1376
-> 10.83.87.92:6783      established fastdp 0e:8c:f8:33:67:7f(ip-10-83-87-92.ec2.internal) mtu=1376
<- 10.83.61.125:55102    established fastdp 46:1b:b4:25:cf:87(ip-10-83-61-125.ec2.internal) mtu=1376
<- 10.83.60.42:48370     established fastdp 7e:44:ff:00:f5:f9(ip-10-83-60-42.ec2.internal) mtu=1376
-> 10.83.41.74:6783      established fastdp 16:d8:40:86:d6:bf(ip-10-83-41-74.ec2.internal) mtu=1376
<- 10.83.80.99:61954     established fastdp 2a:d6:a8:e7:2e:de(ip-10-83-80-99.ec2.internal) mtu=1376
-> 10.83.59.237:6783     established fastdp 22:e1:23:b2:2f:8c(ip-10-83-59-237.ec2.internal) mtu=1376
<- 10.83.45.86:44301     established fastdp b2:8f:dd:99:f4:ae(ip-10-83-45-86.ec2.internal) mtu=1376
-> 10.83.98.46:6783      failed      Received update for IP range I own at 100.113.2.0 v15: incoming message says owner 6e:89:89:29:40:32 v16, retry: 2018-06-14 11:32:31.770019755 +0000 UTC m=+84291.863032889
-> 10.83.126.221:6783    failed      dial tcp4 :0->10.83.126.221:6783: connect: connection timed out, retry: 2018-06-14 11:29:00.021782688 +0000 UTC m=+84080.114795935
-> 10.83.58.17:6783      failed      Received update for IP range I own at 100.113.2.0 v15: incoming message says owner 6e:89:89:29:40:32 v16, retry: 2018-06-14 11:25:37.719566741 +0000 UTC m=+83877.812579878
-> 10.83.78.215:6783     failed      cannot connect to ourself, retry: never
Running 'weave --local status connections' on weave-net-9mr87
----------------------------------------------
Defaulting container name to weave.
Use 'kubectl describe pod/weave-net-9mr87 -n kube-system' to see all of the containers in this pod.
<- 10.83.109.188:18863   established fastdp 9a:5d:d4:e9:78:29(ip-10-83-109-188.ec2.internal) mtu=1376
-> 10.83.114.127:6783    established fastdp d6:4a:51:67:64:13(ip-10-83-114-127.ec2.internal) mtu=1376
<- 10.83.41.74:10803     established fastdp 16:d8:40:86:d6:bf(ip-10-83-41-74.ec2.internal) mtu=1376
-> 10.83.45.86:6783      established fastdp b2:8f:dd:99:f4:ae(ip-10-83-45-86.ec2.internal) mtu=1376
<- 10.83.108.123:49548   established fastdp fa:17:fc:80:d1:6c(ip-10-83-108-123.ec2.internal) mtu=1376
-> 10.83.80.99:6783      established fastdp 2a:d6:a8:e7:2e:de(ip-10-83-80-99.ec2.internal) mtu=1376
<- 10.83.59.237:25988    established fastdp 22:e1:23:b2:2f:8c(ip-10-83-59-237.ec2.internal) mtu=1376
-> 10.83.125.214:6783    established fastdp 16:fb:51:d2:5c:25(ip-10-83-125-214.ec2.internal) mtu=1376
-> 10.83.60.42:6783      established fastdp 7e:44:ff:00:f5:f9(ip-10-83-60-42.ec2.internal) mtu=1376
-> 10.83.42.111:6783     established fastdp 12:84:ba:e1:47:79(ip-10-83-42-111.ec2.internal) mtu=1376
-> 10.83.78.215:6783     established fastdp ee:c9:d9:9b:bd:ec(ip-10-83-78-215.ec2.internal) mtu=1376
<- 10.83.61.125:47415    established fastdp 46:1b:b4:25:cf:87(ip-10-83-61-125.ec2.internal) mtu=1376
-> 10.83.87.92:6783      established fastdp 0e:8c:f8:33:67:7f(ip-10-83-87-92.ec2.internal) mtu=1376
-> 10.83.88.180:6783     established fastdp ae:d0:02:a5:ff:b4(ip-10-83-88-180.ec2.internal) mtu=1376
-> 10.83.58.17:6783      failed      Received update for IP range I own at 100.96.0.0 v9741: incoming message says owner 6e:89:89:29:40:32 v9743, retry: 2018-06-14 11:28:21.015301913 +0000 UTC m=+84025.243656987
-> 10.83.76.130:6783     failed      cannot connect to ourself, retry: never
-> 10.83.98.46:6783      failed      Received update for IP range I own at 100.96.0.0 v9741: incoming message says owner 46:1b:b4:25:cf:87 v9744, retry: 2018-06-14 11:27:52.330933768 +0000 UTC m=+83996.559288868
-> 10.83.126.221:6783    retrying    dial tcp4 :0->10.83.126.221:6783: connect: connection timed out
Running 'weave --local status connections' on weave-net-9vdqx
----------------------------------------------
Defaulting container name to weave.
Use 'kubectl describe pod/weave-net-9vdqx -n kube-system' to see all of the containers in this pod.
<- 10.83.59.237:31036    established fastdp 22:e1:23:b2:2f:8c(ip-10-83-59-237.ec2.internal) mtu=1376
<- 10.83.108.123:14174   established fastdp fa:17:fc:80:d1:6c(ip-10-83-108-123.ec2.internal) mtu=1376
<- 10.83.41.74:58313     established fastdp 16:d8:40:86:d6:bf(ip-10-83-41-74.ec2.internal) mtu=1376
-> 10.83.114.127:6783    established fastdp d6:4a:51:67:64:13(ip-10-83-114-127.ec2.internal) mtu=1376
<- 10.83.78.215:40928    established fastdp ee:c9:d9:9b:bd:ec(ip-10-83-78-215.ec2.internal) mtu=1376
-> 10.83.45.86:6783      established fastdp b2:8f:dd:99:f4:ae(ip-10-83-45-86.ec2.internal) mtu=1376
-> 10.83.87.92:6783      established fastdp 0e:8c:f8:33:67:7f(ip-10-83-87-92.ec2.internal) mtu=1376
<- 10.83.88.180:55270    established fastdp ae:d0:02:a5:ff:b4(ip-10-83-88-180.ec2.internal) mtu=1376
-> 10.83.60.42:6783      established fastdp 7e:44:ff:00:f5:f9(ip-10-83-60-42.ec2.internal) mtu=1376
-> 10.83.42.111:6783     established fastdp 12:84:ba:e1:47:79(ip-10-83-42-111.ec2.internal) mtu=1376
-> 10.83.76.130:6783     established fastdp 9a:55:19:b7:97:7c(ip-10-83-76-130.ec2.internal) mtu=1376
-> 10.83.80.99:6783      established fastdp 2a:d6:a8:e7:2e:de(ip-10-83-80-99.ec2.internal) mtu=1376
<- 10.83.61.125:30354    established fastdp 46:1b:b4:25:cf:87(ip-10-83-61-125.ec2.internal) mtu=1376
<- 10.83.125.214:30387   established fastdp 16:fb:51:d2:5c:25(ip-10-83-125-214.ec2.internal) mtu=1376
-> 10.83.58.17:6783      failed      Inconsistent entries for 100.96.36.0: owned by fa:17:fc:80:d1:6c but incoming message says 6e:89:89:29:40:32, retry: 2018-06-14 11:25:38.789173427 +0000 UTC m=+83754.709480414
-> 10.83.98.46:6783      failed      Inconsistent entries for 100.105.64.0: owned by fa:17:fc:80:d1:6c but incoming message says 6e:89:89:29:40:32, retry: 2018-06-14 11:25:48.485915806 +0000 UTC m=+83764.406222788
-> 10.83.126.221:6783    failed      dial tcp4 :0->10.83.126.221:6783: connect: no route to host, retry: 2018-06-14 11:30:05.84487768 +0000 UTC m=+84021.765184757
-> 10.83.109.188:6783    failed      cannot connect to ourself, retry: never
Running 'weave --local status connections' on weave-net-h5wcj
----------------------------------------------
Defaulting container name to weave.
Use 'kubectl describe pod/weave-net-h5wcj -n kube-system' to see all of the containers in this pod.
-> 10.83.88.180:6783     failed      read tcp4 10.83.98.46:53004->10.83.88.180:6783: read: connection reset by peer, retry: 2018-06-14 11:29:16.377667883 +0000 UTC m=+2054.107086596
-> 10.83.58.17:6783      failed      Inconsistent entries for 100.97.0.36: owned by 2a:20:75:a8:38:e9 but incoming message says 3a:47:b8:a9:34:ef, retry: 2018-06-14 11:32:45.741249992 +0000 UTC m=+2263.470668683
-> 10.83.76.130:6783     failed      read tcp4 10.83.98.46:38697->10.83.76.130:6783: read: connection reset by peer, retry: 2018-06-14 11:26:47.594652351 +0000 UTC m=+1905.324071028
-> 10.83.78.215:6783     failed      read tcp4 10.83.98.46:47933->10.83.78.215:6783: read: connection reset by peer, retry: 2018-06-14 11:26:03.4515573 +0000 UTC m=+1861.180976012
-> 10.83.42.111:6783     failed      Inconsistent entries for 100.105.64.0: owned by 6e:89:89:29:40:32 but incoming message says fa:17:fc:80:d1:6c, retry: 2018-06-14 11:29:50.44202945 +0000 UTC m=+2088.171448117
-> 10.83.59.237:6783     failed      Inconsistent entries for 100.105.64.0: owned by 6e:89:89:29:40:32 but incoming message says fa:17:fc:80:d1:6c, retry: 2018-06-14 11:32:47.025322481 +0000 UTC m=+2264.754741227
-> 10.83.60.42:6783      failed      Inconsistent entries for 100.105.64.0: owned by 6e:89:89:29:40:32 but incoming message says fa:17:fc:80:d1:6c, retry: 2018-06-14 11:29:21.857391059 +0000 UTC m=+2059.586809785
-> 10.83.87.92:6783      failed      Inconsistent entries for 100.105.64.0: owned by 6e:89:89:29:40:32 but incoming message says fa:17:fc:80:d1:6c, retry: 2018-06-14 11:27:04.77568889 +0000 UTC m=+1922.505107616
-> 10.83.108.123:6783    failed      Inconsistent entries for 100.105.64.0: owned by 6e:89:89:29:40:32 but incoming message says fa:17:fc:80:d1:6c, retry: 2018-06-14 11:27:16.422466316 +0000 UTC m=+1934.151885043
-> 10.83.98.46:6783      failed      cannot connect to ourself, retry: never
-> 10.83.125.214:6783    failed      write tcp4 10.83.98.46:21423->10.83.125.214:6783: write: connection reset by peer, retry: 2018-06-14 11:30:01.332060558 +0000 UTC m=+2099.061479252
-> 10.83.109.188:6783    failed      Inconsistent entries for 100.105.64.0: owned by 6e:89:89:29:40:32 but incoming message says fa:17:fc:80:d1:6c, retry: 2018-06-14 11:29:39.671812835 +0000 UTC m=+2077.401231586
-> 10.83.41.74:6783      failed      Inconsistent entries for 100.105.64.0: owned by 6e:89:89:29:40:32 but incoming message says fa:17:fc:80:d1:6c, retry: 2018-06-14 11:28:26.938954228 +0000 UTC m=+2004.668372957
-> 10.83.114.127:6783    failed      write tcp4 10.83.98.46:11946->10.83.114.127:6783: write: connection reset by peer, retry: 2018-06-14 11:27:33.910939201 +0000 UTC m=+1951.640357936
-> 10.83.45.86:6783      failed      Inconsistent entries for 100.105.64.0: owned by 6e:89:89:29:40:32 but incoming message says fa:17:fc:80:d1:6c, retry: 2018-06-14 11:30:01.670949968 +0000 UTC m=+2099.400368672
-> 10.83.61.125:6783     failed      Inconsistent entries for 100.97.0.36: owned by 2a:20:75:a8:38:e9 but incoming message says 3a:47:b8:a9:34:ef, retry: 2018-06-14 11:27:49.079178243 +0000 UTC m=+1966.808596976
-> 10.83.80.99:6783      failed      read tcp4 10.83.98.46:52649->10.83.80.99:6783: read: connection reset by peer, retry: 2018-06-14 11:28:16.808492709 +0000 UTC m=+1994.537911427
Running 'weave --local status connections' on weave-net-l2c8d
----------------------------------------------
Defaulting container name to weave.
Use 'kubectl describe pod/weave-net-l2c8d -n kube-system' to see all of the containers in this pod.
-> 10.83.59.237:6783     failed      Inconsistent entries for 100.96.36.0: owned by 6e:89:89:29:40:32 but incoming message says fa:17:fc:80:d1:6c, retry: 2018-06-14 11:30:27.339833685 +0000 UTC m=+1858.591352797
-> 10.83.42.111:6783     failed      Inconsistent entries for 100.96.36.0: owned by 6e:89:89:29:40:32 but incoming message says fa:17:fc:80:d1:6c, retry: 2018-06-14 11:26:25.688472069 +0000 UTC m=+1616.939991047
-> 10.83.61.125:6783     failed      Received update for IP range I own at 100.126.0.0 v10066: incoming message says owner ae:d0:02:a5:ff:b4 v10401, retry: 2018-06-14 11:33:22.334129662 +0000 UTC m=+2033.585648696
-> 10.83.41.74:6783      failed      Inconsistent entries for 100.96.36.0: owned by 6e:89:89:29:40:32 but incoming message says fa:17:fc:80:d1:6c, retry: 2018-06-14 11:26:14.442909091 +0000 UTC m=+1605.694428105
-> 10.83.76.130:6783     failed      read tcp4 10.83.58.17:40573->10.83.76.130:6783: read: connection reset by peer, retry: 2018-06-14 11:28:35.847360488 +0000 UTC m=+1747.098879463
-> 10.83.78.215:6783     failed      read tcp4 10.83.58.17:11003->10.83.78.215:6783: read: connection reset by peer, retry: 2018-06-14 11:27:56.413148287 +0000 UTC m=+1707.664667257
-> 10.83.98.46:6783      failed      Inconsistent entries for 100.97.0.36: owned by 3a:47:b8:a9:34:ef but incoming message says 2a:20:75:a8:38:e9, retry: 2018-06-14 11:28:28.70787515 +0000 UTC m=+1739.959394166
-> 10.83.80.99:6783      failed      Inconsistent entries for 100.96.36.0: owned by 6e:89:89:29:40:32 but incoming message says fa:17:fc:80:d1:6c, retry: 2018-06-14 11:28:58.681237192 +0000 UTC m=+1769.932756205
-> 10.83.114.127:6783    failed      Inconsistent entries for 100.96.36.0: owned by 6e:89:89:29:40:32 but incoming message says fa:17:fc:80:d1:6c, retry: 2018-06-14 11:30:57.939384677 +0000 UTC m=+1889.190903646
-> 10.83.58.17:6783      failed      cannot connect to ourself, retry: never
-> 10.83.60.42:6783      failed      read tcp4 10.83.58.17:35738->10.83.60.42:6783: read: connection reset by peer, retry: 2018-06-14 11:26:09.202236771 +0000 UTC m=+1600.453755744
-> 10.83.45.86:6783      failed      Inconsistent entries for 100.96.36.0: owned by 6e:89:89:29:40:32 but incoming message says fa:17:fc:80:d1:6c, retry: 2018-06-14 11:28:39.650034941 +0000 UTC m=+1750.901554017
-> 10.83.109.188:6783    failed      Inconsistent entries for 100.96.36.0: owned by 6e:89:89:29:40:32 but incoming message says fa:17:fc:80:d1:6c, retry: 2018-06-14 11:30:40.313034644 +0000 UTC m=+1871.564553678
-> 10.83.87.92:6783      failed      Inconsistent entries for 100.96.36.0: owned by 6e:89:89:29:40:32 but incoming message says fa:17:fc:80:d1:6c, retry: 2018-06-14 11:28:55.560646201 +0000 UTC m=+1766.812165217
-> 10.83.108.123:6783    failed      Inconsistent entries for 100.96.36.0: owned by 6e:89:89:29:40:32 but incoming message says fa:17:fc:80:d1:6c, retry: 2018-06-14 11:29:02.675724678 +0000 UTC m=+1773.927243704
-> 10.83.125.214:6783    failed      Inconsistent entries for 100.96.36.0: owned by 6e:89:89:29:40:32 but incoming message says fa:17:fc:80:d1:6c, retry: 2018-06-14 11:29:00.585894378 +0000 UTC m=+1771.837413407
-> 10.83.88.180:6783     failed      Received update for IP range I own at 100.126.0.0 v10066: incoming message says owner ae:d0:02:a5:ff:b4 v10401, retry: 2018-06-14 11:31:11.480243818 +0000 UTC m=+1902.731762985
Running 'weave --local status connections' on weave-net-mzm7x
----------------------------------------------
Defaulting container name to weave.
Use 'kubectl describe pod/weave-net-mzm7x -n kube-system' to see all of the containers in this pod.
<- 10.83.60.42:29859     established fastdp 7e:44:ff:00:f5:f9(ip-10-83-60-42.ec2.internal) mtu=1376
<- 10.83.109.188:51219   established fastdp 9a:5d:d4:e9:78:29(ip-10-83-109-188.ec2.internal) mtu=1376
<- 10.83.59.237:10622    established fastdp 22:e1:23:b2:2f:8c(ip-10-83-59-237.ec2.internal) mtu=1376
-> 10.83.61.125:6783     established fastdp 46:1b:b4:25:cf:87(ip-10-83-61-125.ec2.internal) mtu=1376
<- 10.83.41.74:20246     established fastdp 16:d8:40:86:d6:bf(ip-10-83-41-74.ec2.internal) mtu=1376
-> 10.83.114.127:6783    established fastdp d6:4a:51:67:64:13(ip-10-83-114-127.ec2.internal) mtu=1376
-> 10.83.108.123:6783    established fastdp fa:17:fc:80:d1:6c(ip-10-83-108-123.ec2.internal) mtu=1376
<- 10.83.87.92:36680     established fastdp 0e:8c:f8:33:67:7f(ip-10-83-87-92.ec2.internal) mtu=1376
<- 10.83.125.214:34279   established fastdp 16:fb:51:d2:5c:25(ip-10-83-125-214.ec2.internal) mtu=1376
-> 10.83.42.111:6783     established fastdp 12:84:ba:e1:47:79(ip-10-83-42-111.ec2.internal) mtu=1376
-> 10.83.78.215:6783     established fastdp ee:c9:d9:9b:bd:ec(ip-10-83-78-215.ec2.internal) mtu=1376
<- 10.83.76.130:31920    established fastdp 9a:55:19:b7:97:7c(ip-10-83-76-130.ec2.internal) mtu=1376
-> 10.83.45.86:6783      established fastdp b2:8f:dd:99:f4:ae(ip-10-83-45-86.ec2.internal) mtu=1376
<- 10.83.88.180:21758    established fastdp ae:d0:02:a5:ff:b4(ip-10-83-88-180.ec2.internal) mtu=1376
-> 10.83.126.221:6783    failed      dial tcp4 :0->10.83.126.221:6783: connect: connection timed out, retry: 2018-06-14 11:30:28.945977082 +0000 UTC m=+84156.011272833
-> 10.83.98.46:6783      failed      Inconsistent entries for 100.105.64.0: owned by fa:17:fc:80:d1:6c but incoming message says 6e:89:89:29:40:32, retry: 2018-06-14 11:29:19.584910211 +0000 UTC m=+84086.650205957
-> 10.83.58.17:6783      failed      Inconsistent entries for 100.96.36.0: owned by fa:17:fc:80:d1:6c but incoming message says 6e:89:89:29:40:32, retry: 2018-06-14 11:31:58.47465015 +0000 UTC m=+84245.539945759
-> 10.83.80.99:6783      failed      cannot connect to ourself, retry: never
Running 'weave --local status connections' on weave-net-nh6p4
----------------------------------------------
Defaulting container name to weave.
Use 'kubectl describe pod/weave-net-nh6p4 -n kube-system' to see all of the containers in this pod.
-> 10.83.88.180:6783     established fastdp ae:d0:02:a5:ff:b4(ip-10-83-88-180.ec2.internal) mtu=1376
<- 10.83.41.74:46144     established fastdp 16:d8:40:86:d6:bf(ip-10-83-41-74.ec2.internal) mtu=1376
-> 10.83.87.92:6783      established fastdp 0e:8c:f8:33:67:7f(ip-10-83-87-92.ec2.internal) mtu=1376
-> 10.83.109.188:6783    established fastdp 9a:5d:d4:e9:78:29(ip-10-83-109-188.ec2.internal) mtu=1376
<- 10.83.108.123:30342   established fastdp fa:17:fc:80:d1:6c(ip-10-83-108-123.ec2.internal) mtu=1376
<- 10.83.78.215:44092    established fastdp ee:c9:d9:9b:bd:ec(ip-10-83-78-215.ec2.internal) mtu=1376
-> 10.83.45.86:6783      established fastdp b2:8f:dd:99:f4:ae(ip-10-83-45-86.ec2.internal) mtu=1376
-> 10.83.61.125:6783     established fastdp 46:1b:b4:25:cf:87(ip-10-83-61-125.ec2.internal) mtu=1376
<- 10.83.125.214:25229   established fastdp 16:fb:51:d2:5c:25(ip-10-83-125-214.ec2.internal) mtu=1376
-> 10.83.42.111:6783     established fastdp 12:84:ba:e1:47:79(ip-10-83-42-111.ec2.internal) mtu=1376
-> 10.83.60.42:6783      established fastdp 7e:44:ff:00:f5:f9(ip-10-83-60-42.ec2.internal) mtu=1376
-> 10.83.114.127:6783    established fastdp d6:4a:51:67:64:13(ip-10-83-114-127.ec2.internal) mtu=1376
-> 10.83.76.130:6783     established fastdp 9a:55:19:b7:97:7c(ip-10-83-76-130.ec2.internal) mtu=1376
-> 10.83.80.99:6783      established fastdp 2a:d6:a8:e7:2e:de(ip-10-83-80-99.ec2.internal) mtu=1376
-> 10.83.98.46:6783      failed      Inconsistent entries for 100.105.64.0: owned by fa:17:fc:80:d1:6c but incoming message says 6e:89:89:29:40:32, retry: 2018-06-14 11:29:15.966023365 +0000 UTC m=+83969.896079258
-> 10.83.59.237:6783     failed      cannot connect to ourself, retry: never
-> 10.83.58.17:6783      failed      write tcp4 10.83.59.237:31865->10.83.58.17:6783: write: connection reset by peer, retry: 2018-06-14 11:29:29.378304351 +0000 UTC m=+83983.308360281
-> 10.83.126.221:6783    failed      dial tcp4 :0->10.83.126.221:6783: connect: connection timed out, retry: 2018-06-14 11:30:15.868698158 +0000 UTC m=+84029.798754184
Running 'weave --local status connections' on weave-net-qbl5b
----------------------------------------------
Defaulting container name to weave.
Use 'kubectl describe pod/weave-net-qbl5b -n kube-system' to see all of the containers in this pod.
<- 10.83.109.188:28757   established fastdp 9a:5d:d4:e9:78:29(ip-10-83-109-188.ec2.internal) mtu=1376
<- 10.83.59.237:44165    established fastdp 22:e1:23:b2:2f:8c(ip-10-83-59-237.ec2.internal) mtu=1376
<- 10.83.76.130:61619    established fastdp 9a:55:19:b7:97:7c(ip-10-83-76-130.ec2.internal) mtu=1376
<- 10.83.108.123:39192   established fastdp fa:17:fc:80:d1:6c(ip-10-83-108-123.ec2.internal) mtu=1376
-> 10.83.88.180:6783     established fastdp ae:d0:02:a5:ff:b4(ip-10-83-88-180.ec2.internal) mtu=1376
-> 10.83.45.86:6783      established fastdp b2:8f:dd:99:f4:ae(ip-10-83-45-86.ec2.internal) mtu=1376
<- 10.83.42.111:19399    established fastdp 12:84:ba:e1:47:79(ip-10-83-42-111.ec2.internal) mtu=1376
<- 10.83.87.92:56853     established fastdp 0e:8c:f8:33:67:7f(ip-10-83-87-92.ec2.internal) mtu=1376
-> 10.83.78.215:6783     established fastdp ee:c9:d9:9b:bd:ec(ip-10-83-78-215.ec2.internal) mtu=1376
<- 10.83.125.214:23211   established fastdp 16:fb:51:d2:5c:25(ip-10-83-125-214.ec2.internal) mtu=1376
<- 10.83.41.74:22143     established fastdp 16:d8:40:86:d6:bf(ip-10-83-41-74.ec2.internal) mtu=1376
<- 10.83.80.99:46426     established fastdp 2a:d6:a8:e7:2e:de(ip-10-83-80-99.ec2.internal) mtu=1376
<- 10.83.60.42:51528     established fastdp 7e:44:ff:00:f5:f9(ip-10-83-60-42.ec2.internal) mtu=1376
<- 10.83.61.125:12704    established fastdp 46:1b:b4:25:cf:87(ip-10-83-61-125.ec2.internal) mtu=1376
-> 10.83.98.46:6783      failed      Inconsistent entries for 100.105.64.0: owned by fa:17:fc:80:d1:6c but incoming message says 6e:89:89:29:40:32, retry: 2018-06-14 11:27:22.117036827 +0000 UTC m=+84070.657088638
-> 10.83.114.127:6783    failed      cannot connect to ourself, retry: never
-> 10.83.126.221:6783    failed      dial tcp4 :0->10.83.126.221:6783: connect: no route to host, retry: 2018-06-14 11:27:19.348685417 +0000 UTC m=+84067.888737367
-> 10.83.58.17:6783      failed      Inconsistent entries for 100.96.36.0: owned by fa:17:fc:80:d1:6c but incoming message says 6e:89:89:29:40:32, retry: 2018-06-14 11:27:07.322506294 +0000 UTC m=+84055.862558090
Running 'weave --local status connections' on weave-net-qh7s5
----------------------------------------------
Defaulting container name to weave.
Use 'kubectl describe pod/weave-net-qh7s5 -n kube-system' to see all of the containers in this pod.
-> 10.83.114.127:6783    established fastdp d6:4a:51:67:64:13(ip-10-83-114-127.ec2.internal) mtu=1376
-> 10.83.59.237:6783     established fastdp 22:e1:23:b2:2f:8c(ip-10-83-59-237.ec2.internal) mtu=1376
-> 10.83.60.42:6783      established fastdp 7e:44:ff:00:f5:f9(ip-10-83-60-42.ec2.internal) mtu=1376
-> 10.83.80.99:6783      established fastdp 2a:d6:a8:e7:2e:de(ip-10-83-80-99.ec2.internal) mtu=1376
<- 10.83.108.123:54931   established fastdp fa:17:fc:80:d1:6c(ip-10-83-108-123.ec2.internal) mtu=1376
-> 10.83.88.180:6783     established fastdp ae:d0:02:a5:ff:b4(ip-10-83-88-180.ec2.internal) mtu=1376
<- 10.83.41.74:21263     established fastdp 16:d8:40:86:d6:bf(ip-10-83-41-74.ec2.internal) mtu=1376
-> 10.83.61.125:6783     established fastdp 46:1b:b4:25:cf:87(ip-10-83-61-125.ec2.internal) mtu=1376
<- 10.83.76.130:45151    established fastdp 9a:55:19:b7:97:7c(ip-10-83-76-130.ec2.internal) mtu=1376
-> 10.83.109.188:6783    established fastdp 9a:5d:d4:e9:78:29(ip-10-83-109-188.ec2.internal) mtu=1376
-> 10.83.45.86:6783      established fastdp b2:8f:dd:99:f4:ae(ip-10-83-45-86.ec2.internal) mtu=1376
-> 10.83.42.111:6783     established fastdp 12:84:ba:e1:47:79(ip-10-83-42-111.ec2.internal) mtu=1376
-> 10.83.78.215:6783     established fastdp ee:c9:d9:9b:bd:ec(ip-10-83-78-215.ec2.internal) mtu=1376
-> 10.83.87.92:6783      established fastdp 0e:8c:f8:33:67:7f(ip-10-83-87-92.ec2.internal) mtu=1376
-> 10.83.98.46:6783      failed      Inconsistent entries for 100.105.64.0: owned by fa:17:fc:80:d1:6c but incoming message says 6e:89:89:29:40:32, retry: 2018-06-14 11:27:37.770269076 +0000 UTC m=+83869.405886652
-> 10.83.125.214:6783    failed      cannot connect to ourself, retry: never
-> 10.83.126.221:6783    failed      dial tcp4 :0->10.83.126.221:6783: connect: no route to host, retry: 2018-06-14 11:27:04.881744793 +0000 UTC m=+83836.517362462
-> 10.83.58.17:6783      failed      Inconsistent entries for 100.96.36.0: owned by fa:17:fc:80:d1:6c but incoming message says 6e:89:89:29:40:32, retry: 2018-06-14 11:32:41.52100929 +0000 UTC m=+84173.156626951
Running 'weave --local status connections' on weave-net-w9w9s
----------------------------------------------
Defaulting container name to weave.
Use 'kubectl describe pod/weave-net-w9w9s -n kube-system' to see all of the containers in this pod.
<- 10.83.109.188:48502   established fastdp 9a:5d:d4:e9:78:29(ip-10-83-109-188.ec2.internal) mtu=1376
<- 10.83.87.92:45114     established fastdp 0e:8c:f8:33:67:7f(ip-10-83-87-92.ec2.internal) mtu=1376
<- 10.83.60.42:17872     established fastdp 7e:44:ff:00:f5:f9(ip-10-83-60-42.ec2.internal) mtu=1376
<- 10.83.59.237:27796    established fastdp 22:e1:23:b2:2f:8c(ip-10-83-59-237.ec2.internal) mtu=1376
<- 10.83.42.111:13949    established fastdp 12:84:ba:e1:47:79(ip-10-83-42-111.ec2.internal) mtu=1376
-> 10.83.78.215:6783     established fastdp ee:c9:d9:9b:bd:ec(ip-10-83-78-215.ec2.internal) mtu=1376
<- 10.83.88.180:24330    established fastdp ae:d0:02:a5:ff:b4(ip-10-83-88-180.ec2.internal) mtu=1376
<- 10.83.61.125:45596    established fastdp 46:1b:b4:25:cf:87(ip-10-83-61-125.ec2.internal) mtu=1376
<- 10.83.108.123:57311   established fastdp fa:17:fc:80:d1:6c(ip-10-83-108-123.ec2.internal) mtu=1376
<- 10.83.114.127:64346   established fastdp d6:4a:51:67:64:13(ip-10-83-114-127.ec2.internal) mtu=1376
<- 10.83.125.214:12791   established fastdp 16:fb:51:d2:5c:25(ip-10-83-125-214.ec2.internal) mtu=1376
<- 10.83.41.74:56660     established fastdp 16:d8:40:86:d6:bf(ip-10-83-41-74.ec2.internal) mtu=1376
<- 10.83.80.99:54719     established fastdp 2a:d6:a8:e7:2e:de(ip-10-83-80-99.ec2.internal) mtu=1376
<- 10.83.76.130:58535    established fastdp 9a:55:19:b7:97:7c(ip-10-83-76-130.ec2.internal) mtu=1376
-> 10.83.98.46:6783      failed      Inconsistent entries for 100.105.64.0: owned by fa:17:fc:80:d1:6c but incoming message says 6e:89:89:29:40:32, retry: 2018-06-14 11:26:41.285648311 +0000 UTC m=+84034.787572182
-> 10.83.126.221:6783    failed      dial tcp4 :0->10.83.126.221:6783: connect: connection timed out, retry: 2018-06-14 11:29:43.146954682 +0000 UTC m=+84216.648878636
-> 10.83.45.86:6783      failed      cannot connect to ourself, retry: never
-> 10.83.58.17:6783      failed      Inconsistent entries for 100.96.36.0: owned by fa:17:fc:80:d1:6c but incoming message says 6e:89:89:29:40:32, retry: 2018-06-14 11:30:13.058231491 +0000 UTC m=+84246.560155285
Running 'weave --local status connections' on weave-net-wvlnc
----------------------------------------------
Defaulting container name to weave.
Use 'kubectl describe pod/weave-net-wvlnc -n kube-system' to see all of the containers in this pod.
-> 10.83.45.86:6783      established fastdp b2:8f:dd:99:f4:ae(ip-10-83-45-86.ec2.internal) mtu=1376
-> 10.83.76.130:6783     established fastdp 9a:55:19:b7:97:7c(ip-10-83-76-130.ec2.internal) mtu=1376
-> 10.83.78.215:6783     established fastdp ee:c9:d9:9b:bd:ec(ip-10-83-78-215.ec2.internal) mtu=1376
<- 10.83.125.214:39681   established fastdp 16:fb:51:d2:5c:25(ip-10-83-125-214.ec2.internal) mtu=1376
<- 10.83.80.99:37240     established fastdp 2a:d6:a8:e7:2e:de(ip-10-83-80-99.ec2.internal) mtu=1376
-> 10.83.59.237:6783     established fastdp 22:e1:23:b2:2f:8c(ip-10-83-59-237.ec2.internal) mtu=1376
-> 10.83.88.180:6783     established fastdp ae:d0:02:a5:ff:b4(ip-10-83-88-180.ec2.internal) mtu=1376
-> 10.83.41.74:6783      established fastdp 16:d8:40:86:d6:bf(ip-10-83-41-74.ec2.internal) mtu=1376
<- 10.83.87.92:55814     established fastdp 0e:8c:f8:33:67:7f(ip-10-83-87-92.ec2.internal) mtu=1376
<- 10.83.114.127:12361   established fastdp d6:4a:51:67:64:13(ip-10-83-114-127.ec2.internal) mtu=1376
-> 10.83.109.188:6783    established fastdp 9a:5d:d4:e9:78:29(ip-10-83-109-188.ec2.internal) mtu=1376
-> 10.83.42.111:6783     established fastdp 12:84:ba:e1:47:79(ip-10-83-42-111.ec2.internal) mtu=1376
-> 10.83.60.42:6783      established fastdp 7e:44:ff:00:f5:f9(ip-10-83-60-42.ec2.internal) mtu=1376
<- 10.83.108.123:52011   established fastdp fa:17:fc:80:d1:6c(ip-10-83-108-123.ec2.internal) mtu=1376
-> 10.83.98.46:6783      failed      Inconsistent entries for 100.97.0.36: owned by 3a:47:b8:a9:34:ef but incoming message says 2a:20:75:a8:38:e9, retry: 2018-06-14 11:27:24.566478723 +0000 UTC m=+78938.833355396
-> 10.83.89.115:6783     failed      dial tcp4 :0->10.83.89.115:6783: connect: connection timed out, retry: 2018-06-14 11:31:28.361942045 +0000 UTC m=+79182.628818762
-> 10.83.58.17:6783      failed      read tcp4 10.83.61.125:36602->10.83.58.17:6783: read: connection reset by peer, retry: 2018-06-14 11:34:34.458989108 +0000 UTC m=+79368.725865781
-> 10.83.61.125:6783     failed      cannot connect to ourself, retry: never
-> 10.83.108.81:6783     failed      dial tcp4 :0->10.83.108.81:6783: connect: connection timed out, retry: 2018-06-14 11:28:21.152854771 +0000 UTC m=+78995.419731614
Running 'weave --local status connections' on weave-net-zhrmd
----------------------------------------------
Defaulting container name to weave.
Use 'kubectl describe pod/weave-net-zhrmd -n kube-system' to see all of the containers in this pod.
-> 10.83.60.42:6783      established fastdp 7e:44:ff:00:f5:f9(ip-10-83-60-42.ec2.internal) mtu=1376
-> 10.83.59.237:6783     established fastdp 22:e1:23:b2:2f:8c(ip-10-83-59-237.ec2.internal) mtu=1376
-> 10.83.114.127:6783    established fastdp d6:4a:51:67:64:13(ip-10-83-114-127.ec2.internal) mtu=1376
-> 10.83.88.180:6783     established fastdp ae:d0:02:a5:ff:b4(ip-10-83-88-180.ec2.internal) mtu=1376
<- 10.83.87.92:27093     established fastdp 0e:8c:f8:33:67:7f(ip-10-83-87-92.ec2.internal) mtu=1376
-> 10.83.45.86:6783      established fastdp b2:8f:dd:99:f4:ae(ip-10-83-45-86.ec2.internal) mtu=1376
<- 10.83.78.215:48104    established fastdp ee:c9:d9:9b:bd:ec(ip-10-83-78-215.ec2.internal) mtu=1376
-> 10.83.125.214:6783    established fastdp 16:fb:51:d2:5c:25(ip-10-83-125-214.ec2.internal) mtu=1376
-> 10.83.76.130:6783     established fastdp 9a:55:19:b7:97:7c(ip-10-83-76-130.ec2.internal) mtu=1376
-> 10.83.109.188:6783    established fastdp 9a:5d:d4:e9:78:29(ip-10-83-109-188.ec2.internal) mtu=1376
<- 10.83.80.99:25749     established fastdp 2a:d6:a8:e7:2e:de(ip-10-83-80-99.ec2.internal) mtu=1376
<- 10.83.41.74:20169     established fastdp 16:d8:40:86:d6:bf(ip-10-83-41-74.ec2.internal) mtu=1376
-> 10.83.61.125:6783     established fastdp 46:1b:b4:25:cf:87(ip-10-83-61-125.ec2.internal) mtu=1376
<- 10.83.42.111:22124    established fastdp 12:84:ba:e1:47:79(ip-10-83-42-111.ec2.internal) mtu=1376
-> 10.83.108.123:6783    failed      cannot connect to ourself, retry: never
-> 10.83.98.46:6783      failed      Received update for IP range I own at 100.96.36.0 v4: incoming message says owner 46:1b:b4:25:cf:87 v5, retry: 2018-06-14 11:30:58.746286293 +0000 UTC m=+81575.949094494
-> 10.83.58.17:6783      failed      Inconsistent entries for 100.96.36.0: owned by fa:17:fc:80:d1:6c but incoming message says 6e:89:89:29:40:32, retry: 2018-06-14 11:26:38.271243376 +0000 UTC m=+81315.474051605

[bboreham]

@itskingori thanks; we really would like logs (as in kubectl logs ... weave) to debug.

If you have cleared a pod's state and restarted it and then it goes bad within the day then there is a good chance the logs do have the vital information. If the pod has restarted please fetch the previous logs too with kubectl logs -p.

To be clear, in case of message -> 10.83.98.46:6783 failed Received update for IP ... message says owner 46:1b:b4:25:cf:87 v5, we would like the logs for the pod that gives this message and for the pod with ID 46:1b:b4:25:cf:87

[itskingori]

@bboreham Got it! Will keep you guys posted.

[itskingori]

@bboreham @brb While it hasn't happened again, I've done some investigation and I'd like to pass a theory past you two. Could it be as a result of the pods being abruptly killed .e.g. an OOM?

In the case of #3310 (comment), we cleared the state of weave-net-h5wcj and weave-net-l2c8d, then the cluster healed.

You can see that weave-net-h5wcj was killed at around 13:55, weave-net-l2c8d at around 13:59 ... 👇

We installed via kops, therefore our resource requests and limits are set to 👇

resources:
  limits:
    memory: 200Mi
  requests:
    cpu: 50m
    memory: 200Mi

Here are the logs of weave-net-h5wcj at around 13:55 👇

And here are the logs of weave-net-l2c8d at around 13:59 👇

You can see the logging profile of weave-net-h5wcj around the kill below 👇

And the logging profile of weave-net-l2c8d around the kill below 👇

So:

1. Those kills definitely affected weave.
2. Is there a possibility that the state was affected by the kill?
3. Should we increase memory assigned to weave?

[itskingori]

@bboreham @brb Another theory, could it be that the infinite loop fixed in #3317 is causing the weave pod to use more and more memory? See 30 days usage below:

My colleague @tsu-shiuan, thinks that increasing memory would be hiding the problem ... and that while the OOMs are likely the cause of the issue described in #3310 (comment), that there's an underlying problem causing the increase in memory.

Some points to note:

1. We upgraded weave to 2.3.0 about 18 days ago i.e. around May 28th.
2. Before that we have upgraded weave to 2.2.0 about 4 months ago.

The disruption right after the 28th is us rotating nodes, so that's expected ... but you can see after that things are not looking good.

[brb]

@zacblazic @itskingori Thanks for the info.

This only started happening after upgrading to 2.3.0. We never had this issue with 2.2.0.

I doubt that any change from 2.2.0 to 2.3.0 could have triggered the problem (maybe "Build Weave Net with Go 1.10.1 #3273", but let's keep it as a last resort). My guess is that the upgrade restarted Weave Net on each node which triggered the faulty reclaim DEBU: 2018/06/05 11:20:29.571731 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}, as reclaiming happens upon the Weave Net initialization.

Regarding the second issue, is there a way to manually remove the annotation or clear this state?

I don't have a cmd at hand, but try updating the annotation with kubectl annotate configmap weave-net -n=kube-system --overwrite kube-peers.weave.works/peers='$PEERS' where $PEERS is the existing annotation value (you can get it with kubectl get configmap weave-net ... -o yaml) minus the faulty peer. The annotation format is JSON. After the change, you need to restart Weave on nodes which are in the infinite loop.

Anyway, do you have any logs for ip-10-83-124-112.ec2.internal?

Could it be as a result of the pods being abruptly killed .e.g. an OOM?

Might be that boltdb which stores IPAM fails to write to disk as the process gets killed. So, after the restart Weave Net restores a stale data (@bboreham shouldn't the CRDT IPAM ring be able to merge in such situation anyway?).

Interested in your logs after you have reproduced the issue. Just please upload them as txt to make grepping possible. Ta.

[kuznero]

I can see exactly same kind of error Received update for IP range I own but simply when I try to add another node to a cluster. The strange thing is that it does not happen always.

[bboreham]

@kuznero please open your own issue and supply logs.

[bboreham]

(@bboreham shouldn't the CRDT IPAM ring be able to merge in such situation anyway?).

@brb it can merge consistent data of any age. The problem here is that the data is inconsistent with what someone else has.

Issue #2084 covers the case where the inconsistency doesn't actually matter.

[bboreham]

OK, I didn't read the whole question, sorry.

Could it be as a result of the pods being abruptly killed .e.g. an OOM?

Might be that boltdb which stores IPAM fails to write to disk as the process gets killed. So, after the restart Weave Net restores a stale data

Something like this sequence?

• a donates IP range r to b
• a dies and restarts, loads previous state
• both a and b think they own r

To avoid this, we persist to disk before sending the message from a to b, and that goes through to a fsync syscall.

I guess it's possible there is a bug in the above. Or it's something else.

[brb]

@zacblazic @itskingori @kuznero What file system (with options) is of the
/var/lib/weave host dir? mount can show this information.

@kuznero Can you easily reproduce the issue?