v2 plugin can get stuck with wrong peer list

[bboreham]

As reported on Slack

The node was rebooted, but the v2 plugin came up thinking it was a restart and did not re-fetch the list of peers.

Nov 08 08:41:27 azure-swarm-manager systemd[1]: Starting Docker Application Container Engine...
Nov 08 08:41:29 azure-swarm-manager dockerd[1348]: time="2017-11-08T08:41:29.897300800Z" level=info msg="libcontainerd: new containerd process, pid: 1579"
Nov 08 08:41:31 azure-swarm-manager dockerd[1348]: time="2017-11-08T08:41:31.007597200Z" level=warning msg="libcontainerd: client is out of sync, restore was called on a fully synced container (d2eb3b91fa7aec6f7e7043ba3584272fa102465011d
Nov 08 08:41:31 azure-swarm-manager dockerd[1348]: time="2017-11-08T08:41:31.008556600Z" level=warning msg="libcontainerd: failed to retrieve container d2eb3b91fa7aec6f7e7043ba3584272fa102465011d289ace0677260b971a6f2 state: rpc error: co
Nov 08 08:41:31 azure-swarm-manager dockerd[1348]: time="2017-11-08T08:41:31Z" level=info msg="mkdir: can't create directory '/host/var/lib/weave': File exists" plugin=d2eb3b91fa7aec6f7e7043ba3584272fa102465011d289ace0677260b971a6f2
Nov 08 08:41:31 azure-swarm-manager dockerd[1348]: time="2017-11-08T08:41:31Z" level=info msg="Starting launch.sh" plugin=d2eb3b91fa7aec6f7e7043ba3584272fa102465011d289ace0677260b971a6f2
Nov 08 08:41:32 azure-swarm-manager dockerd[1348]: time="2017-11-08T08:41:32Z" level=info msg="INFO: 2017/11/08 08:41:32.377431 Command line options: map[port:6783 datapath:datapath docker-api: nickname:azure-swarm-manager db-prefix:/hos
Nov 08 08:41:32 azure-swarm-manager dockerd[1348]: time="2017-11-08T08:41:32Z" level=info msg="INFO: 2017/11/08 08:41:32.380729 weave  2.0.5" plugin=d2eb3b91fa7aec6f7e7043ba3584272fa102465011d289ace0677260b971a6f2
Nov 08 08:41:32 azure-swarm-manager dockerd[1348]: time="2017-11-08T08:41:32Z" level=info msg="INFO: 2017/11/08 08:41:32.686952 Bridge type is bridged_fastdp" plugin=d2eb3b91fa7aec6f7e7043ba3584272fa102465011d289ace0677260b971a6f2
Nov 08 08:41:32 azure-swarm-manager dockerd[1348]: time="2017-11-08T08:41:32Z" level=info msg="INFO: 2017/11/08 08:41:32.687118 Communication between peers is unencrypted." plugin=d2eb3b91fa7aec6f7e7043ba3584272fa102465011d289ace0677260b
Nov 08 08:41:32 azure-swarm-manager dockerd[1348]: time="2017-11-08T08:41:32Z" level=info msg="INFO: 2017/11/08 08:41:32.707000 Our name is 56:82:4f:1d:8c:99(azure-swarm-manager)" plugin=d2eb3b91fa7aec6f7e7043ba3584272fa102465011d289ace0
Nov 08 08:41:32 azure-swarm-manager dockerd[1348]: time="2017-11-08T08:41:32Z" level=info msg="INFO: 2017/11/08 08:41:32.707260 Restart/resume detected - using persisted peer list: []" plugin=d2eb3b91fa7aec6f7e7043ba3584272fa102465011d28
Nov 08 08:41:32 azure-swarm-manager dockerd[1348]: time="2017-11-08T08:41:32Z" level=info msg="INFO: 2017/11/08 08:41:32.707351 Checking for pre-existing addresses on weave bridge" plugin=d2eb3b91fa7aec6f7e7043ba3584272fa102465011d289ace
Nov 08 08:41:32 azure-swarm-manager dockerd[1348]: time="2017-11-08T08:41:32Z" level=info msg="DEBU: 2017/11/08 08:41:32.711231 fastdp: broadcast{1 false} {56:82:4f:1d:8c:99 33:33:00:00:00:16}" plugin=d2eb3b91fa7aec6f7e7043ba3584272fa102
Nov 08 08:41:32 azure-swarm-manager dockerd[1348]: time="2017-11-08T08:41:32Z" level=info msg="DEBU: 2017/11/08 08:41:32.711331 Creating ODP flow FlowSpec{keys: [EthernetFlowKey{src: 56:82:4f:1d:8c:99, dst: 33:33:00:00:00:16} InPortFlowK
Nov 08 08:41:32 azure-swarm-manager dockerd[1348]: time="2017-11-08T08:41:32Z" level=info msg="INFO: 2017/11/08 08:41:32.712303 [allocator 56:82:4f:1d:8c:99] Initialising with persisted data" plugin=d2eb3b91fa7aec6f7e7043ba3584272fa10246
Nov 08 08:41:32 azure-swarm-manager dockerd[1348]: time="2017-11-08T08:41:32Z" level=info msg="INFO: 2017/11/08 08:41:32.712334 Sniffing traffic on datapath (via ODP)" plugin=d2eb3b91fa7aec6f7e7043ba3584272fa102465011d289ace0677260b971a6
Nov 08 08:41:32 azure-swarm-manager dockerd[1348]: time="2017-11-08T08:41:32Z" level=info msg="INFO: 2017/11/08 08:41:32.739449 Listening for HTTP control messages on 127.0.0.1:6784" plugin=d2eb3b91fa7aec6f7e7043ba3584272fa102465011d289a
Nov 08 08:41:32 azure-swarm-manager dockerd[1348]: time="2017-11-08T08:41:32Z" level=info msg="INFO: 2017/11/08 08:41:32.741363 Listening for metrics requests on 0.0.0.0:6782" plugin=d2eb3b91fa7aec6f7e7043ba3584272fa102465011d289ace06772
Nov 08 08:41:32 azure-swarm-manager dockerd[1348]: time="2017-11-08T08:41:32Z" level=info msg="INFO: 2017/11/08 08:41:32.751601 Waiting for Weave API Server..." plugin=d2eb3b91fa7aec6f7e7043ba3584272fa102465011d289ace0677260b971a6f2
Nov 08 08:41:32 azure-swarm-manager dockerd[1348]: time="2017-11-08T08:41:32Z" level=info msg="DEBU: 2017/11/08 08:41:32.765843 [http] GET /status" plugin=d2eb3b91fa7aec6f7e7043ba3584272fa102465011d289ace0677260b971a6f2
Nov 08 08:41:32 azure-swarm-manager dockerd[1348]: time="2017-11-08T08:41:32Z" level=info msg="INFO: 2017/11/08 08:41:32.766695 Finished waiting for Weave API Server" plugin=d2eb3b91fa7aec6f7e7043ba3584272fa102465011d289ace0677260b971a6f
...

[bboreham]

Once it's in this state it doesn't come out as discussed at #1692 and #2400

I think this log line relates:

Nov 08 08:41:39 azure-swarm-manager dockerd[1348]: time="2017-11-08T08:41:39.322265000Z" level=warning msg="Failed the node discovery in driver: NetworkDriver.DiscoverNew: 404 page not found

[brb]

Does it get stuck as the issue title suggests? I expect that eventually (depending on mesh backoff timer) some other peer will connect to it.

[bboreham]

Per the Slack conversation, this never happened.
If the peer didn't exist when the other nodes fired up, how would they add it?