weave-kube reclaim is not able to rm a dead peer if a running peer has the same hostname

[brb]

As seen in #3384 (comment), it's possible for a new node to have an IP addr (and a hostname) of a dead node. In this situation, the weave-kube reclaimer won't be able to rmpeer the dead node, because it doesn't identify running nodes by their peer names, but their hostname (https://github.com/weaveworks/weave/blob/v2.4.0/prog/kube-utils/main.go#L114).

Amazon ASG users are extremely exposed to this problem.

[Raffo]

Thanks for this issue! Does it mean that the peer name is determined in a unique way (as explained in the doc) but the deletion is just wrong cause it only uses the hostname?

[bboreham]

To clarify (as noted in #3384) what is supposed to happen:

• Kubernetes has a list of Nodes.
• Weave Net maintains a list of Nodes that have been added as "peers" to the Weave Net cluster.

Because we don't get a chance to clean up when a Node is being shut down, we only notice afterwards
that a node has been deleted. Specifically we spot that that a node is in the second list and not in the first list.

This bug is that code which computes the difference of those two lists is doing it on hostname not on peer name.

To cause a problem the hostname re-use has to happen before any Weave Net pod starting up has had a chance to remove it.

[bboreham]

An idea to address this:
Store the Kubernetes UID of each Node in the peer list, then we can differentiate new Nodes that have the same name.

[Raffo]

Using the Kubernetes UUID sounds like a good idea as we know for sure that NodeName and Hostname are not unique.