Uneven distribution of IP address ownership

[bboreham]

This is right after startup in a 3-node cluster:

# weave status ipam
76:a1:dd:6c:12:51(brya-0)               524288 IPs (50.0% of total) (1 active)
0e:e4:4e:dd:92:ca(brya-2)               393216 IPs (37.5% of total) 
ba:23:5a:0c:f3:10(brya-1)               131072 IPs (12.5% of total) 
# weave report
...
    "IPAM": {
        "Paxos": null,
        "Range": "10.32.0.0/12",
        "RangeNumIPs": 1048576,
        "ActiveIPs": 1,
        "DefaultSubnet": "10.32.0.0/12",
        "Entries": [
            {
                "Token": "10.32.0.0",
                "Size": 393216,
                "Peer": "0e:e4:4e:dd:92:ca",
                "Nickname": "brya-2",
                "IsKnownPeer": true,
                "Version": 5
            },
            {
                "Token": "10.38.0.0",
                "Size": 131072,
                "Peer": "ba:23:5a:0c:f3:10",
                "Nickname": "brya-1",
                "IsKnownPeer": true,
                "Version": 3
            },
            {
                "Token": "10.40.0.0",
                "Size": 524288,
                "Peer": "76:a1:dd:6c:12:51",
                "Nickname": "brya-0",
                "IsKnownPeer": true,
                "Version": 1
            }
        ],
        "PendingClaims": null,
        "PendingAllocates": null
    }
}

I would expect 50/25/25; the 10.38.0.0 token is bang in the middle but the 10.40.0.0 one is a quarter of the way between the two points.

(This particular distribution does not have any negative consequences, but I'm curious how it comes about)

[bboreham]

As I commented on #3171:

We could have some modest background task where any peer can say at any time "I perceive that I have 1% of the address space and someone else has 90%; I will ask for some more" - that would help with "re-distribute" and also lessen the impact of a delay in reclaiming.

We also want to avoid gratuitously fragmenting the overall space. Although that may be ok as a consequence of existing heuristics.

[bboreham]

Here's another example: newly-created kops cluster:

# weave status ipam
1a:14:77:48:b4:0d(ip-172-20-58-132.eu-central-1.compute.internal)  1048576 IPs (50.0% of total) (1 active)
8a:45:a2:e8:9a:f9(ip-172-20-70-4.eu-central-1.compute.internal)   524288 IPs (25.0% of total) 
4a:b2:c6:50:52:f5(ip-172-20-35-59.eu-central-1.compute.internal)   393216 IPs (18.8% of total) 
62:67:b2:a1:15:50(ip-172-20-127-112.eu-central-1.compute.internal)   131072 IPs (06.2% of total)
# weave report
...
    "IPAM": {
        "Paxos": null,
        "Range": "100.96.0.0/11",
        "RangeNumIPs": 2097152,
        "ActiveIPs": 1,
        "DefaultSubnet": "100.96.0.0/11",
        "Entries": [
            {
                "Token": "100.96.0.0",
                "Size": 524288,
                "Peer": "1a:14:77:48:b4:0d",
                "Nickname": "ip-172-20-58-132.eu-central-1.compute.internal",
                "IsKnownPeer": true,
                "Version": 5
            },
            {
                "Token": "100.104.0.0",
                "Size": 393216,
                "Peer": "4a:b2:c6:50:52:f5",
                "Nickname": "ip-172-20-35-59.eu-central-1.compute.internal",
                "IsKnownPeer": true,
                "Version": 4
            },
            {
                "Token": "100.110.0.0",
                "Size": 131072,
                "Peer": "62:67:b2:a1:15:50",
                "Nickname": "ip-172-20-127-112.eu-central-1.compute.internal",
                "IsKnownPeer": true,
                "Version": 2
            },
            {
                "Token": "100.112.0.0",
                "Size": 524288,
                "Peer": "1a:14:77:48:b4:0d",
                "Nickname": "ip-172-20-58-132.eu-central-1.compute.internal",
                "IsKnownPeer": true,
                "Version": 0
            },
            {
                "Token": "100.120.0.0",
                "Size": 524288,
                "Peer": "8a:45:a2:e8:9a:f9",
                "Nickname": "ip-172-20-70-4.eu-central-1.compute.internal",
                "IsKnownPeer": true,
                "Version": 2
            }
        ],
        "PendingClaims": null,
        "PendingAllocates": null
    }