arping on container start/attach limits scalability

[rade]

As of #457, we send an unsolicited ARP response whenever a container is attached to the weave network. We do this in order to work around a bug in the Linux kernel. More details in #441.

This does limit scalability, since a) the ARP response is broadcast, and b) weave peers create an entry in their MAC cache when seeing that response. Both of these are especially problematic when there is lots of container churn, and on restart of a cluster, when there are potentially lots of containers being attached to the weave network in a short space of time.

[rade]

The fix to the kernel bug is in 3.19. Which will take a while to get widely deployed. But perhaps we should make the unsolicited ARPing conditional on pre-3.19 kernels, so that on more recent kernels we can avoid the aforementioned downsides. The only problem is that we'd really need to know the kernel version on all the hosts other than ourselves. Hmm. I suspect most deployments are uniform though. And I guess we could add some switch/env-var to depart from whatever behaviour we make the default.

@dpw, what do you reckon?

[dpw]

Have we or our users observed measurable performance consequences of the unsolicited ARP? It sounds marginal to me, even if you had a big weave cluster with all hosts churning away starting containers, but that's just a guess.

[rade]

Oh, I'm quite sure it would take "a while" to run into this. It would likely manifest itself as packet loss.

It's a bit of a blemish. AFAIK it's the only thing in weave that would impose a global limit on the rate of container starts.

[dpw]

Even if we're looking at this because it might bite someone in the future, rather than because it is a problem for anyone now, it would still be good to see a projection based on some kind of measurements (e.g. might it become an issue at 100 hosts? 1000? 10000?).

Also, if you have that much container churn, aren't they likely to contain client applications, and therefore be ARPing in order to locate servers? Ok, unsolicited ARPs will make matters worse, but if we are worried about ARP limiting weave scalability, surely it would be good to have some estimate of where that limit lies taking all causes of ARPs into account.

[rade]

I am not opposed to running some measurements.

Good point re client containers ARPing for servers.