networking broken for containers with moved MAC

[rade]

when a MAC previously associated with a container on one host, is subsequently associated with container on a different host, the latter container has no network connectivity.

host1$ weave launch
host2$ weave launch $HOST1
host1$ A=$(weave run 10.0.0.1/24 -ti alpine /bin/sh)
host1$ B=$(weave run 10.0.0.2/24 -ti alpine /bin/sh)
host2$ C=$(weave run 10.0.0.3/24 --privileged -ti alpine /bin/sh)
host2$ docker exec $C ping -nq -W 1 -c 1 10.0.0.2
PING 10.0.0.2 (10.0.0.2): 56 data bytes
--- 10.0.0.2 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.947/0.947/0.947 ms
host1$ docker exec $A ip link show ethwe | sed -n -e 's|^ *link/ether \([0-9a-f:]*\).*|\1|p'
aa:c5:15:f8:a3:e8
host1$ docker rm -f $A
host2$ docker exec $C ip link set ethwe address aa:c5:15:f8:a3:e8
host2:~$ docker exec $C ping -nq -W 1 -c 1 10.0.0.2
PING 10.0.0.2 (10.0.0.2): 56 data bytes
--- 10.0.0.2 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss

docker logs weave on host2 shows

ERRO: 2016/07/10 08:07:41.346535 Captured frame from MAC (aa:c5:15:f8:a3:e8) associated with another peer 0e:2a:9b:8e:23:fd(host1)

Running weave report confirms that there is a corresponding MAC cache entry.

@bboreham was pondering whether we can make a better decision by inspecting the destination mac, i.e. only drop the mac if the source mac appears to be from a different peer and the destination mac is for a local peer.

Which, by my reading of the code, simply means turning the error into a warning and carrying on.

We'd need to think carefully in what situations that might create a loop.

[rade]

Hmm. Curiously making that suggested changed didn't fix the problem for me. Something weird is going on...

[rade]

There's a bug in the MAC cache: MacCache.AddForced never returns a conflict peer. That means the conflictPeer != nil branch in NetworkRouter.handleForwardedPacket is not reached, which in turn means we are not invalidating routes when MACs move.

I was invoking the same AddForced in the revised handleCapturedPacket code.

Curiously fixing that bug still does not make things work. Even though the logs clearly indicate that both peers detect the moved MAC.

[rade]

Curiously fixing that bug still does not make things work.

It does when I run with WEAVE_NO_FASTDP=1.

[rade]

when running with fastdp, after pinging from B to C

host1$ docker exec $B ping -nq -W 1 -c 1 10.0.0.3

(i.e. the opposite direction of the failing ping), then subsequently the ping from C to B starts working.

tcpdumping 'weave' on both machines shows that during the first C-to-B ping the ARP request gets through to B and a reply is sent, but that never appears on C.

[jakolehm]

What is the probability to get same mac address on different host? Just wondering because we hit this bug pretty easily.

[rade]

What is the probability to get same mac address on different host?

It's 46 random bits; so pretty low.

we hit this bug pretty easily.

I filed this bug separately from #2433 because I have no evidence to suggest that the symptoms there are explained by this.