Amplification of pod traffic

[erik-stephens]

I think this is "works as designed". However, I think the side-effects are too great as the cluster grows.

Our peer connection limit is left at the default of 100 and we have 58 nodes. We could reduce that to help minimize the effect but that will only take us so far.

What you expected to happen?

Less broadcasting of traffic / links to not get saturated.

What happened?

Weave forwards traffic that it does not know the destination for to all of its peers. As the cluster grows, the amount of amplified traffic grows even greater.

How to reproduce it?

I think pod scheduling activity and maybe some periodic gossip/discovery phase triggers a peer to clear its table then re-populate it. Based on log output, we're seeing roughly 1-2 peers discovered per second. For a cluster of our size (and growing), that's a relatively large window for traffic to be amplified.

• route changes triggering route invalidation
• all the flows deleted leaving big window for amplification?

Anything else we need to know?

On bare-metal provisioned with kubeadm

Versions:

$ weave version
weave 2.5.1

$ docker version
Client:
 Version:           18.06.2-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        6d37f41
 Built:             Sun Feb 10 03:48:06 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.2-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       6d37f41
  Built:            Sun Feb 10 03:46:30 2019
  OS/Arch:          linux/amd64
  Experimental:     false

$ uname -a
Linux r26c4na 4.4.0-142-generic #168-Ubuntu SMP Wed Jan 16 21:00:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.2", GitCommit:"cff46ab41ff0bb44d8584413b598ad8360ec1def", GitTreeState:"clean", BuildDate:"2019-01-13T23:16:01Z", GoVersion:"go1.11.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.2", GitCommit:"cff46ab41ff0bb44d8584413b598ad8360ec1def", GitTreeState:"clean", BuildDate:"2019-01-10T23:28:14Z", GoVersion:"go1.11.4", Compiler:"gc", Platform:"linux/amd64"}

Logs:

Our biggest clue came from seeing pod traffic spike in unrelated pods during a large hourly workload in other pods, sometimes saturating the link.

peer-1: Discovered remote MAC ...
peer-2: Discovered remote MAC ...
peer-1: Captured frame from MAC ... associated with peer-2

Evidence that peers are receiving broadcasted traffic during the window where it populates its tables.

Network:

Our network seems fine: no packet loss, low latency, bonded interfaces.

[murali-reddy]

@erik-stephens thanks for reporting the issue

I think pod scheduling activity and maybe some periodic gossip/discovery phase triggers a peer to clear its table then re-populate it. Based on log output, we're seeing roughly 1-2 peers discovered per second.

So there are two aspects. One is the case where the network topology is changing (i.e nodes are getting added or deleted) in which case routes are recalculated and InvalidateRoutes() method you shared is called. In this case existing flows are flushed. If you enable DEBUG logging you should see logs like

DEBU: 2019/02/28 09:19:41.241824 InvalidateRoutes

Please check in the logs if InvalidateRoutes() is actually getting called.

Second aspect is pod are getting created and destroyed this does not change the topology and hence routes are not recalculated.

Only other case where routes are invalidated is when there is conflict of learned MAC. In which case you might see this issue.

Could you please share sample of DEBUG enabled logs when you observe link saturation?

Also if possible please enable -pkt-debug so that we get clear view of the activity.

[nvson28]

Hi @erik-stephens ,
It seems that we are facing a similar issue with yours. I'm curious, since the limit for peer connection is 100 and your cluster has 58 nodes, that means each node has connections to all of its peers.

1. How is it possible that weave needs to send traffic to a destination that it does not know?
2. How did you check that there was traffic to unknown destinations.

Thanks in advance.

[erik-stephens]

I should make it clear that there is a good chance my analysis is incorrect - I'm just an operator:

1. It's my understanding that a weave pod will broadcast traffic that it doesn't know how to route to all of its peers. I think it's an intentional design trade-off to work around network partitions. I suspect there is more "clear table then re-populate" going on than expected and it's in these windows where we see this broadcasting.

2. We have an hourly workload that produces a pretty unique traffic signature. Seeing that signature in other pods' traffic is our biggest clue. We need to try to reproduce with debugging enabled in our test cluster for more concrete evidence.

[nvson28]

I tried to look further into the amplified traffic issue and found that:

1. Weave pods constantly log out errors like this: ERRO: 2019/03/04 06:54:37.532838 Captured frame from MAC (72:bf:b6:15:cb:23) to (b6:15:ff:83:6f:3e) associated with another peer 72:bf:b6:15:cb:23(node-1)
2. In the cluster, some nodes have much higher UDP traffic going through port 6784 than other nodes with the same functionality (about double the UDP traffic)
3. When I look at weave report on a node, under Router/MACs, I see a lot of duplicated entries for a node:

{
                "Mac": "72:bf:b6:15:cb:23",
                "Name": "72:bf:b6:15:cb:23",
                "NickName": "node-1",
                "LastSeen": "2019-03-04T04:14:41.187709378Z"
},
{
                "Mac": "b6:15:ff:83:6f:3e",
                "Name": "72:bf:b6:15:cb:23",
                "NickName": "node-1",
                "LastSeen": "2019-03-04T04:14:41.190557712Z"
}
...

I noticed that there are about 10 entries for this node-1 alone, and it happens for a few other
nodes in my cluster as well.
Also, only in the first entry, the Mac and Name values are the same, while for other entries, the Mac and Name values are different.

4. The to MAC from (1.) also appears in the MAC list in (3.), in this case its b6:15:ff:83:6f:3e

I would really appreciate any pointers to this issue. Thanks a lot.

[murali-reddy]

It seems that we are facing a similar issue with yours.

@nvson28 What are the symptoms you are seeing, are you observing excessive packet broadcasts?

point-3 means, a mac address is associated with a peer (under Name), there can be several mac's associated with a peer.

AFAICS the error you observed can not be cause for broadcasting packets.

[nvson28]

Hi @murali-reddy
We have a Kubernetes cluster using weave, cluster size is about 50 nodes. The observed symptoms are:

1. There are 3 servers with the same functionality (acting as proxies), with almost the same incoming traffic. However, among those 3 servers, some exhibits much larger outgoing bandwidth (about twice the bandwidth) than the others.
2. I monitor the outgoing traffic using tcpdump, and found that the "affected" proxy sends a lot of data via UDP port 6784 to all other nodes in the cluster. While for "unaffected" proxies, the data sent over UDP port 6784 to other nodes is much less.
3. If I delete the weave-net pod on an "affected" server, the outgoing traffic becomes low, but will strike back high again some time later (probably 1 to 2 days)

Really appreciate any pointers. Thanks!

[bboreham]

Just a brief observation that traffic on 6784 UDP is going via the "fastdp" channel, i.e. directed by flows in the ODP kernel module. If the problem were related to route (flow) invalidation after a topology change then you should see the traffic on 6783 UDP which is where the "sleeve" transport sends it prior to setting up fastdp flows.

If you get tcpdump to save the packets to a file you should be able to look inside, using a tool like Wireshark, and examine the to/from MAC addresses in the encapsulated packets, thereby gain some clue what this extra traffic is.

[nvson28]

Hi @bboreham ,
Thanks for your suggestion. I have tried to visualize the UDP port 6784 data via Wireshark, and found that:

• The UDP packets encapsulate our HTTP requests from the "affected proxy" server to upstreams
• The same HTTP request is encapsulated and sent to ALL the nodes in the cluster from the "affected proxy".
• And for the proxy POD (in k8s), we are using host network. Not sure if that adds any clue to the problem

Btw, is there a way to decrypt the data added by weave in the UDP packets ? On Wireshark, in the UDP packets, only the encapsulated HTTP content can be visualized in plain text.

[bboreham]

Traffic sent to all nodes implies it thought the destination address should be on the Weave network but didn't know which node that MAC address lived on. But you are saying the destination was outside of the Weave network?

I can't quite visualise the flow - could you draw it out somehow (even ASCII) and add the subnets?
Also the route table on the host and perhaps inside the source pod.

Port 6784 traffic is VXLAN-encapsulated, so you should be able to get Wireshark to decode that.
The format is defined at https://tools.ietf.org/html/rfc7348.

[erik-stephens]

Closing since we are on another driver - sorry, wasn't able to debug this further.