four layers of gob is bad

[rade]

The sending of topology gossip currently goes through four layers of gob encoding, and involves n_peers+3 encoders...

• 1. one GobEncoder for each peer, into which we encode the connections
• 2. one GobEncoder for the set of peers, into which all peers are
  encoded, with their connection information represented by the
  result of the aforementioned encoding.
• 3. one GobEncoder for encoding the gossip (hash, source, and the
  result of (2)) as a ProtocolMsg payload.
• 4. one GobEncoder in the TCPSender, which is used to encode a byte
  slice - comprising the ProtocolMsg tag and payload - as a single
  message

This has a bad design smell to it and probably is also rather inefficient.

[rade]

Two layers of gob is the best we can do when crypto is enabled. Our crypto operates on message streams, i.e. streams of byte slices, rather than byte streams. The 4th gob layer converts between the two. And the 3rd layer converts between our protocol data objects and byte slices.

We could do something different in the non-crypto case but it's really not worth the hassle.

[rade]

I've done some more profiling, running

NUM_WEAVES=1 bin/multiweave launch -connlimit 100 -profile /tmp
NUM_WEAVES=49 FIRST_WEAVE=2 bin/multiweave launch -connlimit 100

(wait until everything goes quiet - takes about 3 minutes on my machine)

sudo kill -INT $(docker inspect --format '{{.State.Pid}}' weave1)
docker cp weave1:/tmp/cpu.pprof .
go tool pprof --nodefraction=0.02 --edgefraction=0.01 weaver/weaver cpu.pprof 

The gob codec takes up ~12%, in what is an extremely gob-heavy scenario. So while there is still room for improvement, it's not worth spending a lot of effort on it. So the main motivation to eliminate another layer would be improved design/APIs.