potential test issues

[ggggggggg]

motivated by the failure of recent dastard to work with autotune, I looked at the tests here to see if I could identify any red flags. I believe we're currently using the pebbe zmq, so I focused on that test.

This sets the sender high water mark to 1.1 million, when you will send exactly 1 million packets only. That differs from real usage in that we can never have the hwm be above our total number of packets.

golang-zmq-comparison/pebbe_test.go

Line 68 in f5f61c5

publisher.SetSndhwm(1100000)

Here we see the payload is always "Rhubarb", much smaller than the typical tdm payload for autotune of 10k samples

golang-zmq-comparison/pebbe_test.go

Line 87 in f5f61c5

publisher.Send("Rhubarb", 0)

Further, you start from a synchronized point... eg no messages in the queue, get all subscribers connected, then send all messages. In real usage we start with the publisher having already sent enough messages to fill the hwm many times.

Finally, the sender and reciever are using the same package, I suspect the packages is mostly tested against itself. Perhaps there is variation in behavior from package to package that causes issues?

I don't have any proposed actions currently, just wanted to document some differences.

[joefowler]

These are fair points. Do you propose that we design more realistic tests and try them?

That's reasonable, but I am not sure we can realistically choose any of the other 3 packages. The one we used to use seems like to get further and further behind the current and future releases of the underlying czmq library. And the other two are nowhere near complete enough or fast enough to suit our high-performance needs.

[joefowler]

Incidentally, my handwritten notes from Sept 20, 2022 show that I never observed a single message when using the current pebbe/zmq4 package in these tests. By contrast, the package zeromq/goczmq (our previous choice) dropped 1%-4% of all packets. (I said to you on Slack that they both dropped a small fraction of packets, but in fact I see that the new library didn't actually drop any at all.)

Okay, I admit there are weaknesses in the tests, as this issue raises. But I still stand by our choice of the pebbe module.

[ggggggggg]

I could imagine repeating all these tests with a receiver written in python and a payload closer to what is used for autotune, but I'm not super enthusiastic about doing so. Maybe a better approach would be to test against dastard and write a "packet drop detector" in various zmq receiving languages, and see if we can find a config where we observe no packet drops over some period of time from the way dastard actually sends packets. Maybe that could guide a re-write of the autotune acquisition code that gets all packets. It's so hard to figure out where a problem is with zmq.

It was clear at the EBIT that we would benefit from being on a modern version of dastard, eg the ability to turn off triggering on a channel would have been useful.