Flaky Test: TestClusterJoinAndReconnect/TestJoinLeave flake

[gotjosh]

Per https://app.circleci.com/pipelines/github/prometheus/alertmanager/3414/workflows/c8f0b621-4406-438c-96ee-be09c7a45d92/jobs/15139 seems to be flaky but really hard to reproduce.

It took me a few retries before I saw it failing:

🙇 -> go test -run "TestClusterJoinAndReconnect/TestJoinLeave" -count 300
--- FAIL: TestClusterJoinAndReconnect (0.02s)
    --- FAIL: TestClusterJoinAndReconnect/TestJoinLeave (0.00s)
        cluster_test.go:104: 
                Error Trace:    /Users/gotjosh/go/src/github.com/prometheus/alertmanager/cluster/cluster_test.go:104
                Error:          Not equal: 
                                expected: 2
                                actual  : 1
                Test:           TestClusterJoinAndReconnect/TestJoinLeave
FAIL
exit status 1
FAIL    github.com/prometheus/alertmanager/cluster      249.449s

[dswarbrick]

This test fails pretty regularly during autopkgtest in the prometheus-alertmanager Debian package also. It seems to be less reliable on non-amd64 archs. Also, what strikes me as curious is how quickly it fails. Something is failing very early, but the tests don't log enough detail to ascertain what.

arm64:

152s --- FAIL: TestClusterJoinAndReconnect (0.84s)
152s     --- PASS: TestClusterJoinAndReconnect/TestJoinLeave (0.80s)
152s     --- PASS: TestClusterJoinAndReconnect/TestReconnect (0.00s)
152s     --- PASS: TestClusterJoinAndReconnect/TestRemoveFailedPeers (0.00s)
152s     --- PASS: TestClusterJoinAndReconnect/TestInitiallyFailingPeers (0.00s)
152s     --- FAIL: TestClusterJoinAndReconnect/TestTLSConnection (0.04s)

armhf:

152s --- FAIL: TestClusterJoinAndReconnect (1.01s)
152s     --- PASS: TestClusterJoinAndReconnect/TestJoinLeave (0.81s)
152s     --- PASS: TestClusterJoinAndReconnect/TestReconnect (0.00s)
152s     --- PASS: TestClusterJoinAndReconnect/TestRemoveFailedPeers (0.00s)
152s     --- PASS: TestClusterJoinAndReconnect/TestInitiallyFailingPeers (0.00s)
152s     --- FAIL: TestClusterJoinAndReconnect/TestTLSConnection (0.19s)

i386:

251s --- FAIL: TestClusterJoinAndReconnect (0.96s)
251s     --- PASS: TestClusterJoinAndReconnect/TestJoinLeave (0.81s)
251s     --- PASS: TestClusterJoinAndReconnect/TestReconnect (0.00s)
251s     --- PASS: TestClusterJoinAndReconnect/TestRemoveFailedPeers (0.00s)
251s     --- PASS: TestClusterJoinAndReconnect/TestInitiallyFailingPeers (0.00s)
251s     --- FAIL: TestClusterJoinAndReconnect/TestTLSConnection (0.15s)

s390x:

163s --- FAIL: TestClusterJoinAndReconnect (0.82s)
163s     --- PASS: TestClusterJoinAndReconnect/TestJoinLeave (0.80s)
163s     --- PASS: TestClusterJoinAndReconnect/TestReconnect (0.00s)
163s     --- PASS: TestClusterJoinAndReconnect/TestRemoveFailedPeers (0.00s)
163s     --- PASS: TestClusterJoinAndReconnect/TestInitiallyFailingPeers (0.00s)
163s     --- FAIL: TestClusterJoinAndReconnect/TestTLSConnection (0.02s)

Compare to a successful test, where TestTLSConnection nearly always takes between 0.8s and 0.85s:

149s === RUN   TestClusterJoinAndReconnect/TestTLSConnection
149s --- PASS: TestClusterJoinAndReconnect (1.61s)
149s     --- PASS: TestClusterJoinAndReconnect/TestJoinLeave (0.80s)
149s     --- PASS: TestClusterJoinAndReconnect/TestReconnect (0.00s)
149s     --- PASS: TestClusterJoinAndReconnect/TestRemoveFailedPeers (0.00s)
149s     --- PASS: TestClusterJoinAndReconnect/TestInitiallyFailingPeers (0.00s)
149s     --- PASS: TestClusterJoinAndReconnect/TestTLSConnection (0.80s)

[TheMeier]

Still seeing this from time to time.