TestRetry fails on Darwin

[ryanwuer]

What did you do?

• clone the repo
• checkout to branch 'release-0.21'
• first execute 'make build' , then execute 'make test'

What did you expect to see?
All tests can be passed

What did you see instead? Under which circumstances?
--- FAIL: TestRetry (6.51s)
acceptance.go:179: stdout:
acceptance.go:180: stderr:
level=info ts=2021-01-21T09:39:04.729Z caller=main.go:220 msg="Starting Alertmanager" version="(version=0.21.0, branch=develop, revision=94386bf72204899a96b83e1b6d22fce1c1306e4f)"
level=info ts=2021-01-21T09:39:04.729Z caller=main.go:221 build_context="(go=go1.15.3, user=wrj@MacBook-Pro-WRJ.local, date=20210121-09:04:44)"
level=debug ts=2021-01-21T09:39:04.729Z caller=cluster.go:149 component=cluster msg="resolved peers to following addresses" peers=127.0.0.1:50007
level=debug ts=2021-01-21T09:39:04.729Z caller=delegate.go:230 component=cluster received=NotifyJoin node=01EWJ4ZJCSG0953MEZDW06QXPM addr=127.0.0.1:50007
level=debug ts=2021-01-21T09:39:04.730Z caller=cluster.go:306 component=cluster memberlist="2021/01/21 17:39:04 [DEBUG] memberlist: Initiating push/pull sync with: 127.0.0.1:50007\n"
level=debug ts=2021-01-21T09:39:04.730Z caller=cluster.go:306 component=cluster memberlist="2021/01/21 17:39:04 [DEBUG] memberlist: Stream connection from=127.0.0.1:50022\n"
level=debug ts=2021-01-21T09:39:04.730Z caller=cluster.go:233 component=cluster msg="joined cluster" peers=1
level=info ts=2021-01-21T09:39:04.730Z caller=cluster.go:623 component=cluster msg="Waiting for gossip to settle..." interval=2s
level=info ts=2021-01-21T09:39:04.730Z caller=cluster.go:632 component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=1.56µs
level=debug ts=2021-01-21T09:39:04.749Z caller=main.go:359 externalURL=http://MacBook-Pro-WRJ.local:50004
level=info ts=2021-01-21T09:39:04.749Z caller=coordinator.go:119 component=configuration msg="Loading configuration file" file=/var/folders/_n/mbr_vhxs7k34wmqjsrcr2fdh0000gp/T/am_test077575705/config.yml
level=info ts=2021-01-21T09:39:04.749Z caller=coordinator.go:131 component=configuration msg="Completed loading of configuration file" file=/var/folders/_n/mbr_vhxs7k34wmqjsrcr2fdh0000gp/T/am_test077575705/config.yml
level=debug ts=2021-01-21T09:39:04.752Z caller=main.go:469 routePrefix=/
level=info ts=2021-01-21T09:39:04.753Z caller=main.go:489 msg=Listening address=127.0.0.1:50004
level=debug ts=2021-01-21T09:39:05.521Z caller=dispatch.go:138 component=dispatcher msg="Received alert" alert=test1[5ae90ff][active]
level=debug ts=2021-01-21T09:39:06.523Z caller=dispatch.go:473 component=dispatcher aggrGroup="{}:{alertname="test1"}" msg=flushing alerts=[test1[5ae90ff][active]]
level=debug ts=2021-01-21T09:39:06.526Z caller=notify.go:685 component=dispatcher receiver=default integration=webhook[0] msg="Notify success" attempts=1
level=debug ts=2021-01-21T09:39:06.526Z caller=notify.go:685 component=dispatcher receiver=default integration=webhook[1] msg="Notify success" attempts=1
level=debug ts=2021-01-21T09:39:07.526Z caller=dispatch.go:473 component=dispatcher aggrGroup="{}:{alertname="test1"}" msg=flushing alerts=[test1[5ae90ff][active]]
level=debug ts=2021-01-21T09:39:08.530Z caller=dispatch.go:473 component=dispatcher aggrGroup="{}:{alertname="test1"}" msg=flushing alerts=[test1[5ae90ff][active]]
level=debug ts=2021-01-21T09:39:09.531Z caller=dispatch.go:473 component=dispatcher aggrGroup="{}:{alertname="test1"}" msg=flushing alerts=[test1[5ae90ff][active]]
level=debug ts=2021-01-21T09:39:09.532Z caller=notify.go:685 component=dispatcher receiver=default integration=webhook[1] msg="Notify success" attempts=1
level=debug ts=2021-01-21T09:39:09.532Z caller=notify.go:685 component=dispatcher receiver=default integration=webhook[0] msg="Notify success" attempts=1
level=debug ts=2021-01-21T09:39:10.533Z caller=dispatch.go:473 component=dispatcher aggrGroup="{}:{alertname="test1"}" msg=flushing alerts=[test1[5ae90ff][active]]
level=info ts=2021-01-21T09:39:11.019Z caller=main.go:528 msg="Received SIGTERM, exiting gracefully..."
level=debug ts=2021-01-21T09:39:11.019Z caller=cluster.go:539 component=cluster msg="leaving cluster"
level=debug ts=2021-01-21T09:39:11.019Z caller=delegate.go:236 component=cluster received=NotifyLeave node=01EWJ4ZJCSG0953MEZDW06QXPM addr=127.0.0.1:50007
level=debug ts=2021-01-21T09:39:11.019Z caller=cluster.go:492 component=cluster msg="peer left" peer=01EWJ4ZJCSG0953MEZDW06QXPM
send_test.go:217:
collector "webhook":

    interval [6,6.5]
    ---
    - &{map[] 0001-01-01T00:00:00.000Z <nil> [] 2021-01-21T17:39:05.515+08:00 <nil> <nil> { map[alertname:test1]}}[1:]
      [ ✗ ]
    interval [2,2.5]
    ---
    - &{map[] 0001-01-01T00:00:00.000Z <nil> [] 2021-01-21T17:39:05.515+08:00 <nil> <nil> { map[alertname:test1]}}[1:]
      [ ✓ ]
    
    received:
    @ 2.010035187
    - &{map[] 0001-01-01T00:00:00.000Z <nil> [] 2021-01-21T17:39:05.521+08:00 <nil> <nil> { map[alertname:test1]}}[1.005199:]
    @ 5.016554237
    - &{map[] 0001-01-01T00:00:00.000Z <nil> [] 2021-01-21T17:39:05.521+08:00 <nil> <nil> { map[alertname:test1]}}[1.005199:]
    
send_test.go:217: 
    collector "webhook_failing":
    
    interval [6,6.5]
    ---
    - &{map[] 0001-01-01T00:00:00.000Z <nil> [] 2021-01-21T17:39:05.515+08:00 <nil> <nil> { map[alertname:test1]}}[1:]
      [ ✗ ]
    
    received:
    @ 5.016386806
    - &{map[] 0001-01-01T00:00:00.000Z <nil> [] 2021-01-21T17:39:05.521+08:00 <nil> <nil> { map[alertname:test1]}}[1.005199:]

FAIL
FAIL github.com/prometheus/alertmanager/test/with_api_v2/acceptance 26.649s
ok github.com/prometheus/alertmanager/types (cached)
? github.com/prometheus/alertmanager/ui [no test files]
FAIL
make: *** [common-test] Error 1

• System information:
  Darwin 19.6.0 x86_64

• Alertmanager version:
  v0.21

[beorn7]

In my experience, those tests are a bit flaky at times. Do you get this error consistently if you retry running make test?

Help to make these tests less flaky is certainly welcome.

[ryanwuer]

Yeah, i got this error consistently. Never succeed.

[beorn7]

Perhaps this test is reliably failing on Darwin only? Anyway, help appreciated…