[HA] Show more info about the cluster status in the UI if it is failing

[giganteous]

What did you do?

I configured 2 alertmanagers to talk to each other with only TCP port 9094 open between each other, UDP port 9094 blocked.

What did you expect to see?

An indication of an error state.

What did you see instead? Under which circumstances?

• Cluster status in the UI: "ready" (<-- in green)
• a reasonably clean daemon output (without --log.level=debug)

Environment

• System information:

  Linux 4.9.0-6-amd64 x86_64

• Alertmanager version:

  alertmanager, version 0.15.2 (branch: HEAD, revision: d19fae3)
  build user: root@3101e5b68a55
  build date: 20180814-10:53:39
  go version: go1.10.3

• Prometheus version: [N/A]

• Alertmanager configuration file:

Port 9094 was used as port in the cluster.listen-address and cluster.peer options.

• Prometheus configuration file:

insert configuration here (if relevant to the issue)

• Logs:

[WARN] memberlist: Was able to connect to 01CRQDCBH4NS9VTT139B9ECNFN but other probes failed, network may be misconfigured

Important: This warn message did not show until I ran both ends with --log.level=debug.

[mxinden]

We are using hashicorp's memberlist library to cluster Alertmanager instances. The library makes it transparent to the user whether it uses UDP or TCP. The default should be UDP, it falls back to TCP for gossip messages otherwise.

When you create a silence on one of the Alertmanager instances, does it eventually show up on the other?

[giganteous]

Yes, but double notifications are an extra result (sometimes). See #1550 for an example.

[giganteous]

And: thanks! today I learned.

Hashicorps memberlist library has no documentation for end users; it might be wonderful to use as a developer, but this sysadmin is pulling his hairs :-)

[mxinden]

PRs like #1563 are very welcome @giganteous. I am sorry for the trouble you went through.

Not sure if we can surface those issues in the UI. I need to have a deeper look into memberlist in this regard.

[nooshin-mirzadeh]

I had the same issue. I've got this message after enabling the debug level:

level=debug ts=2019-12-02T13:41:53.896Z caller=cluster.go:306 component=cluster memberlist="2019/12/02 13:41:53 [WARN] memberlist: Was able to connect to 01DV3CGDM8H91BFE4WEPAF5FV5 but other probes failed, network may be misconfigured\n"

Why the level is debug and then [WARN]? Is it possible to set it in the level warn at the beginning?

[simonpasquier]

Why the level is debug and then [WARN]? Is it possible to set it in the level warn at the beginning?

Because the message comes from the memberlist logging system and we couldn't hook it up properly with the Alertmanager logging. So memberlist log messages are only displayed when debug level is turned on.

[nooshin-mirzadeh]

Why the level is debug and then [WARN]? Is it possible to set it in the level warn at the beginning?

Because the message comes from the memberlist logging system and we couldn't hook it up properly with the Alertmanager logging. So memberlist log messages are only displayed when debug level is turned on.

So can't you please check the log level of the memebrlist in this function and then set the log level? instead of always setting it at the debug level?