Alert Manager Flapping

[MattPOlson]

What did you do?

Deployed an AlertManager HA cluster as documented in Docker containers.

What did you expect to see?

The cluster to behave as expected and not send duplicate alerts.

What did you see instead? Under which circumstances?

Both nodes continue to drop in and out of the cluster, causing duplicate alerts.

Environment

• System information:

  Linux 3.10.0-957.12.2.el7.x86_64 x86_64

• Alertmanager version:

  alertmanager, version 0.15.3 (branch: HEAD, revision: d4a7697)
  build user: root@4ecc17c53d26
  build date: 20181109-15:40:48
  go version: go1.11.2

• Prometheus version:

  prometheus, version 2.5.0 (branch: HEAD, revision: 67dc912ac8b24f94a1fc478f352d25179c94ab9b)
  build user: root@578ab108d0b9
  build date: 20181106-11:40:44
  go version: go1.11.1

• Alertmanager configuration file:

global:
    slack_api_url: 'https://hooks.slack.com/services/'

templates:
  - '/etc/alertmanager/*.tmpl'

route:
  receiver: default-receiver

receivers:
- name: default-receiver
  slack_configs:
  - channel: "#prometheus"
    icon_url: 
    title: '{{ template "custom_title" . }}'
    text: '{{ template "custom_slack_message" . }}'
    send_resolved: true

Command Line Info
command:
      - '--config.file=/etc/alertmanager/alertmanager.yml'
      - '--storage.path=/alertmanager'
      - '--web.external-url=http://alertmanager.${URLSLUG:-dev}.bdreporting.local'
      - '--cluster.peer=alertmanager_2:9094'
      - '--log.level=debug'
      - '--cluster.probe-timeout=1s'
      - '--cluster.probe-interval=2s'

• Prometheus configuration file:

global:
  evaluation_interval: 30s
  scrape_interval: 1m
  scrape_timeout: 40s

rule_files:
  - "rabbit.rules.yml"
  - "linux.host.rules.yml"
  - "windows.host.rules.yml"
  - "disk.rules.yml"
  - "service.rules.yml"
  - "msmq.rules.yml"

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'RabbitMQ'
    scrape_interval: 5s
    static_configs:
      - targets: ['***********:9419']

  - job_name: '***********'
    scrape_interval: 5s
    static_configs:
      - targets: ['***********']

  - job_name: '***********'
    scrape_interval: 5s
    static_configs:
      - targets: ['***********']

  - job_name: '***********'
    scrape_interval: 5s
    static_configs:
      - targets: ['***********:9100']

  - job_name: '***********'
    scrape_interval: 5s
    static_configs:
      - targets: ['***********:9182']

  - job_name: '***********'
    scrape_interval: 5s
    static_configs:
      - targets: ['***********:9182']

  - job_name: '***********'
    scrape_interval: 5s
    static_configs:
      - targets: ['***********:9182']

  - job_name: '***********'
    scrape_interval: 5s
    static_configs:
      - targets: ['***********:9182']

  - job_name: '***********'
    scrape_interval: 5s
    static_configs:
      - targets: ['***********:9182']

  - job_name: '***********'
    scrape_interval: 5s
    static_configs:
      - targets: ['***********:9182']

  - job_name: '***********'
    scrape_interval: 5s
    static_configs:
      - targets: ['***********:9182']

  - job_name: '***********'
    scrape_interval: 5s
    static_configs:
      - targets: ['***********:9182']

  - job_name: '***********'
    scrape_interval: 5s
    static_configs:
      - targets: ['***********:9100']

  - job_name: '***********'
    scrape_interval: 5s
    static_configs:
      - targets: ['***********:9100']

  - job_name: '***********'
    scrape_interval: 5s
    static_configs:
      - targets: ['***********:9100']

  - job_name: '***********'
    scrape_interval: 5s
    static_configs:
      - targets: ['***********:9100']
      
  - job_name: '***********'
    scrape_interval: 5s
    static_configs:
      - targets: ['***********:9100']

  - job_name: '***********'
    scrape_interval: 5s
    static_configs:
      - targets: ['***********:9100']

  - job_name: '***********'
    scrape_interval: 5s
    static_configs:
      - targets: ['***********:9100']

  - job_name: '***********'
    scrape_interval: 5s
    static_configs:
      - targets: ['***********:9100']

alerting:
  alertmanagers:
  - scheme: http
    static_configs:
    - targets: ['alertmanager_1:9093','alertmanager_2:9093']

• Logs:

level=info ts=2019-06-03T14:58:18.116285222Z caller=main.go:174 msg="Starting Alertmanager" version="(version=0.15.3, branch=HEAD, revision=d4a7697cc90f8bce62efe7c44b63b542578ec0a1)"
level=info ts=2019-06-03T14:58:18.116347974Z caller=main.go:175 build_context="(go=go1.11.2, user=root@4ecc17c53d26, date=20181109-15:40:48)"
level=debug ts=2019-06-03T14:58:18.117402164Z caller=cluster.go:143 component=cluster msg="resolved peers to following addresses" peers=10.0.2.75:9094
level=info ts=2019-06-03T14:58:18.12031295Z caller=cluster.go:155 component=cluster msg="setting advertise address explicitly" addr=172.18.0.5 port=9094
level=debug ts=2019-06-03T14:58:18.121806787Z caller=delegate.go:209 component=cluster received=NotifyJoin node=01DCEXJ8W8JSCTPD03JEC56TYG addr=172.18.0.5:9094
level=debug ts=2019-06-03T14:58:18.122796452Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 14:58:18 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T14:58:18.123695101Z caller=delegate.go:209 component=cluster received=NotifyJoin node=01DCC87CWBY4X4DK18CJVR6KWY addr=172.18.0.3:9094
level=debug ts=2019-06-03T14:58:18.123772653Z caller=nflog.go:534 component=nflog msg="gossiping new entry" entry="entry:<group_key:\"{}:{}\" receiver:<group_name:\"default-receiver\" integration:\"slack\" > timestamp:<seconds:1559571001 nanos:459212304 > resolved_alerts:4473130254019522639 > expires_at:<seconds:1560003001 nanos:459212304 > "
level=debug ts=2019-06-03T14:58:18.123924483Z caller=cluster.go:224 component=cluster msg="joined cluster" peers=1
level=info ts=2019-06-03T14:58:18.123966776Z caller=main.go:322 msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml
level=info ts=2019-06-03T14:58:18.123966458Z caller=cluster.go:570 component=cluster msg="Waiting for gossip to settle..." interval=2s
level=info ts=2019-06-03T14:58:18.128455464Z caller=main.go:398 msg=Listening address=:9093
level=info ts=2019-06-03T14:58:20.124128464Z caller=cluster.go:595 component=cluster msg="gossip not settled" polls=0 before=0 now=2 elapsed=2.000109457s
level=debug ts=2019-06-03T14:58:21.122283129Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 14:58:21 [DEBUG] memberlist: Failed ping: 01DCC87CWBY4X4DK18CJVR6KWY (timeout reached)\n"
level=debug ts=2019-06-03T14:58:22.124284346Z caller=cluster.go:592 component=cluster msg="gossip looks settled" elapsed=4.000268229s
level=debug ts=2019-06-03T14:58:24.124446227Z caller=cluster.go:592 component=cluster msg="gossip looks settled" elapsed=6.000401218s
level=debug ts=2019-06-03T14:58:26.124593188Z caller=cluster.go:592 component=cluster msg="gossip looks settled" elapsed=8.000576192s
level=debug ts=2019-06-03T14:58:28.124765762Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 14:58:28 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=info ts=2019-06-03T14:58:28.124772758Z caller=cluster.go:587 component=cluster msg="gossip settled; proceeding" elapsed=10.000756916s
level=debug ts=2019-06-03T14:58:28.125663236Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T14:58:31.122496694Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 14:58:31 [INFO] memberlist: Suspect 01DCC87CWBY4X4DK18CJVR6KWY has failed, no acks received\n"
level=debug ts=2019-06-03T14:58:31.863106986Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 14:58:31 [DEBUG] memberlist: Stream connection from=10.0.2.10:35646\n"
level=debug ts=2019-06-03T14:58:31.863526841Z caller=delegate.go:209 component=cluster received=NotifyJoin node=01DCEXJP9K726FK207GYR8JJV3 addr=172.18.0.3:9094
level=debug ts=2019-06-03T14:58:33.122240332Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 14:58:33 [DEBUG] memberlist: Failed ping: 01DCC87CWBY4X4DK18CJVR6KWY (timeout reached)\n"
level=debug ts=2019-06-03T14:58:38.124775953Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 14:58:38 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T14:58:38.125520128Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 14:58:38 [WARN] memberlist: Refuting a suspect message (from: 01DCEXJ8W8JSCTPD03JEC56TYG)\n"
level=debug ts=2019-06-03T14:58:38.125585824Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T14:58:39.122728947Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 14:58:39 [INFO] memberlist: Marking 01DCC87CWBY4X4DK18CJVR6KWY as failed, suspect timeout reached (0 peer confirmations)\n"
level=debug ts=2019-06-03T14:58:39.122810324Z caller=delegate.go:215 component=cluster received=NotifyLeave node=01DCC87CWBY4X4DK18CJVR6KWY addr=172.18.0.3:9094
level=debug ts=2019-06-03T14:58:39.122824667Z caller=cluster.go:439 component=cluster msg="peer left" peer=01DCEXJP9K726FK207GYR8JJV3
level=debug ts=2019-06-03T14:58:41.864182973Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 14:58:41 [DEBUG] memberlist: Stream connection from=10.0.2.10:35654\n"
level=debug ts=2019-06-03T14:58:43.122603461Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 14:58:43 [INFO] memberlist: Suspect 01DCC87CWBY4X4DK18CJVR6KWY has failed, no acks received\n"
level=debug ts=2019-06-03T14:58:44.122878515Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 14:58:44 [DEBUG] memberlist: Failed ping: 01DCEXJP9K726FK207GYR8JJV3 (timeout reached)\n"
level=debug ts=2019-06-03T14:58:48.124846859Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 14:58:48 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T14:58:48.125805442Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T14:58:51.86367012Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 14:58:51 [DEBUG] memberlist: Stream connection from=10.0.2.10:35662\n"
level=debug ts=2019-06-03T14:58:51.864624139Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 14:58:51 [WARN] memberlist: Refuting a suspect message (from: 01DCEXJ8W8JSCTPD03JEC56TYG)\n"
level=debug ts=2019-06-03T14:58:54.123072644Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 14:58:54 [INFO] memberlist: Suspect 01DCEXJP9K726FK207GYR8JJV3 has failed, no acks received\n"
level=debug ts=2019-06-03T14:58:57.122205934Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 14:58:57 [DEBUG] memberlist: Failed ping: 01DCEXJP9K726FK207GYR8JJV3 (timeout reached)\n"
level=debug ts=2019-06-03T14:58:58.125993497Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 14:58:58 [DEBUG] memberlist: Failed to join 172.18.0.3: dial tcp 172.18.0.3:9094: i/o timeout\n"
level=debug ts=2019-06-03T14:58:58.126035633Z caller=cluster.go:386 component=cluster msg=reconnect result=failure peer=01DCEXJP9K726FK207GYR8JJV3 addr=172.18.0.3:9094
level=debug ts=2019-06-03T14:58:58.126636485Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 14:58:58 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T14:58:58.127429002Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T14:59:01.864524128Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 14:59:01 [DEBUG] memberlist: Stream connection from=10.0.2.10:35668\n"
level=debug ts=2019-06-03T14:59:08.122100704Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 14:59:08 [INFO] memberlist: Suspect 01DCEXJP9K726FK207GYR8JJV3 has failed, no acks received\n"
level=debug ts=2019-06-03T14:59:08.127587918Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 14:59:08 [DEBUG] memberlist: Failed to join 172.18.0.3: dial tcp 172.18.0.3:9094: i/o timeout\n"
level=debug ts=2019-06-03T14:59:08.127613826Z caller=cluster.go:386 component=cluster msg=reconnect result=failure peer=01DCEXJP9K726FK207GYR8JJV3 addr=172.18.0.3:9094
level=debug ts=2019-06-03T14:59:08.12818742Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 14:59:08 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T14:59:08.129004399Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T14:59:09.122385495Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 14:59:09 [DEBUG] memberlist: Failed ping: 01DCEXJP9K726FK207GYR8JJV3 (timeout reached)\n"
level=debug ts=2019-06-03T14:59:11.86346731Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 14:59:11 [DEBUG] memberlist: Stream connection from=10.0.2.10:35672\n"
level=debug ts=2019-06-03T14:59:11.863818568Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 14:59:11 [WARN] memberlist: Refuting a suspect message (from: 01DCEXJ8W8JSCTPD03JEC56TYG)\n"
level=debug ts=2019-06-03T14:59:13.376051197Z caller=dispatch.go:201 component=dispatcher msg="Received alert" alert=rabbitmq_queue_stuck[70edefc][active]
level=debug ts=2019-06-03T14:59:13.376238326Z caller=dispatch.go:445 component=dispatcher aggrGroup={}:{} msg=Flushing alerts=[rabbitmq_queue_stuck[70edefc][active]]
level=debug ts=2019-06-03T14:59:18.129225349Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 14:59:18 [DEBUG] memberlist: Failed to join 172.18.0.3: dial tcp 172.18.0.3:9094: i/o timeout\n"
level=debug ts=2019-06-03T14:59:18.129271644Z caller=cluster.go:386 component=cluster msg=reconnect result=failure peer=01DCEXJP9K726FK207GYR8JJV3 addr=172.18.0.3:9094
level=debug ts=2019-06-03T14:59:18.129939747Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 14:59:18 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T14:59:18.131069712Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T14:59:21.862993812Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 14:59:21 [DEBUG] memberlist: Stream connection from=10.0.2.10:35678\n"
level=debug ts=2019-06-03T14:59:22.122274973Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 14:59:22 [INFO] memberlist: Suspect 01DCEXJP9K726FK207GYR8JJV3 has failed, no acks received\n"
level=debug ts=2019-06-03T14:59:25.122210841Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 14:59:25 [DEBUG] memberlist: Failed ping: 01DCEXJP9K726FK207GYR8JJV3 (timeout reached)\n"
level=debug ts=2019-06-03T14:59:28.131484304Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 14:59:28 [DEBUG] memberlist: Failed to join 172.18.0.3: dial tcp 172.18.0.3:9094: i/o timeout\n"
level=debug ts=2019-06-03T14:59:28.131527596Z caller=cluster.go:386 component=cluster msg=reconnect result=failure peer=01DCEXJP9K726FK207GYR8JJV3 addr=172.18.0.3:9094
level=debug ts=2019-06-03T14:59:28.132120479Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 14:59:28 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T14:59:28.132901292Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T14:59:30.122485653Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 14:59:30 [INFO] memberlist: Marking 01DCEXJP9K726FK207GYR8JJV3 as failed, suspect timeout reached (0 peer confirmations)\n"
level=debug ts=2019-06-03T14:59:30.122549651Z caller=delegate.go:215 component=cluster received=NotifyLeave node=01DCEXJP9K726FK207GYR8JJV3 addr=172.18.0.3:9094
level=debug ts=2019-06-03T14:59:30.122564552Z caller=cluster.go:439 component=cluster msg="peer left" peer=01DCEXJP9K726FK207GYR8JJV3
level=debug ts=2019-06-03T14:59:31.862449291Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 14:59:31 [DEBUG] memberlist: Stream connection from=10.0.2.10:35684\n"
level=debug ts=2019-06-03T14:59:31.86280666Z caller=delegate.go:209 component=cluster received=NotifyJoin node=01DCEXJP9K726FK207GYR8JJV3 addr=172.18.0.3:9094
level=debug ts=2019-06-03T14:59:31.862828794Z caller=cluster.go:417 component=cluster msg="peer rejoined" peer=01DCEXJP9K726FK207GYR8JJV3
level=debug ts=2019-06-03T14:59:38.133102021Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 14:59:38 [DEBUG] memberlist: Failed to join 172.18.0.3: dial tcp 172.18.0.3:9094: i/o timeout\n"
level=debug ts=2019-06-03T14:59:38.133153751Z caller=cluster.go:386 component=cluster msg=reconnect result=failure peer=01DCEXJP9K726FK207GYR8JJV3 addr=172.18.0.3:9094
level=debug ts=2019-06-03T14:59:38.133884249Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 14:59:38 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T14:59:38.134780648Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T14:59:40.122119992Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 14:59:40 [INFO] memberlist: Suspect 01DCEXJP9K726FK207GYR8JJV3 has failed, no acks received\n"
level=debug ts=2019-06-03T14:59:41.122405632Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 14:59:41 [DEBUG] memberlist: Failed ping: 01DCEXJP9K726FK207GYR8JJV3 (timeout reached)\n"
level=debug ts=2019-06-03T14:59:41.862078778Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 14:59:41 [DEBUG] memberlist: Stream connection from=10.0.2.10:35690\n"
level=debug ts=2019-06-03T14:59:43.486227987Z caller=dispatch.go:201 component=dispatcher msg="Received alert" alert=rabbitmq_queue_stuck[70edefc][active]
level=debug ts=2019-06-03T14:59:48.124751406Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 14:59:48 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T14:59:48.125885898Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 14:59:48 [WARN] memberlist: Refuting a suspect message (from: 01DCEXJ8W8JSCTPD03JEC56TYG)\n"
level=debug ts=2019-06-03T14:59:48.125954212Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T14:59:51.861641189Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 14:59:51 [DEBUG] memberlist: Stream connection from=10.0.2.10:35696\n"
level=debug ts=2019-06-03T14:59:56.122312639Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 14:59:56 [INFO] memberlist: Suspect 01DCEXJP9K726FK207GYR8JJV3 has failed, no acks received\n"
level=debug ts=2019-06-03T14:59:58.124657017Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 14:59:58 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T14:59:58.125441952Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T14:59:59.122247114Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 14:59:59 [DEBUG] memberlist: Failed ping: 01DCEXJP9K726FK207GYR8JJV3 (timeout reached)\n"
level=debug ts=2019-06-03T15:00:01.861179866Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:00:01 [DEBUG] memberlist: Stream connection from=10.0.2.10:35700\n"
level=debug ts=2019-06-03T15:00:04.24393138Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:00:04 [ERR] memberlist: Push/Pull with 01DCEXJP9K726FK207GYR8JJV3 failed: dial tcp 172.18.0.3:9094: i/o timeout\n"
level=debug ts=2019-06-03T15:00:08.124821465Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:00:08 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T15:00:08.125669701Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T15:00:11.860776983Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:00:11 [DEBUG] memberlist: Stream connection from=10.0.2.10:35706\n"
level=debug ts=2019-06-03T15:00:14.122111092Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:00:14 [INFO] memberlist: Suspect 01DCEXJP9K726FK207GYR8JJV3 has failed, no acks received\n"
level=debug ts=2019-06-03T15:00:17.122253552Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:00:17 [DEBUG] memberlist: Failed ping: 01DCEXJP9K726FK207GYR8JJV3 (timeout reached)\n"
level=debug ts=2019-06-03T15:00:18.124731551Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:00:18 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T15:00:18.125572496Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T15:00:21.860706196Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:00:21 [DEBUG] memberlist: Stream connection from=10.0.2.10:35710\n"
level=debug ts=2019-06-03T15:00:21.861309902Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:00:21 [WARN] memberlist: Refuting a suspect message (from: 01DCEXJ8W8JSCTPD03JEC56TYG)\n"
level=debug ts=2019-06-03T15:00:28.12483503Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:00:28 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T15:00:28.126100711Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T15:00:31.860789842Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:00:31 [DEBUG] memberlist: Stream connection from=10.0.2.10:35716\n"
level=debug ts=2019-06-03T15:00:32.1221643Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:00:32 [INFO] memberlist: Suspect 01DCEXJP9K726FK207GYR8JJV3 has failed, no acks received\n"
level=debug ts=2019-06-03T15:00:35.122221726Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:00:35 [DEBUG] memberlist: Failed ping: 01DCEXJP9K726FK207GYR8JJV3 (timeout reached)\n"
level=debug ts=2019-06-03T15:00:38.124798585Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:00:38 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T15:00:38.125720488Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T15:00:40.122352686Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:00:40 [INFO] memberlist: Marking 01DCEXJP9K726FK207GYR8JJV3 as failed, suspect timeout reached (0 peer confirmations)\n"
level=debug ts=2019-06-03T15:00:40.122444847Z caller=delegate.go:215 component=cluster received=NotifyLeave node=01DCEXJP9K726FK207GYR8JJV3 addr=172.18.0.3:9094
level=debug ts=2019-06-03T15:00:40.122516925Z caller=cluster.go:439 component=cluster msg="peer left" peer=01DCEXJP9K726FK207GYR8JJV3
level=debug ts=2019-06-03T15:00:41.860679217Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:00:41 [DEBUG] memberlist: Stream connection from=10.0.2.10:35724\n"
level=debug ts=2019-06-03T15:00:41.861152682Z caller=delegate.go:209 component=cluster received=NotifyJoin node=01DCEXJP9K726FK207GYR8JJV3 addr=172.18.0.3:9094
level=debug ts=2019-06-03T15:00:41.861171938Z caller=cluster.go:417 component=cluster msg="peer rejoined" peer=01DCEXJP9K726FK207GYR8JJV3
level=debug ts=2019-06-03T15:00:43.390735149Z caller=dispatch.go:201 component=dispatcher msg="Received alert" alert=rabbitmq_queue_stuck[70edefc][active]
level=debug ts=2019-06-03T15:00:48.124696915Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:00:48 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T15:00:48.125578271Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T15:00:50.122078987Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:00:50 [INFO] memberlist: Suspect 01DCEXJP9K726FK207GYR8JJV3 has failed, no acks received\n"
level=debug ts=2019-06-03T15:00:51.860766834Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:00:51 [DEBUG] memberlist: Stream connection from=10.0.2.10:35728\n"
level=debug ts=2019-06-03T15:00:53.122256417Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:00:53 [DEBUG] memberlist: Failed ping: 01DCEXJP9K726FK207GYR8JJV3 (timeout reached)\n"
level=debug ts=2019-06-03T15:00:58.124721864Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:00:58 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T15:00:58.125497648Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:00:58 [WARN] memberlist: Refuting a suspect message (from: 01DCEXJ8W8JSCTPD03JEC56TYG)\n"
level=debug ts=2019-06-03T15:00:58.125588066Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T15:01:01.861565526Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:01:01 [DEBUG] memberlist: Stream connection from=10.0.2.10:35734\n"
level=debug ts=2019-06-03T15:01:08.122118565Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:01:08 [INFO] memberlist: Suspect 01DCEXJP9K726FK207GYR8JJV3 has failed, no acks received\n"
level=debug ts=2019-06-03T15:01:08.124670816Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:01:08 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T15:01:08.12597942Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T15:01:11.122257087Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:01:11 [DEBUG] memberlist: Failed ping: 01DCEXJP9K726FK207GYR8JJV3 (timeout reached)\n"
level=debug ts=2019-06-03T15:01:11.860647439Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:01:11 [DEBUG] memberlist: Stream connection from=10.0.2.10:35740\n"
level=debug ts=2019-06-03T15:01:13.511601087Z caller=dispatch.go:201 component=dispatcher msg="Received alert" alert=rabbitmq_queue_stuck[70edefc][active]
level=debug ts=2019-06-03T15:01:14.244373754Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:01:14 [ERR] memberlist: Push/Pull with 01DCEXJP9K726FK207GYR8JJV3 failed: dial tcp 172.18.0.3:9094: i/o timeout\n"
level=debug ts=2019-06-03T15:01:18.12465778Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:01:18 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T15:01:18.125644953Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T15:01:21.860782463Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:01:21 [DEBUG] memberlist: Stream connection from=10.0.2.10:35746\n"
level=debug ts=2019-06-03T15:01:26.122086963Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:01:26 [INFO] memberlist: Suspect 01DCEXJP9K726FK207GYR8JJV3 has failed, no acks received\n"
level=debug ts=2019-06-03T15:01:27.122356264Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:01:27 [DEBUG] memberlist: Failed ping: 01DCEXJP9K726FK207GYR8JJV3 (timeout reached)\n"
level=debug ts=2019-06-03T15:01:28.124742553Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:01:28 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T15:01:28.125613551Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:01:28 [WARN] memberlist: Refuting a suspect message (from: 01DCEXJ8W8JSCTPD03JEC56TYG)\n"
level=debug ts=2019-06-03T15:01:28.125703067Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T15:01:31.860907301Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:01:31 [DEBUG] memberlist: Stream connection from=10.0.2.10:35754\n"
level=debug ts=2019-06-03T15:01:38.124766535Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:01:38 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T15:01:38.125513251Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T15:01:41.860879169Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:01:41 [DEBUG] memberlist: Stream connection from=10.0.2.10:35760\n"
level=debug ts=2019-06-03T15:01:42.12224691Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:01:42 [INFO] memberlist: Suspect 01DCEXJP9K726FK207GYR8JJV3 has failed, no acks received\n"
level=debug ts=2019-06-03T15:01:45.122225231Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:01:45 [DEBUG] memberlist: Failed ping: 01DCEXJP9K726FK207GYR8JJV3 (timeout reached)\n"
level=debug ts=2019-06-03T15:01:48.124685452Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:01:48 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T15:01:48.125458652Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T15:01:50.122436028Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:01:50 [INFO] memberlist: Marking 01DCEXJP9K726FK207GYR8JJV3 as failed, suspect timeout reached (0 peer confirmations)\n"
level=debug ts=2019-06-03T15:01:50.122492643Z caller=delegate.go:215 component=cluster received=NotifyLeave node=01DCEXJP9K726FK207GYR8JJV3 addr=172.18.0.3:9094
level=debug ts=2019-06-03T15:01:50.12250477Z caller=cluster.go:439 component=cluster msg="peer left" peer=01DCEXJP9K726FK207GYR8JJV3
level=debug ts=2019-06-03T15:01:51.861077022Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:01:51 [DEBUG] memberlist: Stream connection from=10.0.2.10:35766\n"
level=debug ts=2019-06-03T15:01:51.861567069Z caller=delegate.go:209 component=cluster received=NotifyJoin node=01DCEXJP9K726FK207GYR8JJV3 addr=172.18.0.3:9094
level=debug ts=2019-06-03T15:01:51.861591347Z caller=cluster.go:417 component=cluster msg="peer rejoined" peer=01DCEXJP9K726FK207GYR8JJV3
level=debug ts=2019-06-03T15:01:58.124799899Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:01:58 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T15:01:58.125713451Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T15:02:00.122130196Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:02:00 [INFO] memberlist: Suspect 01DCEXJP9K726FK207GYR8JJV3 has failed, no acks received\n"
level=debug ts=2019-06-03T15:02:01.861126602Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:02:01 [DEBUG] memberlist: Stream connection from=10.0.2.10:35770\n"
level=debug ts=2019-06-03T15:02:01.861529848Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:02:01 [WARN] memberlist: Refuting a suspect message (from: 01DCEXJ8W8JSCTPD03JEC56TYG)\n"
level=debug ts=2019-06-03T15:02:03.122257986Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:02:03 [DEBUG] memberlist: Failed ping: 01DCEXJP9K726FK207GYR8JJV3 (timeout reached)\n"
level=debug ts=2019-06-03T15:02:08.124773265Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:02:08 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T15:02:08.125672863Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T15:02:11.861364167Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:02:11 [DEBUG] memberlist: Stream connection from=10.0.2.10:35778\n"
level=debug ts=2019-06-03T15:02:13.378188472Z caller=dispatch.go:201 component=dispatcher msg="Received alert" alert=rabbitmq_queue_stuck[70edefc][active]
level=debug ts=2019-06-03T15:02:18.122121429Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:02:18 [INFO] memberlist: Suspect 01DCEXJP9K726FK207GYR8JJV3 has failed, no acks received\n"
level=debug ts=2019-06-03T15:02:18.124890175Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:02:18 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T15:02:18.125609631Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T15:02:21.122221641Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:02:21 [DEBUG] memberlist: Failed ping: 01DCEXJP9K726FK207GYR8JJV3 (timeout reached)\n"
level=debug ts=2019-06-03T15:02:21.861251244Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:02:21 [DEBUG] memberlist: Stream connection from=10.0.2.10:35782\n"
level=debug ts=2019-06-03T15:02:24.244773629Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:02:24 [ERR] memberlist: Push/Pull with 01DCEXJP9K726FK207GYR8JJV3 failed: dial tcp 172.18.0.3:9094: i/o timeout\n"
level=debug ts=2019-06-03T15:02:28.124663994Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:02:28 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T15:02:28.125426104Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T15:02:31.861466641Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:02:31 [DEBUG] memberlist: Stream connection from=10.0.2.10:35790\n"
level=debug ts=2019-06-03T15:02:36.122122226Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:02:36 [INFO] memberlist: Suspect 01DCEXJP9K726FK207GYR8JJV3 has failed, no acks received\n"
level=debug ts=2019-06-03T15:02:38.125049902Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:02:38 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T15:02:38.125994684Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:02:38 [WARN] memberlist: Refuting a suspect message (from: 01DCEXJ8W8JSCTPD03JEC56TYG)\n"
level=debug ts=2019-06-03T15:02:38.126060272Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T15:02:39.122190999Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:02:39 [DEBUG] memberlist: Failed ping: 01DCEXJP9K726FK207GYR8JJV3 (timeout reached)\n"
level=debug ts=2019-06-03T15:02:41.861595074Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:02:41 [DEBUG] memberlist: Stream connection from=10.0.2.10:35796\n"
level=debug ts=2019-06-03T15:02:43.51054279Z caller=dispatch.go:201 component=dispatcher msg="Received alert" alert=rabbitmq_queue_stuck[70edefc][active]
level=debug ts=2019-06-03T15:02:48.124825103Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:02:48 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T15:02:48.125579441Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T15:02:51.861618103Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:02:51 [DEBUG] memberlist: Stream connection from=10.0.2.10:35800\n"
level=debug ts=2019-06-03T15:02:54.122087567Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:02:54 [INFO] memberlist: Suspect 01DCEXJP9K726FK207GYR8JJV3 has failed, no acks received\n"
level=debug ts=2019-06-03T15:02:57.122196876Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:02:57 [DEBUG] memberlist: Failed ping: 01DCEXJP9K726FK207GYR8JJV3 (timeout reached)\n"
level=debug ts=2019-06-03T15:02:58.124748013Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:02:58 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T15:02:58.125811884Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T15:03:01.861926031Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:03:01 [DEBUG] memberlist: Stream connection from=10.0.2.10:35806\n"
level=debug ts=2019-06-03T15:03:08.124841776Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:03:08 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T15:03:08.125717364Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T15:03:11.862192138Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:03:11 [DEBUG] memberlist: Stream connection from=10.0.2.10:35812\n"
level=debug ts=2019-06-03T15:03:11.862704661Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:03:11 [WARN] memberlist: Refuting a suspect message (from: 01DCEXJ8W8JSCTPD03JEC56TYG)\n"
level=debug ts=2019-06-03T15:03:12.122076917Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:03:12 [INFO] memberlist: Suspect 01DCEXJP9K726FK207GYR8JJV3 has failed, no acks received\n"
level=debug ts=2019-06-03T15:03:13.122302828Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:03:13 [DEBUG] memberlist: Failed ping: 01DCEXJP9K726FK207GYR8JJV3 (timeout reached)\n"
level=debug ts=2019-06-03T15:03:18.12472121Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:03:18 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T15:03:18.125567831Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T15:03:21.862100527Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:03:21 [DEBUG] memberlist: Stream connection from=10.0.2.10:35818\n"
level=debug ts=2019-06-03T15:03:28.122238144Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:03:28 [INFO] memberlist: Suspect 01DCEXJP9K726FK207GYR8JJV3 has failed, no acks received\n"
level=debug ts=2019-06-03T15:03:28.124685619Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:03:28 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T15:03:28.125543313Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T15:03:31.122217615Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:03:31 [DEBUG] memberlist: Failed ping: 01DCEXJP9K726FK207GYR8JJV3 (timeout reached)\n"
level=debug ts=2019-06-03T15:03:31.862075361Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:03:31 [DEBUG] memberlist: Stream connection from=10.0.2.10:35824\n"
level=debug ts=2019-06-03T15:03:34.24518727Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:03:34 [ERR] memberlist: Push/Pull with 01DCEXJP9K726FK207GYR8JJV3 failed: dial tcp 172.18.0.3:9094: i/o timeout\n"
level=debug ts=2019-06-03T15:03:38.124776231Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:03:38 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T15:03:38.12593422Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T15:03:41.863568405Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:03:41 [DEBUG] memberlist: Stream connection from=10.0.2.10:35830\n"
level=debug ts=2019-06-03T15:03:43.370854837Z caller=dispatch.go:201 component=dispatcher msg="Received alert" alert=rabbitmq_queue_stuck[70edefc][active]
level=debug ts=2019-06-03T15:03:46.12213577Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:03:46 [INFO] memberlist: Suspect 01DCEXJP9K726FK207GYR8JJV3 has failed, no acks received\n"
level=debug ts=2019-06-03T15:03:48.124683377Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:03:48 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T15:03:48.125413805Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:03:48 [WARN] memberlist: Refuting a suspect message (from: 01DCEXJ8W8JSCTPD03JEC56TYG)\n"
level=debug ts=2019-06-03T15:03:48.125514581Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T15:03:49.122250703Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:03:49 [DEBUG] memberlist: Failed ping: 01DCEXJP9K726FK207GYR8JJV3 (timeout reached)\n"
level=debug ts=2019-06-03T15:03:51.865757883Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:03:51 [DEBUG] memberlist: Stream connection from=10.0.2.10:35836\n"
level=debug ts=2019-06-03T15:03:58.124706582Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:03:58 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T15:03:58.125706608Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T15:04:01.867560016Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:04:01 [DEBUG] memberlist: Stream connection from=10.0.2.10:35840\n"
level=debug ts=2019-06-03T15:04:04.122129442Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:04:04 [INFO] memberlist: Suspect 01DCEXJP9K726FK207GYR8JJV3 has failed, no acks received\n"
level=debug ts=2019-06-03T15:04:05.122446322Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:04:05 [DEBUG] memberlist: Failed ping: 01DCEXJP9K726FK207GYR8JJV3 (timeout reached)\n"
level=debug ts=2019-06-03T15:04:08.124749344Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:04:08 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T15:04:08.125622275Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T15:04:11.869437474Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:04:11 [DEBUG] memberlist: Stream connection from=10.0.2.10:35846\n"
level=debug ts=2019-06-03T15:04:13.376393067Z caller=dispatch.go:445 component=dispatcher aggrGroup={}:{} msg=Flushing alerts=[rabbitmq_queue_stuck[70edefc][active]]
level=debug ts=2019-06-03T15:04:13.505707895Z caller=dispatch.go:201 component=dispatcher msg="Received alert" alert=rabbitmq_queue_stuck[70edefc][active]
level=debug ts=2019-06-03T15:04:18.124709134Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:04:18 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T15:04:18.125469597Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T15:04:20.122341654Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:04:20 [INFO] memberlist: Suspect 01DCEXJP9K726FK207GYR8JJV3 has failed, no acks received\n"
level=debug ts=2019-06-03T15:04:21.870782579Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:04:21 [DEBUG] memberlist: Stream connection from=10.0.2.10:35850\n"
level=debug ts=2019-06-03T15:04:23.122250551Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:04:23 [DEBUG] memberlist: Failed ping: 01DCEXJP9K726FK207GYR8JJV3 (timeout reached)\n"
level=debug ts=2019-06-03T15:04:28.124770554Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:04:28 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T15:04:28.125580517Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:04:28 [WARN] memberlist: Refuting a suspect message (from: 01DCEXJ8W8JSCTPD03JEC56TYG)\n"
level=debug ts=2019-06-03T15:04:28.125669857Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T15:04:31.871901633Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:04:31 [DEBUG] memberlist: Stream connection from=10.0.2.10:35856\n"
level=debug ts=2019-06-03T15:04:38.122148238Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:04:38 [INFO] memberlist: Suspect 01DCEXJP9K726FK207GYR8JJV3 has failed, no acks received\n"
level=debug ts=2019-06-03T15:04:38.124659568Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:04:38 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T15:04:38.125463767Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T15:04:41.122279824Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:04:41 [DEBUG] memberlist: Failed ping: 01DCEXJP9K726FK207GYR8JJV3 (timeout reached)\n"
level=debug ts=2019-06-03T15:04:41.872869858Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:04:41 [DEBUG] memberlist: Stream connection from=10.0.2.10:35862\n"
level=debug ts=2019-06-03T15:04:44.245551422Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:04:44 [ERR] memberlist: Push/Pull with 01DCEXJP9K726FK207GYR8JJV3 failed: dial tcp 172.18.0.3:9094: i/o timeout\n"
level=debug ts=2019-06-03T15:04:48.124785706Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:04:48 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T15:04:48.125719019Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T15:04:51.873271906Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:04:51 [DEBUG] memberlist: Stream connection from=10.0.2.10:35868\n"
level=debug ts=2019-06-03T15:04:56.122129279Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:04:56 [INFO] memberlist: Suspect 01DCEXJP9K726FK207GYR8JJV3 has failed, no acks received\n"
level=debug ts=2019-06-03T15:04:58.124740666Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:04:58 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T15:04:58.125475505Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:04:58 [WARN] memberlist: Refuting a suspect message (from: 01DCEXJ8W8JSCTPD03JEC56TYG)\n"
level=debug ts=2019-06-03T15:04:58.12553574Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T15:04:59.122202234Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:04:59 [DEBUG] memberlist: Failed ping: 01DCEXJP9K726FK207GYR8JJV3 (timeout reached)\n"
level=debug ts=2019-06-03T15:05:01.87357737Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:05:01 [DEBUG] memberlist: Stream connection from=10.0.2.10:35874\n"
level=debug ts=2019-06-03T15:05:04.2591756Z caller=dispatch.go:201 component=dispatcher msg="Received alert" alert=high_cpu_load[a035d08][active]
level=debug ts=2019-06-03T15:05:04.365215829Z caller=dispatch.go:201 component=dispatcher msg="Received alert" alert=high_cpu_load[a035d08][active]
level=debug ts=2019-06-03T15:05:08.124684308Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:05:08 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T15:05:08.125488184Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T15:05:11.873386489Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:05:11 [DEBUG] memberlist: Stream connection from=10.0.2.10:35876\n"
level=debug ts=2019-06-03T15:05:13.38380925Z caller=dispatch.go:201 component=dispatcher msg="Received alert" alert=rabbitmq_queue_stuck[70edefc][active]
level=debug ts=2019-06-03T15:05:14.122087215Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:05:14 [INFO] memberlist: Suspect 01DCEXJP9K726FK207GYR8JJV3 has failed, no acks received\n"
level=debug ts=2019-06-03T15:05:15.122379127Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:05:15 [DEBUG] memberlist: Failed ping: 01DCEXJP9K726FK207GYR8JJV3 (timeout reached)\n"
level=debug ts=2019-06-03T15:05:18.124720924Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:05:18 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T15:05:18.125671212Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T15:05:21.873082125Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:05:21 [DEBUG] memberlist: Stream connection from=10.0.2.10:35882\n"
level=debug ts=2019-06-03T15:05:28.124704446Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:05:28 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T15:05:28.125580984Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T15:05:30.122285725Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:05:30 [INFO] memberlist: Suspect 01DCEXJP9K726FK207GYR8JJV3 has failed, no acks received\n"
level=debug ts=2019-06-03T15:05:31.872781034Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:05:31 [DEBUG] memberlist: Stream connection from=10.0.2.10:35888\n"
level=debug ts=2019-06-03T15:05:31.873125236Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:05:31 [WARN] memberlist: Refuting a suspect message (from: 01DCEXJ8W8JSCTPD03JEC56TYG)\n"
level=debug ts=2019-06-03T15:05:33.122239141Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:05:33 [DEBUG] memberlist: Failed ping: 01DCEXJP9K726FK207GYR8JJV3 (timeout reached)\n"
level=debug ts=2019-06-03T15:05:38.124739798Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:05:38 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T15:05:38.125655509Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T15:05:41.872747939Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:05:41 [DEBUG] memberlist: Stream connection from=10.0.2.10:35894\n"
level=debug ts=2019-06-03T15:05:43.491998135Z caller=dispatch.go:201 component=dispatcher msg="Received alert" alert=rabbitmq_queue_stuck[70edefc][active]
level=debug ts=2019-06-03T15:05:48.12213023Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:05:48 [INFO] memberlist: Suspect 01DCEXJP9K726FK207GYR8JJV3 has failed, no acks received\n"
level=debug ts=2019-06-03T15:05:48.12469577Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:05:48 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T15:05:48.125875145Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T15:05:51.122265759Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:05:51 [DEBUG] memberlist: Failed ping: 01DCEXJP9K726FK207GYR8JJV3 (timeout reached)\n"
level=debug ts=2019-06-03T15:05:51.871945785Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:05:51 [DEBUG] memberlist: Stream connection from=10.0.2.10:35900\n"
level=debug ts=2019-06-03T15:05:54.246026342Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:05:54 [ERR] memberlist: Push/Pull with 01DCEXJP9K726FK207GYR8JJV3 failed: dial tcp 172.18.0.3:9094: i/o timeout\n"
level=debug ts=2019-06-03T15:05:58.124690089Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:05:58 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T15:05:58.125487943Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T15:06:01.870189411Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:06:01 [DEBUG] memberlist: Stream connection from=10.0.2.10:35904\n"
level=debug ts=2019-06-03T15:06:04.291708029Z caller=dispatch.go:201 component=dispatcher msg="Received alert" alert=high_cpu_load[a035d08][resolved]
level=debug ts=2019-06-03T15:06:04.331728114Z caller=dispatch.go:201 component=dispatcher msg="Received alert" alert=high_cpu_load[a035d08][resolved]
level=debug ts=2019-06-03T15:06:06.122118766Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:06:06 [INFO] memberlist: Suspect 01DCEXJP9K726FK207GYR8JJV3 has failed, no acks received\n"
level=debug ts=2019-06-03T15:06:07.122429252Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:06:07 [DEBUG] memberlist: Failed ping: 01DCEXJP9K726FK207GYR8JJV3 (timeout reached)\n"
level=debug ts=2019-06-03T15:06:08.124704548Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:06:08 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T15:06:08.125527863Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:06:08 [WARN] memberlist: Refuting a suspect message (from: 01DCEXJ8W8JSCTPD03JEC56TYG)\n"
level=debug ts=2019-06-03T15:06:08.125614103Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T15:06:11.868362582Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:06:11 [DEBUG] memberlist: Stream connection from=10.0.2.10:35912\n"
level=debug ts=2019-06-03T15:06:18.124647747Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:06:18 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T15:06:18.12584083Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T15:06:21.866564371Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:06:21 [DEBUG] memberlist: Stream connection from=10.0.2.10:35916\n"
level=debug ts=2019-06-03T15:06:22.122293742Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:06:22 [INFO] memberlist: Suspect 01DCEXJP9K726FK207GYR8JJV3 has failed, no acks received\n"
level=debug ts=2019-06-03T15:06:25.122218471Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:06:25 [DEBUG] memberlist: Failed ping: 01DCEXJP9K726FK207GYR8JJV3 (timeout reached)\n"
level=debug ts=2019-06-03T15:06:28.124800531Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:06:28 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T15:06:28.125593142Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T15:06:30.122475686Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:06:30 [INFO] memberlist: Marking 01DCEXJP9K726FK207GYR8JJV3 as failed, suspect timeout reached (0 peer confirmations)\n"
level=debug ts=2019-06-03T15:06:30.12254172Z caller=delegate.go:215 component=cluster received=NotifyLeave node=01DCEXJP9K726FK207GYR8JJV3 addr=172.18.0.3:9094
level=debug ts=2019-06-03T15:06:30.122555472Z caller=cluster.go:439 component=cluster msg="peer left" peer=01DCEXJP9K726FK207GYR8JJV3
level=debug ts=2019-06-03T15:06:31.865169888Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:06:31 [DEBUG] memberlist: Stream connection from=10.0.2.10:35930\n"
level=debug ts=2019-06-03T15:06:31.865838314Z caller=delegate.go:209 component=cluster received=NotifyJoin node=01DCEXJP9K726FK207GYR8JJV3 addr=172.18.0.3:9094
level=debug ts=2019-06-03T15:06:31.86587165Z caller=cluster.go:417 component=cluster msg="peer rejoined" peer=01DCEXJP9K726FK207GYR8JJV3
level=debug ts=2019-06-03T15:06:38.124773292Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:06:38 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T15:06:38.125655118Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T15:06:40.122095371Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:06:40 [INFO] memberlist: Suspect 01DCEXJP9K726FK207GYR8JJV3 has failed, no acks received\n"
level=debug ts=2019-06-03T15:06:41.865514752Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:06:41 [DEBUG] memberlist: Stream connection from=10.0.2.10:35934\n"
level=debug ts=2019-06-03T15:06:41.866106571Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:06:41 [WARN] memberlist: Refuting a suspect message (from: 01DCEXJ8W8JSCTPD03JEC56TYG)\n"
level=debug ts=2019-06-03T15:06:43.122319135Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:06:43 [DEBUG] memberlist: Failed ping: 01DCEXJP9K726FK207GYR8JJV3 (timeout reached)\n"
level=debug ts=2019-06-03T15:06:43.372998556Z caller=dispatch.go:201 component=dispatcher msg="Received alert" alert=rabbitmq_queue_stuck[70edefc][active]
level=debug ts=2019-06-03T15:06:48.124742727Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:06:48 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T15:06:48.125463234Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T15:06:51.865793811Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:06:51 [DEBUG] memberlist: Stream connection from=10.0.2.10:35952\n"
level=debug ts=2019-06-03T15:06:58.122169074Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:06:58 [INFO] memberlist: Suspect 01DCEXJP9K726FK207GYR8JJV3 has failed, no acks received\n"
level=debug ts=2019-06-03T15:06:58.124778019Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:06:58 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T15:06:58.125648628Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T15:07:01.122247496Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:07:01 [DEBUG] memberlist: Failed ping: 01DCEXJP9K726FK207GYR8JJV3 (timeout reached)\n"
level=debug ts=2019-06-03T15:07:01.865510515Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:07:01 [DEBUG] memberlist: Stream connection from=10.0.2.10:35958\n"
level=debug ts=2019-06-03T15:07:04.246433104Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:07:04 [ERR] memberlist: Push/Pull with 01DCEXJP9K726FK207GYR8JJV3 failed: dial tcp 172.18.0.3:9094: i/o timeout\n"
level=debug ts=2019-06-03T15:07:04.250650323Z caller=dispatch.go:201 component=dispatcher msg="Received alert" alert=high_cpu_load[a035d08][active]
level=debug ts=2019-06-03T15:07:04.291193211Z caller=dispatch.go:201 component=dispatcher msg="Received alert" alert=high_cpu_load[a035d08][active]
level=debug ts=2019-06-03T15:07:08.124697953Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:07:08 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T15:07:08.125525754Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T15:07:11.865044872Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:07:11 [DEBUG] memberlist: Stream connection from=10.0.2.10:35960\n"
level=debug ts=2019-06-03T15:07:13.484792747Z caller=dispatch.go:201 component=dispatcher msg="Received alert" alert=rabbitmq_queue_stuck[70edefc][active]
level=debug ts=2019-06-03T15:07:16.122126367Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:07:16 [INFO] memberlist: Suspect 01DCEXJP9K726FK207GYR8JJV3 has failed, no acks received\n"
level=debug ts=2019-06-03T15:07:18.124674487Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:07:18 [DEBUG] memberlist: Initiating push/pull sync with: 10.0.2.75:9094\n"
level=debug ts=2019-06-03T15:07:18.125616454Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:07:18 [WARN] memberlist: Refuting a suspect message (from: 01DCEXJ8W8JSCTPD03JEC56TYG)\n"
level=debug ts=2019-06-03T15:07:18.125683052Z caller=cluster.go:389 component=cluster msg=reconnect result=success peer= addr=10.0.2.75:9094
level=debug ts=2019-06-03T15:07:19.122387022Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:07:19 [DEBUG] memberlist: Failed ping: 01DCEXJP9K726FK207GYR8JJV3 (timeout reached)\n"
level=debug ts=2019-06-03T15:07:21.864748187Z caller=cluster.go:287 component=cluster memberlist="2019/06/03 15:07:21 [DEBUG] memberlist: Stream connection from=10.0.2.10:35966\n"

[simonpasquier]

There's definitely a connectivity issue and you seem to have 2 subnets (172.18.0.X and 10.0.2.X) configured. Can you provide more details about your setup?

[MattPOlson]

We are running in a Docker Swarm, each instance of Alert Manager is it's own service with 1 replica and each service is running on a different node. From my understanding the 172 subnet is the internal Docker network.

[simonpasquier]

I'm not familiar enough with Docker Swarm to really help. What seems to happen: AlertManager A discovers its peer (AlertManager B) via the service name (10.x.x.x subnet) and establishes the connection but it advertises its internal address (172.x.x.x subnet) which triggers the connectivity issue.
If you manage to set --cluster.advertise-address with the 10.x.x.x address, it would probably work but I'm not sure how you can do this.

[MattPOlson]

I'm wondering if there is a issue with DNS resolver in the alert manager docker image. Both services are deployed to the same network and I'm unable to perform a nslookup on tasks.alertmanager_2 from the alertmanager_1 container.

/alertmanager # nslookup tasks.alertmanager_1
Server: 127.0.0.11
Address: 127.0.0.11:53

Non-authoritative answer:

*** Can't find tasks.alertmanager_1: No answer

I deployed netshoot, attached it to the same network and nslookup work fine.

[1] 🐳 → nslookup tasks.alertmanager_1
Server: 127.0.0.11
Address: 127.0.0.11#53

Non-authoritative answer:
Name: tasks.alertmanager_1
Address: 10.0.1.71

[simonpasquier]

AlertManager uses the default Go's name resolver. You can try to set the GODEBUG variable to netdns=1 which would print debugging information that could help find the issue.

[MattPOlson]

After looking into this some more, we were unable to get this to work in ha mode on Docker Swarm. Has anyone been able to get it working in Docker Swarm, it this configuration event supported?

[0x63lv]

Could be related to #1550

[MattPOlson]

Could be related to #1550

It is, the problem is in advertise.go with this function sockaddr.GetPrivateIP, it returns the first public or private IP address on the default interface which as far I can tell in docker swarm is the 172* address which does not work with AlertMamager clustering. Using this function and passing in what interface to get the IP from is a better solution. sockaddr.GetInterfaceIP("eth0"). I made this changed in a forked branch and it has stopped flapping. The members still reinitialize all the time but we are not getting duplicate alerts anymore.

[jmb12686]

Same issue here as well, running 2 alertmanagers containers in Docker Swarm. I'm attempting
to set them up in HA, however advertise-address is unknown until the container starts up.

The containers attempt to reach the (incorrectly) advertised 172.x.x.x address vs the eth1 interface address @ 10.x.x.x in docker swarm.

[MattPOlson]

Same issue here as well, running 2 alertmanagers containers in Docker Swarm. I'm attempting
to set them up in HA, however advertise-address is unknown until the container starts up.

The containers attempt to reach the (incorrectly) advertised 172.x.x.x address vs the eth1 interface address @ 10.x.x.x in docker swarm.

Take a look at the dev branch in my forked repo
https://github.com/MattPOlson/alertmanager

I made a change that allows you to specify the interface name via a command argument like this:

• '--cluster.advertise-address-interface=eth0'

See if that works for you, I plan on making a PR into the main repo just doing some more testing still.

[take31415]

@MattPOlson I'm also interested in having this in the main repo. Any progress regarding it?

Found a possible solution:
Running an entrypoint script before alertmanager gets started. This is extracting the correct IP and setting it in "--cluster.listen-address=" parameter.
Contra: Had to build a new customized image of alertmanager.

[ScrumpyJack]

Same issue here as well, running 2 alertmanagers containers in Docker Swarm. I'm attempting
to set them up in HA, however advertise-address is unknown until the container starts up.
The containers attempt to reach the (incorrectly) advertised 172.x.x.x address vs the eth1 interface address @ 10.x.x.x in docker swarm.

Take a look at the dev branch in my forked repo https://github.com/MattPOlson/alertmanager

I made a change that allows you to specify the interface name via a command argument like this:

* '--cluster.advertise-address-interface=eth0'

See if that works for you, I plan on making a PR into the main repo just doing some more testing still.

This would help us a lot too when running an HA Alertmanager cluster in a swarn
Can we get your fix into a Pull Request?

[ScrumpyJack]

Just like to OP, here in this snippet you can see the cluster component attempting to connect to the peer on two interfaces, 10.0.19.9 succeeds, but messages get refuted as it also fails to join on the docker swarm interface 172.18.0.5

component=cluster memberlist="2021/11/25 18:43:53 [DEBUG] memberlist: Failed ping: 01FNC3Z6NNJ8YBTZT1GW86PHQ5 (timeout reached)\n"
component=cluster memberlist="2021/11/25 18:44:00 [INFO] memberlist: Suspect 01FNC3Z6NNJ8YBTZT1GW86PHQ5 has failed, no acks received\n"
component=cluster memberlist="2021/11/25 18:44:02 [DEBUG] memberlist: Failed ping: 01FNC3Z6NNJ8YBTZT1GW86PHQ5 (timeout reached)\n"
component=cluster memberlist="2021/11/25 18:44:07 [DEBUG] memberlist: Initiating push/pull sync with:  10.0.19.7:9094\n"
component=cluster msg=refresh result=success addr=10.0.19.7:9094
component=cluster memberlist="2021/11/25 18:44:08 [DEBUG] memberlist: Stream connection from=10.0.19.9:57706\n"
component=cluster memberlist="2021/11/25 18:44:09 [INFO] memberlist: Suspect 01FNC3Z6NNJ8YBTZT1GW86PHQ5 has failed, no acks received\n"
component=cluster memberlist="2021/11/25 18:44:11 [DEBUG] memberlist: Failed ping: 01FNC3Z6NNJ8YBTZT1GW86PHQ5 (timeout reached)\n"
component=cluster memberlist="2021/11/25 18:44:13 [INFO] memberlist: Marking 01FNC3Z6NNJ8YBTZT1GW86PHQ5 as failed, suspect timeout reached (0 peer confirmations)\n"
component=cluster received=NotifyLeave node=01FNC3Z6NNJ8YBTZT1GW86PHQ5 addr=172.18.0.5:9094
component=cluster msg="peer left" peer=01FNC3Z6NNJ8YBTZT1GW86PHQ5
component=cluster memberlist="2021/11/25 18:44:18 [INFO] memberlist: Suspect 01FNC3Z6NNJ8YBTZT1GW86PHQ5 has failed, no acks received\n"
component=cluster memberlist="2021/11/25 18:44:22 [DEBUG] memberlist: Failed to join 172.18.0.5: dial tcp 172.18.0.5:9094: connect: connection refused\n"
component=cluster msg=reconnect result=failure peer=01FNC3Z6NNJ8YBTZT1GW86PHQ5 addr=172.18.0.5:9094 err="1 error occurred:\n\t* Failed to join 172.18.0.5: dial tcp 172.18.0.5:9094: connect: connection refused\n\n"
component=cluster memberlist="2021/11/25 18:44:22 [DEBUG] memberlist: Initiating push/pull sync with:  10.0.19.7:9094\n"
component=cluster memberlist="2021/11/25 18:44:22 [WARN] memberlist: Refuting a suspect message (from: 01FNC3Z698P4CC5MXV58DZ0BS8)\n"
component=cluster msg=refresh result=success addr=10.0.19.7:9094
component=cluster memberlist="2021/11/25 18:44:23 [DEBUG] memberlist: Stream connection from=10.0.19.9:57832\n"
component=cluster received=NotifyJoin node=01FNC3Z6NNJ8YBTZT1GW86PHQ5 addr=172.18.0.5:9094
component=cluster msg="peer rejoined" peer=01FNC3Z6NNJ8YBTZT1GW86PHQ5
component=cluster memberlist="2021/11/25 18:44:24 [DEBUG] memberlist: Failed ping: 01FNC3Z6NNJ8YBTZT1GW86PHQ5 (timeout reached)\n"
component=cluster memberlist="2021/11/25 18:44:30 [ERR] memberlist: Push/Pull with 01FNC3Z6NNJ8YBTZT1GW86PHQ5 failed: dial tcp 172.18.0.5:9094: connect: connection refused\n"
component=cluster memberlist="2021/11/25 18:44:31 [INFO] memberlist: Suspect 01FNC3Z6NNJ8YBTZT1GW86PHQ5 has failed, no acks received\n"
component=cluster memberlist="2021/11/25 18:44:33 [DEBUG] memberlist: Failed ping: 01FNC3Z6NNJ8YBTZT1GW86PHQ5 (timeout reached)\n"
component=cluster memberlist="2021/11/25 18:44:35 [INFO] memberlist: Marking 01FNC3Z6NNJ8YBTZT1GW86PHQ5 as failed, suspect timeout reached (0 peer confirmations)\n"
component=cluster received=NotifyLeave node=01FNC3Z6NNJ8YBTZT1GW86PHQ5 addr=172.18.0.5:9094
component=cluster msg="peer left" peer=01FNC3Z6NNJ8YBTZT1GW86PHQ5
component=cluster memberlist="2021/11/25 18:44:37 [DEBUG] memberlist: Initiating push/pull sync with:  10.0.19.7:9094\n"
component=cluster memberlist="2021/11/25 18:44:37 [WARN] memberlist: Refuting a suspect message (from: 01FNC3Z698P4CC5MXV58DZ0BS8)\n"
component=cluster msg=refresh result=success addr=10.0.19.7:9094
component=cluster memberlist="2021/11/25 18:44:38 [DEBUG] memberlist: Stream connection from=10.0.19.9:57902\n"
component=cluster received=NotifyJoin node=01FNC3Z6NNJ8YBTZT1GW86PHQ5 addr=172.18.0.5:9094
component=cluster msg="peer rejoined" peer=01FNC3Z6NNJ8YBTZT1GW86PHQ5
component=cluster memberlist="2021/11/25 18:44:40 [INFO] memberlist: Suspect 01FNC3Z6NNJ8YBTZT1GW86PHQ5 has failed, no acks received\n"
component=cluster memberlist="2021/11/25 18:44:41 [DEBUG] memberlist: Failed ping: 01FNC3Z6NNJ8YBTZT1GW86PHQ5 (timeout reached)\n"
component=cluster memberlist="2021/11/25 18:44:48 [INFO] memberlist: Suspect 01FNC3Z6NNJ8YBTZT1GW86PHQ5 has failed, no acks received\n"
component=cluster memberlist="2021/11/25 18:44:50 [DEBUG] memberlist: Failed ping: 01FNC3Z6NNJ8YBTZT1GW86PHQ5 (timeout reached)\n"
component=cluster memberlist="2021/11/25 18:44:52 [INFO] memberlist: Marking 01FNC3Z6NNJ8YBTZT1GW86PHQ5 as failed, suspect timeout reached (0 peer confirmations)\n"
component=cluster received=NotifyLeave node=01FNC3Z6NNJ8YBTZT1GW86PHQ5 addr=172.18.0.5:9094
component=cluster msg="peer left" peer=01FNC3Z6NNJ8YBTZT1GW86PHQ5
component=cluster memberlist="2021/11/25 18:44:52 [DEBUG] memberlist: Failed to join 172.18.0.5: dial tcp 172.18.0.5:9094: connect: connection refused\n"
component=cluster msg=reconnect result=failure peer=01FNC3Z6NNJ8YBTZT1GW86PHQ5 addr=172.18.0.5:9094 err="1 error occurred:\n\t* Failed to join 172.18.0.5: dial tcp 172.18.0.5:9094: connect: connection refused\n\n"
component=cluster memberlist="2021/11/25 18:44:52 [DEBUG] memberlist: Initiating push/pull sync with:  10.0.19.7:9094\n"
component=cluster memberlist="2021/11/25 18:44:52 [WARN] memberlist: Refuting a suspect message (from: 01FNC3Z698P4CC5MXV58DZ0BS8)\n"
component=cluster msg=refresh result=success addr=10.0.19.7:9094
component=cluster memberlist="2021/11/25 18:44:53 [DEBUG] memberlist: Stream connection from=10.0.19.9:57960\n"
component=cluster received=NotifyJoin node=01FNC3Z6NNJ8YBTZT1GW86PHQ5 addr=172.18.0.5:9094
component=cluster msg="peer rejoined" peer=01FNC3Z6NNJ8YBTZT1GW86PHQ5
component=cluster memberlist="2021/11/25 18:44:57 [INFO] memberlist: Suspect 01FNC3Z6NNJ8YBTZT1GW86PHQ5 has failed, no acks received\n"
component=cluster memberlist="2021/11/25 18:44:59 [DEBUG] memberlist: Failed ping: 01FNC3Z6NNJ8YBTZT1GW86PHQ5 (timeout reached)\n"
component=cluster memberlist="2021/11/25 18:45:06 [INFO] memberlist: Suspect 01FNC3Z6NNJ8YBTZT1GW86PHQ5 has failed, no acks received\n"
component=cluster memberlist="2021/11/25 18:45:07 [DEBUG] memberlist: Failed ping: 01FNC3Z6NNJ8YBTZT1GW86PHQ5 (timeout reached)\n"
component=cluster memberlist="2021/11/25 18:45:07 [DEBUG] memberlist: Initiating push/pull sync with:  10.0.19.7:9094\n"
component=cluster msg=refresh result=success addr=10.0.19.7:9094
component=cluster memberlist="2021/11/25 18:45:08 [DEBUG] memberlist: Stream connection from=10.0.19.9:58014\n"
component=cluster memberlist="2021/11/25 18:45:08 [WARN] memberlist: Refuting a suspect message (from: 01FNC3Z698P4CC5MXV58DZ0BS8)\n"
component=cluster memberlist="2021/11/25 18:45:14 [INFO] memberlist: Suspect 01FNC3Z6NNJ8YBTZT1GW86PHQ5 has failed, no acks received\n"
component=cluster memberlist="2021/11/25 18:45:16 [DEBUG] memberlist: Failed ping: 01FNC3Z6NNJ8YBTZT1GW86PHQ5 (timeout reached)\n"
component=cluster memberlist="2021/11/25 18:45:22 [DEBUG] memberlist: Initiating push/pull sync with:  10.0.19.7:9094\n"
component=cluster msg=refresh result=success addr=10.0.19.7:9094
component=cluster memberlist="2021/11/25 18:45:23 [DEBUG] memberlist: Stream connection from=10.0.19.9:58068\n"
component=cluster memberlist="2021/11/25 18:45:23 [INFO] memberlist: Suspect 01FNC3Z6NNJ8YBTZT1GW86PHQ5 has failed, no acks received\n"
component=cluster memberlist="2021/11/25 18:45:25 [DEBUG] memberlist: Failed ping: 01FNC3Z6NNJ8YBTZT1GW86PHQ5 (timeout reached)\n"
component=cluster memberlist="2021/11/25 18:45:27 [INFO] memberlist: Marking 01FNC3Z6NNJ8YBTZT1GW86PHQ5 as failed, suspect timeout reached (0 peer confirmations)\n"
component=cluster received=NotifyLeave node=01FNC3Z6NNJ8YBTZT1GW86PHQ5 addr=172.18.0.5:9094
component=cluster msg="peer left" peer=01FNC3Z6NNJ8YBTZT1GW86PHQ5
component=cluster memberlist="2021/11/25 18:45:32 [INFO] memberlist: Suspect 01FNC3Z6NNJ8YBTZT1GW86PHQ5 has failed, no acks received\n"
component=cluster memberlist="2021/11/25 18:45:32 [DEBUG] memberlist: Failed to join 172.18.0.5: dial tcp 172.18.0.5:9094: connect: connection refused\n"
component=cluster msg=reconnect result=failure peer=01FNC3Z6NNJ8YBTZT1GW86PHQ5 addr=172.18.0.5:9094 err="1 error occurred:\n\t* Failed to join 172.18.0.5: dial tcp 172.18.0.5:9094: connect: connection refused\n\n"
component=cluster memberlist="2021/11/25 18:45:37 [DEBUG] memberlist: Initiating push/pull sync with:  10.0.19.7:9094\n"
component=cluster memberlist="2021/11/25 18:45:37 [WARN] memberlist: Refuting a suspect message (from: 01FNC3Z698P4CC5MXV58DZ0BS8)\n"
component=cluster msg=refresh result=success addr=10.0.19.7:9094
component=cluster memberlist="2021/11/25 18:45:38 [DEBUG] memberlist: Stream connection from=10.0.19.9:58158\n"
component=cluster received=NotifyJoin node=01FNC3Z6NNJ8YBTZT1GW86PHQ5 addr=172.18.0.5:9094
component=cluster msg="peer rejoined" peer=01FNC3Z6NNJ8YBTZT1GW86PHQ5
component=cluster memberlist="2021/11/25 18:45:39 [DEBUG] memberlist: Failed ping: 01FNC3Z6NNJ8YBTZT1GW86PHQ5 (timeout reached)\n"
component=cluster memberlist="2021/11/25 18:45:46 [INFO] memberlist: Suspect 01FNC3Z6NNJ8YBTZT1GW86PHQ5 has failed, no acks received\n"
component=cluster memberlist="2021/11/25 18:45:48 [DEBUG] memberlist: Failed ping: 01FNC3Z6NNJ8YBTZT1GW86PHQ5 (timeout reached)\n"
component=cluster memberlist="2021/11/25 18:45:50 [INFO] memberlist: Marking 01FNC3Z6NNJ8YBTZT1GW86PHQ5 as failed, suspect timeout reached (0 peer confirmations)\n"
component=cluster received=NotifyLeave node=01FNC3Z6NNJ8YBTZT1GW86PHQ5 addr=172.18.0.5:9094
component=cluster msg="peer left" peer=01FNC3Z6NNJ8YBTZT1GW86PHQ5
component=cluster memberlist="2021/11/25 18:45:52 [DEBUG] memberlist: Failed to join 172.18.0.5: dial tcp 172.18.0.5:9094: connect: connection refused\n"
component=cluster msg=reconnect result=failure peer=01FNC3Z6NNJ8YBTZT1GW86PHQ5 addr=172.18.0.5:9094 err="1 error occurred:\n\t* Failed to join 172.18.0.5: dial tcp 172.18.0.5:9094: connect: connection refused\n\n"
component=cluster memberlist="2021/11/25 18:45:52 [DEBUG] memberlist: Initiating push/pull sync with:  10.0.19.7:9094\n"
component=cluster memberlist="2021/11/25 18:45:52 [WARN] memberlist: Refuting a suspect message (from: 01FNC3Z698P4CC5MXV58DZ0BS8)\n"
component=cluster msg=refresh result=success addr=10.0.19.7:9094
component=cluster memberlist="2021/11/25 18:45:53 [DEBUG] memberlist: Stream connection from=10.0.19.9:58242\n"
component=cluster received=NotifyJoin node=01FNC3Z6NNJ8YBTZT1GW86PHQ5 addr=172.18.0.5:9094
component=cluster msg="peer rejoined" peer=01FNC3Z6NNJ8YBTZT1GW86PHQ5