for: '0s' resulting in ignored group_wait?

[jordangarside-kiwi]

I'm not 100% sure if this is what's happening, but...

Is it expected that if for is 0s then group_wait will be ignored?

• When I set for: '0s', then it seems like I get the slack notification immediately from alertmanager.
• When I set for: '5s', then it seems to respect the group_wait period.

I've been doing some testing with group_wait and a test alert that looks like this:

{
  alert: 'ResolvableTestAlert',
  expr: 'floor(time()/120) % 2 == 1', // Fires for 120s, resolves for 120s
  for: '0s',
  annotations: {
    summary: 'Resolvable test alert',
    description:
      'This test alert alternates between firing and resolved states every 120 seconds.',
  },
},

My route looks like this:

{
  match: {
    test: 'true',
  },
  receiver: 'alert-tests-slack',
  continue: false,
  group_wait: '45s',
  group_interval: '30s',
  repeat_interval: '5m',
},

I'm using vmalert with a 15s evaluation interval.

[jordangarside-kiwi]

I wonder if it's possibly because the alert is never in a pending state?

[jordangarside-kiwi]

Example logs where it happens when for: '0s':

# alertmanager-0
time=2025-03-02T06:54:21.119Z level=DEBUG source=dispatch.go:530 msg=flushing component=dispatcher aggrGroup="{}/{test=\"true\"}:{alertname=\"ResolvableTestAlert\"}" alerts=[ResolvableTestAlert[8858589][resolved]]
time=2025-03-02T06:54:23.095Z level=DEBUG source=dispatch.go:165 msg="Received alert" component=dispatcher alert=ResolvableTestAlert[8858589][resolved]
time=2025-03-02T06:54:23.106Z level=DEBUG source=dispatch.go:165 msg="Received alert" component=dispatcher alert=ResolvableTestAlert[8858589][resolved]
time=2025-03-02T06:54:38.094Z level=DEBUG source=dispatch.go:165 msg="Received alert" component=dispatcher alert=ResolvableTestAlert[8858589][active]
time=2025-03-02T06:54:38.104Z level=DEBUG source=dispatch.go:165 msg="Received alert" component=dispatcher alert=ResolvableTestAlert[8858589][active]
time=2025-03-02T06:54:38.368Z level=DEBUG source=nflog.go:537 msg="gossiping new entry" component=nflog entry="entry:<group_key:\"{}/{test=\\\"true\\\"}:{alertname=\\\"ResolvableTestAlert\\\"}\" receiver:<group_name:\"alert-tests-slack\" integration:\"slack\" > timestamp:<seconds:1740898478 nanos:221172205 > firing_alerts:12281106902051802923 > expires_at:<seconds:1740899078 nanos:221172205 > "

# alertmanager-1
time=2025-03-02T06:54:23.062Z level=DEBUG source=dispatch.go:165 msg="Received alert" component=dispatcher alert=ResolvableTestAlert[8858589][resolved]
time=2025-03-02T06:54:23.073Z level=DEBUG source=dispatch.go:165 msg="Received alert" component=dispatcher alert=ResolvableTestAlert[8858589][resolved]
time=2025-03-02T06:54:37.091Z level=DEBUG source=dispatch.go:530 msg=flushing component=dispatcher aggrGroup="{}/{test=\"true\"}:{alertname=\"ResolvableTestAlert\"}" alerts=[ResolvableTestAlert[8858589][resolved]]
time=2025-03-02T06:54:38.062Z level=DEBUG source=dispatch.go:165 msg="Received alert" component=dispatcher alert=ResolvableTestAlert[8858589][active]
time=2025-03-02T06:54:38.072Z level=DEBUG source=dispatch.go:165 msg="Received alert" component=dispatcher alert=ResolvableTestAlert[8858589][active]
time=2025-03-02T06:54:38.335Z level=DEBUG source=nflog.go:537 msg="gossiping new entry" component=nflog entry="entry:<group_key:\"{}/{test=\\\"true\\\"}:{alertname=\\\"ResolvableTestAlert\\\"}\" receiver:<group_name:\"alert-tests-slack\" integration:\"slack\" > timestamp:<seconds:1740898478 nanos:221172205 > firing_alerts:12281106902051802923 > expires_at:<seconds:1740899078 nanos:221172205 > "
time=2025-03-02T06:54:51.518Z level=DEBUG source=nflog.go:537 msg="gossiping new entry" component=nflog entry="entry:<group_key:\"{}/{test=\\\"true\\\"}:{alertname=\\\"ResolvableTestAlert\\\"}\" receiver:<group_name:\"alert-tests-slack\" integration:\"slack\" > timestamp:<seconds:1740898491 nanos:418334518 > resolved_alerts:12281106902051802923 > expires_at:<seconds:1740899091 nanos:418334518 > "

# alertmanager-2
time=2025-03-02T06:54:23.072Z level=DEBUG source=dispatch.go:165 msg="Received alert" component=dispatcher alert=ResolvableTestAlert[8858589][resolved]
time=2025-03-02T06:54:23.083Z level=DEBUG source=dispatch.go:165 msg="Received alert" component=dispatcher alert=ResolvableTestAlert[8858589][resolved]
time=2025-03-02T06:54:23.084Z level=DEBUG source=dispatch.go:530 msg=flushing component=dispatcher aggrGroup="{}/{test=\"true\"}:{alertname=\"ResolvableTestAlert\"}" alerts=[ResolvableTestAlert[8858589][resolved]]
time=2025-03-02T06:54:38.072Z level=DEBUG source=dispatch.go:165 msg="Received alert" component=dispatcher alert=ResolvableTestAlert[8858589][active]
time=2025-03-02T06:54:38.072Z level=DEBUG source=dispatch.go:530 msg=flushing component=dispatcher aggrGroup="{}/{test=\"true\"}:{alertname=\"ResolvableTestAlert\"}" alerts=[ResolvableTestAlert[8858589][active]]
time=2025-03-02T06:54:38.082Z level=DEBUG source=dispatch.go:165 msg="Received alert" component=dispatcher alert=ResolvableTestAlert[8858589][active]
time=2025-03-02T06:54:38.221Z level=DEBUG source=notify.go:876 msg="Notify success" component=dispatcher receiver=alert-tests-slack integration=slack[0] aggrGroup="{}/{test=\"true\"}:{alertname=\"ResolvableTestAlert\"}" attempts=1 duration=148.347898ms alerts=[ResolvableTestAlert[8858589][active]]
time=2025-03-02T06:54:51.529Z level=DEBUG source=nflog.go:537 msg="gossiping new entry" component=nflog entry="entry:<group_key:\"{}/{test=\\\"true\\\"}:{alertname=\\\"ResolvableTestAlert\\\"}\" receiver:<group_name:\"alert-tests-slack\" integration:\"slack\" > timestamp:<seconds:1740898491 nanos:418334518 > resolved_alerts:12281106902051802923 > expires_at:<seconds:1740899091 nanos:418334518 > "

I have 2 vmalert servers sending alerts to all 3 alertmanagers.

[grobinson-grafana]

I believe what's happening is the group_interval timer (30s) is in phase with the evaluation interval (15s). As 30s and 15s are both multiples of 15, the group_interval sometimes occurs at the exact same time as the alert is sent, causing a notification to be sent immediately. Remember that the group_interval timer is not relative to the alert, but it's a recurring timer for the alert group as a whole. In between the alert resolving and firing again, the timer is still counting down.

I wrote up a blog post that explains how these timers work if it helps Group wait, Group interval and Repeat interval explained.

[jordangarside-kiwi]

Ohh I see, I didn't realize that group_wait only occurs on the very first alert event when the group is created.

I assumed there was a timer that started every time the group switched from resolved->firing.

Thanks!