Critical deadman alerts not sending notifications into slack

[nathleeuk]

Steps to reproduce:
List the minimal actions needed to reproduce the behavior.

1. Create deadman alert for server in Influx v2.0.2 for linux metric
2. Set up slack notification endpoint
3. Create two notification rules for slack; When status changes from ANY to CRIT & when status changes from ANY to OK
4. Trigger deadman alert (systemctl stop telegraf)
5. Turn telegraf back on to clear deadman alert (reset to ok)

Expected behavior:
When telegraf switched off - Slack notification for agent offline (deadman critical)
When telegraf turned back on - Slack notification for agent online (deadman OK)

Actual behavior:
No dead man alert for CRIT sent to slack
dead man alert for OK was successfully sent to slack

Environment info:
Influx Version: InfluxDB 2.0.2 (git: 84496e5) build_date: 2020-11-19T03:59:35Z

[mhall119]

I was able to reproduce this bug on Cloud 2 using the ANY to CRIT notification rule. The Alert is triggered but the notification rule is not.

It works fine with "Is equal to CRIT" and nothing else changed.

[asalem1]

As far as I can tell, this doesn't look like a UI issue. The UI is configuring the post correctly, seems like there are some issues on the API that need to be resolved in order to get this to work

[kristinarobinson]

This issue will be addressed in our redesign for monitoring and alerting scheduled for Q1.

[glinton]

EDIT
Apparently the reason i was unable to reproduce is because i had the check and the rule on matching schedules (1m, no offset). The only way i was able to reproduce this was by setting my rule schedule to smaller than the check schedule (check 1m, rule 10s; or check 5m, rule 1m).

As a workaround, ensure the scheduled times for your check and any associated rules match exactly.

---

@nathleeuk can you provide more info on how your check and rule were configured. I was unable to reproduce locally with the following check and rules (using slack as the endpoint)

$ ./influx export --check-names "deadman" --rules "06bd11c26effa000,06bd1a60e97fa000"

apiVersion: influxdata.com/v2alpha1
kind: CheckDeadman
metadata:
    name: crumbling-rhodes-c3e001
spec:
    every: 1m0s
    level: CRIT
    name: deadman
    query: |-
        from(bucket: "bkt")
          |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
          |> filter(fn: (r) => r["_measurement"] == "thing")
          |> filter(fn: (r) => r["_field"] == "c")
    staleTime: 10m0s
    status: active
    statusMessageTemplate: 'Check: ${ r._check_name } is: ${ r._level }'
    timeSince: 10s
---
apiVersion: influxdata.com/v2alpha1
kind: NotificationEndpointSlack
metadata:
    name: reverent-meninsky-c3e003
spec:
    description: gregSlack
    name: deadmanSlack
    status: active
    url: https://hooks.slack.com/services/**************
---
apiVersion: influxdata.com/v2alpha1
kind: NotificationRule
metadata:
    name: pedantic-chatterjee-c3e005
spec:
    endpointName: reverent-meninsky-c3e003
    every: 1m0s
    messageTemplate: 'Notification Rule: ${ r._notification_rule_name } triggered
        by check: ${ r._check_name }: ${ r._message }'
    name: testRule
    statusRules:
      - currentLevel: CRIT
---
apiVersion: influxdata.com/v2alpha1
kind: NotificationRule
metadata:
    name: wizardly-heyrovsky-c3e007
spec:
    endpointName: reverent-meninsky-c3e003
    every: 1m0s
    messageTemplate: 'TEST2: triggered by check: ${ r._check_name }: ${ r._message
        } (ANY->CRIT)'
    name: testRule2
    statusRules:
      - currentLevel: CRIT
        previousLevel: ANY

---

Reproduction steps:

1. add data

while true; do
  curl -H "Authorization: Token ${INFLUXDB_TOKEN}" \
  "localhost:8086/api/v2/write?org=org&bucket=bkt" \
  -d 'thing,a=b c=1'; sleep 9s;
done

2. stop sending data after >1m, restart after >1m

[rickspencer3]

I'm going ahead and closing this, as @glinton has tracked down the problem is that the UI is confusing, which, while a serious problem, is slated to be fixed. The solution, for now, is to always set your check schedule and your notification rule schedule to always exactly match.

If it turns out that there is a deeper bug, please reopen the issue. Thank you.

Apparently the reason i was unable to reproduce is because i had the check and the rule on matching schedules (1m, no offset). The only way i was able to reproduce this was by setting my rule schedule to smaller than the check schedule (check 1m, rule 10s; or check 5m, rule 1m).

As a workaround, ensure the scheduled times for your check and any associated rules match exactly.