Send consecutive alerts if endpoint remains unavailable

[mxcd]

Describe the feature request

Currently, when a service becomes unavailable an alert is being triggered once the failure threshold is reached.
If the endpoint remains unavailable, no further notification is being sent out. Keeping everyone informed about a service (still) being down, it would be nice to be able to configure consecutive emails for unhealthy endpoints after a customizable interval.

Why do you personally want this feature to be implemented?

No response

How long have you been using this project?

1 year

Additional information

No response

[bugrakocabay]

Maybe adding a bool flag named consecutive under alerts, that will allow sending notification on every
failure-threshold's multiple.

alerts:
      - type: discord
        description: "healthcheck failed"
        send-on-resolved: true
        failure-threshold: 2
        consecutive: true

If this works, I can implement it.

[mxcd]

Hi @bugrakocabay,
although your interpretation of consecutive in terms of wording is probably more precise, it is not exactly what I intended with this feature request.
I would imagine it to be quite annoying to be notified in the interval of the failure threshold.
Example:

interval: 60s
failure-threshold: 3

This would result in a new notification every 3 minutes.
If I would want a notification every day (as a gentle reminder to fix something that is broken) I would not be able to configure this while keeping acceptable initial triggering times.
Any thoughts on this?

[bugrakocabay]

Hi @mxcd ,
so what you have in mind is, having the flexibility to have different time intervals for the initial alert and subsequent reminders. For instance, you want the first alert to trigger after 3 failures (in our example, that would be after 3 minutes). But for the consecutive reminders, you want it less frequent, say, once every day instead of every 3 minutes.

If so, I think having this kind of configuration would solve it;

alerts:
      - type: discord
        description: "healthcheck failed"
        send-on-resolved: true
        failure-threshold: 3
        reminder-interval: 1d

[mxcd]

Hi @bugrakocabay,

that's exactly what I had in mind. Although I also see a use-case for your consecutive: true logic in case of super-critical systems or applications.

I would opt for having both options, unless @TwiN would veto due to the configuration becoming too cluttered by offering both.

If keeping it lean is desired, I'd suggest just implementing the reminder-interval due to it's higher flexibility since it can also achieve the "3 minute consecutive reminders"

[bugrakocabay]

Hi @TwiN

What is your opinion about this? How should we proceed?

[AreebSiddiqui]

This is exactly what I am missing in this tool. Would love to see it in action.

[amai2012]

Related issue: #379

[TwiN]

Yep - this is a duplicate of #379.
Please 👍 the aforementioned issue if you'd like to see this added.