Best practice for "grace period"/retries for slightly irregular external-endpoints

[arminus]

What's the best practice for daily jobs whose runtimes vary (like backup jobs), so that I can't rely on them to ping an external-endpoint every 24 hours? i.e. sometimes it might be after 23.something, sometimes 25.something, UptimeKuma has a retry time and number, I can't find anything similar in gatus?

[PythonGermany]

How does the retry time in UptimeKuma work?

For gatus it depends on when you push results to the external endpoint.
Maybe just set the heartbeat interval to the backup interval + the maximum time the backup is expected to take before being considered unhealthy.

You have to consider that the heartbeat interval is only the duration after which the endpoint becomes unhealthy if no heartbeat is received. So usually if the backup job does not hang you can just push a success or failure result upon job completion.

[arminus]

How does the retry time in UptimeKuma work?

they have an explicit setting for Retry Interval and a Number of Retries.

For gatus it depends on when you push results to the external endpoint. Maybe just set the heartbeat interval to the backup interval + the maximum time the backup is expected to take before being considered unhealthy.

wouldn't that potentially gradually shift the "window" if a push time continuously degrades over time?

let's say I give it 26h for a daily check, wouldn't I kind of loose the "anchor" because it measures the delta to the last successful ping?
so if a service takes longer and longer (beyond the implicit +2h) I would never be notified?

with the retry interval, I can keep the anchor at a constant 24h and give it a grace period of retry interval multiplied by number of retries and if it gets beyond that grace period, the fail triggers

[PythonGermany]

So in UptimeKuma it is "anchored" because in a situation where a push happens inside the grace period the start of the next grace period is calculated from the start of the current grace period instead of the timepoint of the push?

[arminus]

That's my assumption/expectation, but I haven't tested this or checked the code. At any rate, a "creep" is something I'd like to be made aware of ideally...

[PythonGermany]

In the last few days I've tried to integrate my daily backup jobs into my gatus dashboard using external endpoints.

To see if a failing/missing backup would reliably trigger a notification I turned of my backup test jobs... the result: Even though I configured a heartbeat interval of 25h and no backup/endpoint push has taken place for 35h the endpoint state has not been updated by gatus itself and I have not received any notification from gatus about it.

I looked into the code a bit more and I learned that after restarting gatus the next heartbeat is not calculated relative to the last endpoint push but instead relative to the restart time which is different from what I would expect without looking into the code. There's already an issue for that: #1237

I also found that an issue exists already requesting and discussing grace period configurability here: #1322