Support heart beats in argus

[hmpf]

There currently is no way for argus to detect that a source is not capable of sending incidents.

See #1160 for META-issue, what we decide on should be turned into sub-issues of that one.

We could have a model

class Heartbeat(models.Model):
    id = models.BigIntegerField(primary_key=True)
    source = models.ForeignKey("argus_incidents.SourceSystem")
    last_seen = models.DateTimeField(auto_add_now=True)

and alter SourceSystemType:

   heartbeat = models.BooleanField(default=False)

We would need a background worker that would, for all SourceSystemTypes with heartbeat set to True, check (at an interval set in settings? cron?), look up Heartbeat's with that source, check the last_seen and raise an incident if it's been too long. We could even have heartbeat be a onetoone key to SourceSystem and UPDATE instead of INSERT, but using a foreign key makes it possible to spot trends.

[hmpf]

It might be cheaper and more robust if we put "last_seen" directly on Source:

class SourceSystem(models.Model):
    ...
    last_seen = models.DateTimeField(null=True, blank=True)
    ...

On every received event or heartbeat if that is not an event we also update "last_seen" with the timestamp of the event. Such a field is useful and cheap to show in columns even if the source does not support heart beats, but we'd need a read lock (select for update).

For historical trends we could log every update (INFO) of the field.

We could have a table HeartbeatConfiguration to control for how long we wait until we raise a "missing incident"-event, but should we hang it on the source system type or the source (or both?)

class HeartbeatConfiguration(models.Model):
    every = models.IntegerField(help_text="Expect a heart beat every N minutes")

    source = models.ForeignKey("argus_incidents.SourceSystem")

    .. or ..

    source_type = models.OneToOneKey("argus_incidents.SourceSystemType")

[hmpf]

We need a way to mark which source system types will create "missing heartbeat" incidents, but we don't need config at first, we can have a fallback for the entire site as a setting. Toggling support for heartbeats needs to be a site-wide setting anyway.

[hmpf]

API-wise, the heartbeat setup should be configurable via API so the glue service can configure it itself. Viewable and editable in admin, maybe viewable in frontend.

[lunkwill42]

Isn't this just a duplicate of #1160?

[hmpf]

More flesh and less waffle, this one.

[hmpf]

Absolute minimum change:

last_seen on SourceSystem, updated and logged on every event.

Bonus

• We could have a frontend page with a list just sources and the last time they were seen. (Never seen in their own category?)
• Show in admin, or wherever we show source?

Next step:

Mark source system type for if the glue service produces explicit heart beats. Two global settings: to toggle heart beat support on or off, global fallback period to check for heartbeats for source system types that support it. Background task/cron checks for heartbeats for enabled types and makes an incident if too long since the last time. OPEN QUESTIONS:

• what is the source/owner of the "missing heartbeat" incident? (I'd say, not the actual source!)
• should each heart beat be an event and shown in the events-list, hidden/just logged to syslog, or shown somewhere else, which would need a row per heartbeat?

Last step?

Source system type is marked for how frequently to check for heart beats. API for glue service to configure heart beats. In the docs, mark which glue services produces heart beats.

[lunkwill42]

I think heartbeart frequency should be configurable per source system, since it might be configurable per glue service instance. But there could, of course, be a suggested default value taken from the source type.

Having a page to list sources and their last seen timestamps seems like a good bonus idea.

Re: your open questions:

what is the source/owner of the "missing heartbeat" incident? (I'd say, not the actual source!)

For me, Argus itself is the natural source of the "missing heartbeat" incident. Argus monitors the heartbeart and creates the incident. When a heartbeat is received again, Argus closes the incident.

should each heart beat be an event and shown in the events-list, hidden/just logged to syslog, or shown somewhere else, which would need a row per heartbeat?

I am still concerned about the idea of putting all heartbeats into an event log in the database, it sounds very spammy to me. Having a "last seen" timestamp per source system seems good enough - only when "last seen" becomes too old do we start complaining and logging things.

Also, should we consider any contact with the API from a given source system a form of "heart beat"? I mean, we have been talking about having a formal API endpoint to send the "i am still alive" message - but the point is to make it noticable when a source system or its glue service "dies". The "last seen" timestamp could be updated any time the source contacts the API.