Support heart beats in argus

[hmpf]

There currently is no way for argus to detect that a source is not capable of sending incidents.

We could have a model

class Heartbeat(models.Model):
    id = models.BigIntegerField(primary_key=True)
    source = models.ForeignKey("argus_incidents.SourceSystem")
    last_seen = models.DateTimeField(auto_add_now=True)

and alter SourceSystemType:

   heartbeat = models.BooleanField(default=False)

We would need a background worker that would, for all SourceSystemTypes with heartbeat set to True, check (at an interval set in settings? cron?), look up Heartbeat's with that source, check the last_seen and raise an incident if it's been too long. We could even have heartbeat be a onetoone key to SourceSystem and UPDATE instead of INSERT, but using a foreign key makes it possible to spot trends.

[hmpf]

It might be cheaper and more robust if we put "last_seen" directly on Source:

class SourceSystem(models.Model):
    ...
    last_seen = models.DateTimeField(null=True, blank=True)
    ...

On every received event we also update "last_seen" with the timestamp of the event.

Such a field is useful and cheap to show in columns even if the source does not support heart beats, but we'd need a read lock (select for update).

For historical trends we could log every update (INFO) of the field.

We could have a table HeartbeatConfiguration to control for how long we wait until we raise a "missing incident"-event, but should we hang it on the source system type or the source (or both?)

class HeartbeatConfiguration(models.Model):
    every = models.IntegerField(help_text="Expect a heart beat every N minutes")

    source = models.ForeignKey("argus_incidents.SourceSystem")

    .. or ..

    source_type = models.OneToOneKey("argus_incidents.SourceSystemType")

[hmpf]

API-wise, the heartbeat setup should be configurable via API so the glue service can configure it itself. Viewable and editable in admin, maybe viewable in frontend.

[lunkwill42]

Isn't this just a duplicate of #1160?