Allow for success/failure counts to be reset at user request or periodically #40
Description
Currently, the success and failure counts associated with each healthcheck will start at 0 when first registered and monotonically increase over the life of the healthcheck (or active-monitor controller).
This works alright, however, one downside of such an approach is that you need to know how long the healthcheck has been running in order to have any context for whether the counts are large or small.
Therefore, the aim of this ticket is to provide:
- a mechanism for a user to "reset" the counts so that they will better understand what a non-0 failure count means, for instance OR
- a new property in the spec which allows for the user to indicate how frequently the counts should be reset to 0 (this would allow for alerting if the failure count surpassed N within that time period) OR
- an addl status property which would represent a moving average rate of success/fail over some unit time (ex: 4 successes/hr, 2 failures/day)
Open questions:
- which approach to pursue
- what unit to use as the default time duration, ex: hr, day (if 3rd approach above is pursued)
- are there any better strategies which allow for counts to reliably be used in alerting logic