Skip to content

lifecycle web endpoint for clearing the registry #637

Description

@WojciechKuk

there are endpoints /-/quit and /-/reload
I report the need for the endpoint /-/clear (or /-/reset)

This command would work differently than reload. It would simply clear the registers (as if the TTL of all metrics dropped to zero).

I work with large "clusters" of statsd_exporter, where when the number of instances changes, or their order changes, the metrics distribution key changes (hashing on statsrelay).
In such a case I need a way for all statad_exporters to immediately "forget" the metrics they have accumulated so far, clear the registers and start collecting metrics "in a new distribution".

  • /-/reload doesn't provide me with this - metrics don't get forgotten and new ones start to accumulate mixed with old ones
  • /-/quit doesn't provide this because it's more disruptive. You can of course put a command in the systemd configuration to quickly start the daemon, but it can take a few seconds and it can cause prometheus to mark the backend as "down". I just want to clean the registry, which would be faster (and non-destructive for self-monitoring metrics i.e golang-related)

Other use cases that may appear (and user may need to quickly clean the registry):

  • a wrongly formatted metric accidentally sent to statsd, which needs to be quickly cleaned up so that prometheus does not download it
  • a gauge that is a memento of a non-existent instance of the monitored service - overstating/understating some KPI
  • a mistake in metric routing (e.g. a mistake in redirecting traffic to the wrong backend on the loadbalancer, in a situation where all backends are downloaded equally by prometheus)

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions