metrics: Processing Engine

[peterbarnett03]

Processing Engine Metrics

Right now we serve little information on Processing Engine performance, so this ticket looks to add some basic metrics to track.

Update the /metrics endpoint to serve the following metrics:

New Processing Engine Metrics:

• plugin_execution_duration_seconds_bucket: Amount of time spent executing a plugin, per plugin, per trigger, with trigger type, bucketed into 0.001, 0.0025, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 10, inf seconds
  --- Plugin name should be entered as the file name without .py on the end.
• plugin_execution_duration_seconds_sum: Total amount of time spent executing a plugin, per plugin, per trigger, with trigger type
• plugin_execution_duration_seconds_count: Total number of times a plugin is executed, per plugin, per trigger, with trigger type
• processing_engine_memory_size_bytes: Total size of Processing Engine memory in bytes
  --- If this can be broken down further into threads, that'd be great, but not required
• processing_engine_plugin_errors: Total number of errors, per plugin, per trigger
• processing_engine_memory_size_bytes: Total size of Processing Engine memory in bytes

E.g.

...
plugin_execution_duration_seconds_bucket{plugin="sample_plugin",trigger="sample_trigger",type="on_request",le="0.05"} 2
plugin_execution_duration_seconds_bucket{plugin="sample_plugin",trigger="sample_trigger",type="on_request",le="0.1"} 3
plugin_execution_duration_seconds_bucket{plugin="sample_plugin",trigger="sample_trigger",type="on_request",le="0.25"} 5
...
plugin_execution_duration_seconds_sum{plugin="sample_plugin",trigger="sample_trigger",type="on_request"} 0.68
plugin_execution_duration_seconds_count{plugin="sample_plugin",trigger="sample_trigger",type="on_request"} 5