[Meta] Health Report - future phases

[robbavey]

Following the release of the #16056 in Logstash 8.15, and subsequent addition to the Logstash integration, the first phase of the Logstash Health Report was completed.

This meta issue will capture additional work for future iterations of the Logstash health report 👍

Additional pipeline probes

• periodically populate indicator details with immutable snapshot of pipeline flow metrics
• introduce expression-based probe type with configs degraded,critical, and recovery
• add queue_persisted_growth_events probe
• add queue_backpressure probe (requires wall-clock reorientation of underlying flow metric)
• add input_throughput probe
• add output_throughput probe

Resources Indicator

We will provide a top-level indicator to highlight issues with the process's use of resources as a whole.

• create resources indicator
• create probe resources:gc_pause_ratio based on a new flow metric

Iterative Improvements

We will iterate on the API to improve its usefulness, starting with:

• add support to ?verbose=false to GET /_health_report to avoid populating indicator details when building the report
• add support for GET /_health_report/pipelines/<pipeline-id>
• add support for not propagating an indicator's degraded status (settings)
• add support for ACK-ing a currently degraded indicator status (API)
• add report reference, to include along-side anywhere we report status, and allow fetching a specific report for a reasonable ttl.