[beats receivers] OTel based pipeline must continue to report queue metrics for the elastic_agent integration

[cmacknz]

The primary queue when using Beats receivers is the one in the collector. We need to export queue metrics as part of self-monitoring.

Ideally they are backwards compatible with the existing metrics for queue depth. If they cannot have the same names then the same semantic meaning is the next best thing. We are generally not concerned with breaking changes here.

libbeat.pipeline.events.active measures the number of events in the processing pipeline including those in processors, and has a value that can be slightly above the queue size because of this.

https://github.com/elastic/integrations/blob/037cda7c46f84ca7c1d8055105f5d7fab59fda50/packages/elastic_agent/data_stream/filebeat_metrics/fields/beat-stats-fields.yml#L194-L199

            - name: events
              type: group
              fields:
                - name: active
                  type: long
                  metric_type: counter

Direct measurements of the data in the queue were added recently, specifically with queue.filled.pct giving the queue fill level in percent directly.

https://github.com/elastic/integrations/blob/037cda7c46f84ca7c1d8055105f5d7fab59fda50/packages/elastic_agent/data_stream/filebeat_metrics/fields/beat-stats-fields.yml#L136-L193

                - name: filled.bytes
                  type: long
                  metric_type: gauge
                  description: >
                    Number of bytes filled in the queue
                - name: filled.events
                  type: long
                  metric_type: gauge
                  description: >
                    Number of events filled in the queue
                - name: filled.pct
                  type: float
                  metric_type: gauge
                  description: >
                    Percentage of the queue filled

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[leehinman]

Short term

Get the current data at https://opentelemetry.io/docs/collector/internal-telemetry/ and get it displayed in the elastic-agent dashboards

Longer term

Change what is in https://opentelemetry.io/docs/collector/internal-telemetry/, so we can better provide info like queue pct filled.