monitoring: apm-server only ships a subset of monitoring metrics

[carsonip]

Discovered by #13244

EA managed apm-server is only shipping a subset of apm-server monitoring metrics. This limits observability of EA managed apm-server. The problem is spread over the following use cases:

• APM server self-monitoring
  • metricsets state, stats
  • indices (not data stream) .monitoring-beats-7-*
  • does NOT use TSDS, but fields are unmapped due to dynamic: false
  • code in ES repo monitoring-beats.json
• Stack monitoring: metricbeat beat-xpack module uses index template embedded in ES
  • metricsets state, stats
  • data stream .monitoring-beats-8-mb
  • use case: Elastic Cloud self-monitoring, metricbeat standalone beat-xpack
  • does NOT use TSDS, but fields are unmapped due to dynamic: false
  • code in ES repo monitoring-beats-mb.json
• Metricbeat without xpack: metricbeat beat module
  • metricsets state, stats
  • data stream metricbeat-*
  • use case: metricbeat standalone
  • does NOT use TSDS, fields are mapped dynamically
  • code in beats repo
• EA integration
  • metricsets stats only
  • data stream metrics-elastic_agent.apm_server.* (see code)
  • use case: elastic-agent standalone with monitoring
  • to see EA integration package content: POST kbn:/api/fleet/epm/packages/elastic_agent
  • uses TSDS, see [Elastic Agent] Enable TSDB by default integrations#7214 . The issue with using TSDS is that unmapped fields are dropped, as it uses synthetic source.
  • code in integrations repo
  • it doesn't contain any apm-server fields as of today

Unknowns:

• As the outdated mappings above contain fields that are no longer used, mapping generated from apm-server:5066/stats will not be pure addition. Not sure if dropping a field mapping will be problematic.
  • Should it be a superset of all historically available monitoring fields? That doesn't sound realistic either

[carsonip]

Testing

Completed local testing on all elastic/beats#40127, elastic/elasticsearch#110568, and elastic/integrations#10414. Note that both elastic/elasticsearch#110568 and elastic/integrations#10414 require elastic/beats#40127 to have output.elasticsearch.* metrics parsed.

apm-server self monitoring (indices .monitoring-beats-7-*)

before

after

metricbeat beat-xpack (DS .monitoring-beats-8-mb)

before

after

metricbeat beat standalone (DS metricbeat-*)

before

after

EA agent monitoring (DS metrics-elastic_agent-apm_server-*)

before

after

Long term solution

While the Python script in #13638 generates the correct mapping for all the above use cases, this approach is not very maintainable. Ideally, having dynamic: true like metricbeat standalone enables apm-server monitoring fields to be mapped dynamically and will require minimal maintenance. However, since EA agent mapping already has TSDB enabled, disabling TSDB now and enabling dynamic mapping sounds like a step back.