[Metricbeat] Add elasticsearch/security_stats metricset (#50674) (#50883)

Adds a per-node security_stats metricset that scrapes the new
GET /_security/stats endpoint introduced in Elasticsearch 9.2.
The first metric exposed is the Document Level Security cache
(entries, memory, hits, misses, evictions, hit/miss latency),
giving Stack Monitoring fleet-wide visibility into DLS cache
health for spotting cache thrash, oversized working sets, and
unhealthy hit/miss ratios.

Each event is enriched with node name, roles, and stack version
via a single filter-path-scoped /_nodes call per scrape, shared
across all per-node events emitted in that scrape. This logic
lives on the module's MetricSet as the new NodeEnrichment helper
so future per-node metricsets can reuse it. node.version is also
declared at the module level alongside id, name, roles, master,
and mlockall.

The shared metricbeat/docker-compose.yml elasticsearch service
now runs with xpack.security.enabled=true plus an anonymous
superuser, since /_security/stats is only registered when
security is enabled. Anonymous superuser keeps the rest of the
elasticsearch integration test suite working without threading
credentials through every metricset's setup.

* docs: register security_stats metricset page in toc.yml

mage update regenerates per-metricset markdown but doesn't touch
the navigation toc.yml. Add the missing entry so docs-build can
locate the security_stats page in the Elasticsearch module section.

* docs: replace "e.g." with "for example" per Vale style guide

Elastic.Latinisms forbids Latin abbreviations in docs. Replace
the lone "e.g." in the new node.version field description and
regenerate the affected files.

* metricbeat/elasticsearch: clean up pre-existing lint issues

Two pre-existing lint findings in elasticsearch_integration_test.go
became blocking once this branch touched the file (golangci-lint
runs with --whole-files). Both fixes are mechanical:

- Replace math/rand with math/rand/v2 in randString and drop the
  redundant per-call seeded local Rand.
- Add the comma-ok form to the version.number type assertion in
  getElasticsearchVersion so errcheck (with check-type-assertions)
  is satisfied.

* metricbeat/elasticsearch: dedupe node.version field declaration

The new module-level node.version added for security_stats collided
with a pre-existing node.version in the node metricset's local
fields.yml, breaking `metricbeat export index-pattern` with
"field <elasticsearch.node.version> is duplicated".

Drop the metricset-local declaration in favor of the shared
module-level one, which carries a richer description and is the
right scope for a field emitted by multiple per-node metricsets.

* metricbeat: provision file-realm users for secured ES test stack

Enabling xpack.security on the shared elasticsearch service for
security_stats coverage broke Kibana boot: Kibana 9.x's interactive
setup plugin holds preboot when ES has security on without
ELASTICSEARCH_USERNAME, and the existing Kibana healthchecks
(curl -u beats:testing, curl -u myelastic:changeme) started actually
validating against ES instead of being silently ignored.

Provision the named users that the existing healthchecks expect via
elasticsearch-users useradd in the startup command, and give Kibana
real ES credentials. Anonymous=superuser is preserved so the
integration tests' credential-less HTTP probes keep working without
threading credentials through every metricset's setup.

* x-pack/metricbeat: give kibana credentials to secured ES

The previous commit enabled xpack.security on the shared Elasticsearch
service and gave the OSS metricbeat kibana service real credentials,
but x-pack/metricbeat hand-copies its kibana stanza (depends_on can't
be extended) so the env didn't propagate. With no
ELASTICSEARCH_USERNAME, Kibana entered interactive setup, the
Dockerfile healthcheck (curl -u myelastic:changeme /api/stats) never
reached green, and the proxy_dep busybox blocked all integration
tests from starting.

Mirror the env vars into the x-pack kibana stanza and note the
duplication contract in a comment so future secured-ES changes are
applied in both places.

* metricbeat/elasticsearch: gate security_stats on xpack feature flag

CI exposed that the previous PR commits enabled xpack.security on the
shared metricbeat docker-compose stack to exercise /_security/stats,
but that change rippled wider than fits in this PR: Kibana boot,
healthcheck users, OTel test framework default credentials, and the
Python `get_version` helper all assume an open ES. Revert both
metricbeat and x-pack/metricbeat docker-compose.yml to their
upstream/main shape and address the underlying problem in the
metricset itself.

`security_stats.checkAvailability` now mirrors the pattern used by
ccr and ml_job: a free in-memory version comparison short-circuits
old clusters first, then a proactive `GET /_xpack` probe checks
`features.security.enabled` so we can emit a specific operator-facing
log message and avoid hitting an endpoint we know would return 400.
A new `Security` field is added to the shared `elasticsearch.XPack`
struct to support the check.

The elasticsearch_integration_test.go suite skips security_stats
unconditionally for now, with a TODO pointing at a focused follow-up
PR that migrates the metricbeat compose stack to an x-pack-security-
enabled posture (file-realm users, Kibana credentials, test fixture
auth). At that point the skip becomes vacuous and the metricset is
exercised against a real /_security/stats response.

---------


(cherry picked from commit 7119b64)

Co-authored-by: Elliot Barlas <elliotbarlas@gmail.com>
Co-authored-by: Visha Angelova <91186315+vishaangelova@users.noreply.github.com>

Author: 3 people

changelog/fragments/1777351182-add-elasticsearch-security-stats-metricset.yaml

@@ -0,0 +1,12 @@
+kind: feature
+summary: Add `elasticsearch/security_stats` metricset for the Elasticsearch module
+description: |
+  Adds a new `security_stats` metricset to the Elasticsearch module that
+  scrapes the per-node `GET /_security/stats` endpoint (available since
+  Elasticsearch 9.2). The first metric exposed is the Document Level Security
+  cache (entries, memory, hits, misses, evictions, hit/miss latency), enabling
+  fleet-wide observability of DLS cache health from Stack Monitoring. Each
+  event is enriched with node name, roles, and version via a single
+  filter-path-scoped /_nodes call per scrape so consumers can slice by node,
+  role, or stack version without joining across data streams.
+component: metricbeat

docs/reference/metricbeat/exported-fields-elasticsearch.md

@@ -982,6 +982,12 @@ Elasticsearch module
     type: boolean
 
 
+**`elasticsearch.node.version`**
+:   Elasticsearch version reported by the node (for example, `9.2.0`).
+
+    type: keyword
+
+
 ## ccr [_ccr]
 
 Cross-cluster replication stats
@@ -2096,12 +2102,6 @@ ml
 
 node
 
-**`elasticsearch.node.version`**
-:   Node version.
-
-    type: keyword
-
-
 ## jvm [_jvm]
 
 JVM Info.
@@ -2874,6 +2874,66 @@ File system summary
     type: long
 
 
+## security.stats [_security.stats]
+
+```{applies_to}
+stack: ga 9.2.0
+```
+
+Per-node security statistics collected from the Elasticsearch Security Stats API (`GET /_security/stats`). Available since Elasticsearch 9.2.
+
+## dls [_dls]
+
+Document Level Security (DLS) counters.
+
+## cache [_cache]
+
+Per-node cache used to materialize the bitsets that enforce DLS queries. Sourced from the `roles.dls.bit_set_cache` block in the `GET /_security/stats` response.
+
+**`elasticsearch.security.stats.dls.cache.entries.count`**
+:   Current number of cached entries on this node.
+
+    type: long
+
+
+**`elasticsearch.security.stats.dls.cache.memory.bytes`**
+:   Current memory consumed by the DLS cache on this node, in bytes.
+
+    type: long
+
+    format: bytes
+
+
+**`elasticsearch.security.stats.dls.cache.hits.count`**
+:   Number of cache lookups served from the cache since node startup.
+
+    type: long
+
+
+**`elasticsearch.security.stats.dls.cache.misses.count`**
+:   Number of cache lookups that had to materialize an entry since node startup.
+
+    type: long
+
+
+**`elasticsearch.security.stats.dls.cache.evictions.count`**
+:   Number of entries evicted (due to size limit or expiration) since node startup.
+
+    type: long
+
+
+**`elasticsearch.security.stats.dls.cache.hits.time.ms`**
+:   Cumulative time spent serving cache hits since node startup, in milliseconds.
+
+    type: long
+
+
+**`elasticsearch.security.stats.dls.cache.misses.time.ms`**
+:   Cumulative time spent materializing entries on cache misses since node startup, in milliseconds.
+
+    type: long
+
+
 ## shard [_shard]
 
 shard fields

docs/reference/metricbeat/metricbeat-metricset-elasticsearch-security_stats.md

@@ -0,0 +1,85 @@
+---
+mapped_pages:
+  - https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-metricset-elasticsearch-security_stats.html
+applies_to:
+  stack: ga 9.2.0
+  serverless: ga
+---
+
+% This file is generated! See metricbeat/scripts/mage/docs_collector.go
+
+# Elasticsearch security_stats metricset [metricbeat-metricset-elasticsearch-security_stats]
+
+This is the `security_stats` metricset of the Elasticsearch module. It queries the Security Stats API endpoint (`GET /_security/stats`, available since Elasticsearch 9.2) to collect per-node security counters. The endpoint exposes Document Level Security (DLS) cache statistics, which are useful for spotting cache thrash, oversized working sets, and unhealthy hit/miss ratios across a fleet.
+
+Each emitted event is enriched with `node.{name,roles,version}` (alongside `node.id`) via a single side-channel `/_nodes` call per scrape, so consumers can slice by node, role, or stack version without joining across data streams.
+
+The `/_security/stats` endpoint is only served when the Elasticsearch security feature is enabled (`xpack.security.enabled: true`). The metricset checks `GET /_xpack` on each scrape. When security is disabled, it emits a throttled debug log, but no events.
+
+Authorization follows the same model as `/_cluster/stats` and `/_nodes/stats`: the caller needs the `monitor` cluster privilege.
+
+## Fields [_fields]
+
+For a description of each field in the metricset, see the [exported fields](/reference/metricbeat/exported-fields-elasticsearch.md) section.
+
+Here is an example document generated by this metricset:
+
+```json
+{
+    "@timestamp": "2026-04-27T20:00:00.000Z",
+    "elasticsearch": {
+        "cluster": {
+            "id": "WocBBA0QRma0sGpdQ7vLfQ",
+            "name": "docker-cluster"
+        },
+        "node": {
+            "id": "f5i3v9hMT_q__q6B9WOo5A",
+            "name": "instance-0000000019",
+            "roles": ["data_hot", "ingest"],
+            "version": "9.2.0"
+        },
+        "security": {
+            "stats": {
+                "dls": {
+                    "cache": {
+                        "entries": {
+                            "count": 12
+                        },
+                        "memory": {
+                            "bytes": 4096
+                        },
+                        "hits": {
+                            "count": 8421,
+                            "time": {
+                                "ms": 51
+                            }
+                        },
+                        "misses": {
+                            "count": 137,
+                            "time": {
+                                "ms": 219
+                            }
+                        },
+                        "evictions": {
+                            "count": 4
+                        }
+                    }
+                }
+            }
+        }
+    },
+    "event": {
+        "dataset": "elasticsearch.security.stats",
+        "duration": 115000,
+        "module": "elasticsearch"
+    },
+    "metricset": {
+        "name": "security_stats",
+        "period": 10000
+    },
+    "service": {
+        "address": "172.19.0.2:9200",
+        "type": "elasticsearch"
+    }
+}
+```

docs/reference/metricbeat/metricbeat-module-elasticsearch.md

@@ -112,4 +112,5 @@ The following metricsets are available:
 * [node](/reference/metricbeat/metricbeat-metricset-elasticsearch-node.md)
 * [node_stats](/reference/metricbeat/metricbeat-metricset-elasticsearch-node_stats.md)
 * [pending_tasks](/reference/metricbeat/metricbeat-metricset-elasticsearch-pending_tasks.md)
+* [security_stats](/reference/metricbeat/metricbeat-metricset-elasticsearch-security_stats.md)  {applies_to}`stack: ga 9.2.0`
 * [shard](/reference/metricbeat/metricbeat-metricset-elasticsearch-shard.md)

docs/reference/metricbeat/metricbeat-modules.md

@@ -33,7 +33,7 @@ This section contains detailed information about the metric collecting modules c
 | [CouchDB](/reference/metricbeat/metricbeat-module-couchdb.md) | ![Prebuilt dashboards are available](images/icon-yes.png "") | [server](/reference/metricbeat/metricbeat-metricset-couchdb-server.md) |
 | [Docker](/reference/metricbeat/metricbeat-module-docker.md) | ![Prebuilt dashboards are available](images/icon-yes.png "") | [container](/reference/metricbeat/metricbeat-metricset-docker-container.md)<br>[cpu](/reference/metricbeat/metricbeat-metricset-docker-cpu.md)<br>[diskio](/reference/metricbeat/metricbeat-metricset-docker-diskio.md)<br>[event](/reference/metricbeat/metricbeat-metricset-docker-event.md)<br>[healthcheck](/reference/metricbeat/metricbeat-metricset-docker-healthcheck.md)<br>[image](/reference/metricbeat/metricbeat-metricset-docker-image.md)<br>[info](/reference/metricbeat/metricbeat-metricset-docker-info.md)<br>[memory](/reference/metricbeat/metricbeat-metricset-docker-memory.md)<br>[network](/reference/metricbeat/metricbeat-metricset-docker-network.md)<br>[network_summary](/reference/metricbeat/metricbeat-metricset-docker-network_summary.md) {applies_to}`stack: beta` |
 | [Dropwizard](/reference/metricbeat/metricbeat-module-dropwizard.md) | ![No prebuilt dashboards](images/icon-no.png "") | [collector](/reference/metricbeat/metricbeat-metricset-dropwizard-collector.md) |
-| [Elasticsearch](/reference/metricbeat/metricbeat-module-elasticsearch.md) | ![No prebuilt dashboards](images/icon-no.png "") | [ccr](/reference/metricbeat/metricbeat-metricset-elasticsearch-ccr.md)<br>[cluster_stats](/reference/metricbeat/metricbeat-metricset-elasticsearch-cluster_stats.md)<br>[enrich](/reference/metricbeat/metricbeat-metricset-elasticsearch-enrich.md)<br>[index](/reference/metricbeat/metricbeat-metricset-elasticsearch-index.md)<br>[index_recovery](/reference/metricbeat/metricbeat-metricset-elasticsearch-index_recovery.md)<br>[index_summary](/reference/metricbeat/metricbeat-metricset-elasticsearch-index_summary.md)<br>[ingest_pipeline](/reference/metricbeat/metricbeat-metricset-elasticsearch-ingest_pipeline.md) {applies_to}`stack: beta`<br>[ml_job](/reference/metricbeat/metricbeat-metricset-elasticsearch-ml_job.md)<br>[node](/reference/metricbeat/metricbeat-metricset-elasticsearch-node.md)<br>[node_stats](/reference/metricbeat/metricbeat-metricset-elasticsearch-node_stats.md)<br>[pending_tasks](/reference/metricbeat/metricbeat-metricset-elasticsearch-pending_tasks.md)<br>[shard](/reference/metricbeat/metricbeat-metricset-elasticsearch-shard.md) |
+| [Elasticsearch](/reference/metricbeat/metricbeat-module-elasticsearch.md) | ![No prebuilt dashboards](images/icon-no.png "") | [ccr](/reference/metricbeat/metricbeat-metricset-elasticsearch-ccr.md)<br>[cluster_stats](/reference/metricbeat/metricbeat-metricset-elasticsearch-cluster_stats.md)<br>[enrich](/reference/metricbeat/metricbeat-metricset-elasticsearch-enrich.md)<br>[index](/reference/metricbeat/metricbeat-metricset-elasticsearch-index.md)<br>[index_recovery](/reference/metricbeat/metricbeat-metricset-elasticsearch-index_recovery.md)<br>[index_summary](/reference/metricbeat/metricbeat-metricset-elasticsearch-index_summary.md)<br>[ingest_pipeline](/reference/metricbeat/metricbeat-metricset-elasticsearch-ingest_pipeline.md) {applies_to}`stack: beta`<br>[ml_job](/reference/metricbeat/metricbeat-metricset-elasticsearch-ml_job.md)<br>[node](/reference/metricbeat/metricbeat-metricset-elasticsearch-node.md)<br>[node_stats](/reference/metricbeat/metricbeat-metricset-elasticsearch-node_stats.md)<br>[pending_tasks](/reference/metricbeat/metricbeat-metricset-elasticsearch-pending_tasks.md)<br>[security_stats](/reference/metricbeat/metricbeat-metricset-elasticsearch-security_stats.md) {applies_to}`stack: ga 9.2.0`<br>[shard](/reference/metricbeat/metricbeat-metricset-elasticsearch-shard.md) |
 | [Envoyproxy](/reference/metricbeat/metricbeat-module-envoyproxy.md) | ![No prebuilt dashboards](images/icon-no.png "") | [server](/reference/metricbeat/metricbeat-metricset-envoyproxy-server.md) |
 | [Etcd](/reference/metricbeat/metricbeat-module-etcd.md) | ![No prebuilt dashboards](images/icon-no.png "") | [leader](/reference/metricbeat/metricbeat-metricset-etcd-leader.md)<br>[metrics](/reference/metricbeat/metricbeat-metricset-etcd-metrics.md) {applies_to}`stack: beta`<br>[self](/reference/metricbeat/metricbeat-metricset-etcd-self.md)<br>[store](/reference/metricbeat/metricbeat-metricset-etcd-store.md) |
 | [Google Cloud Platform](/reference/metricbeat/metricbeat-module-gcp.md) | ![Prebuilt dashboards are available](images/icon-yes.png "") | [billing](/reference/metricbeat/metricbeat-metricset-gcp-billing.md)<br>[carbon](/reference/metricbeat/metricbeat-metricset-gcp-carbon.md) {applies_to}`stack: beta`<br>[compute](/reference/metricbeat/metricbeat-metricset-gcp-compute.md)<br>[dataproc](/reference/metricbeat/metricbeat-metricset-gcp-dataproc.md)<br>[firestore](/reference/metricbeat/metricbeat-metricset-gcp-firestore.md)<br>[gke](/reference/metricbeat/metricbeat-metricset-gcp-gke.md)<br>[loadbalancing](/reference/metricbeat/metricbeat-metricset-gcp-loadbalancing.md)<br>[metrics](/reference/metricbeat/metricbeat-metricset-gcp-metrics.md)<br>[pubsub](/reference/metricbeat/metricbeat-metricset-gcp-pubsub.md)<br>[storage](/reference/metricbeat/metricbeat-metricset-gcp-storage.md)<br>[vertexai_logs](/reference/metricbeat/metricbeat-metricset-gcp-vertexai_logs.md) {applies_to}`stack: beta 9.2.0` |

docs/reference/toc.yml

@@ -935,6 +935,7 @@ toc:
               - file: metricbeat/metricbeat-metricset-elasticsearch-node.md
               - file: metricbeat/metricbeat-metricset-elasticsearch-node_stats.md
               - file: metricbeat/metricbeat-metricset-elasticsearch-pending_tasks.md
+              - file: metricbeat/metricbeat-metricset-elasticsearch-security_stats.md
               - file: metricbeat/metricbeat-metricset-elasticsearch-shard.md
           - file: metricbeat/metricbeat-module-envoyproxy.md
             children:

metricbeat/include/list_common.go

@@ -11,6 +11,7 @@
   #  - node
   #  - node_stats
   #  - pending_tasks
+  #  - security_stats
   #  - shard
   xpack.enabled: true
   period: 10s

metricbeat/module/elasticsearch/_meta/config-xpack.yml

@@ -693,3 +693,8 @@
               type: boolean
               description: >
                 Is mlockall enabled on the node?
+            - name: version
+              type: keyword
+              description: >
+                Elasticsearch version reported by the node (for example,
+                `9.2.0`).

metricbeat/module/elasticsearch/_meta/fields.yml

@@ -50,6 +50,7 @@ func NewModule(base mb.BaseModule) (mb.Module, error) {
 		"index_summary",
 		"ml_job",
 		"node_stats",
+		"security_stats",
 		"shard",
 	}
 	optionalXpackMetricsets := []string{"ingest_pipeline"}
@@ -63,6 +64,10 @@ var (
 	// EnrichStatsAPIAvailableVersion is the version of Elasticsearch since when the Enrich stats API is available.
 	EnrichStatsAPIAvailableVersion = version.MustNew("7.5.0")
 
+	// SecurityStatsAPIAvailableVersion is the version of Elasticsearch since when the Security Stats API
+	// (GET /_security/stats) is available.
+	SecurityStatsAPIAvailableVersion = version.MustNew("9.2.0")
+
 	// BulkStatsAvailableVersion is the version since when bulk indexing stats are available
 	BulkStatsAvailableVersion = version.MustNew("8.0.0")
 
@@ -333,6 +338,9 @@ type XPack struct {
 		ML struct {
 			Enabled bool `json:"enabled"`
 		} `json:"ml"`
+		Security struct {
+			Enabled bool `json:"enabled"`
+		} `json:"security"`
 	} `json:"features"`
 }