[Entity Analytics] Anomaly detection behavior maintainer (elastic#269309)

## Summary

> [!NOTE]
> This only contains the server side changes for the entity behavior
feature. UI changes to come in subsequent PRs.

Introduces a new entity maintainer (`ml-anomaly-detection-jobs`) that
maintains the `entity.behaviors.anomaly_job_ids` field for an entity in
the entity store. This maintainer runs every 24 hours and looks back 90
days in order to capture all of the anomalous behavior for an entity in
the last 90 days.

During each run, the maintainer:
- Iterates over user and host entities from the entity store in batches
- For each batch, fetches anomaly records from security ML jobs for the
last 90 days and above the configured threshold minimum
- If anomaly records exist for an entity, its entity store entry is
updated to include the anomaly job ID.
- If anomaly records exist for an entity, additional supporting details
will be queried and stored in a details datastream
`.entity_analytics.ml-ad-jobs-latest-${namespace}`

The additional details that are fetched are job dependent:

For jobs that use the `rare` function (for example, rare country login),
only the anomalous value is stored in the anomaly record (for example,
`Iran`). In order to determine the baseline behavior for the entity, we
use the ML job configuration to aggregate against the source index (for
example, an aggregation to determine where an entity commonly logs in)

For other job types that are metric or count functions (for example,
high number of failed logins), the record document contains the typical
value and the anomalous value so we already have the baseline behavior.

For all job types, we grab the latest 3 anomalous documents. This is to
support the "Raw Evidence" portion of the expanded section in the
initial UI mockups. Note that the exact format of these documents may
change as we finalize the mockups but since this feature is behind a
feature flag, it should be ok to merge and finalize later.

<img width="463" height="374" alt="Screenshot 2026-05-18 at 4 12 06 PM"
src="https://github.com/user-attachments/assets/19af8c51-650d-40b2-8b9e-548daee8ac5e"
/>

## To Verify

1. Modify the default lookback period of the entity store logs
extraction task (because we're populating historical data)

```
--- a/x-pack/solutions/security/plugins/entity_store/server/domain/saved_objects/global_state/constants.ts
+++ b/x-pack/solutions/security/plugins/entity_store/server/domain/saved_objects/global_state/constants.ts
@@ -10,14 +10,14 @@ import { z } from '@kbn/zod/v4';
 export const DEFAULT_HISTORY_SNAPSHOT_FREQUENCY = '24h';

 export const LOG_EXTRACTION_DELAY_DEFAULT = '1m';
-export const LOG_EXTRACTION_LOOKBACK_PERIOD_DEFAULT = '3h';
+export const LOG_EXTRACTION_LOOKBACK_PERIOD_DEFAULT = '30d';
 export const LOG_EXTRACTION_FREQUENCY_DEFAULT = '1m';
 // Max amount of entities to extract in one ESQL query
 export const LOG_EXTRACTION_DOCS_LIMIT_DEFAULT = 10000;
 // Max raw log documents per logs to be processed in a query (inside elastic search)
 export const LOG_EXTRACTION_MAX_LOGS_PER_PAGE_DEFAULT = 40000;
 export const LOG_EXTRACTION_TIMEOUT_DEFAULT = '59s';
-export const LOG_EXTRACTION_MAX_TIME_WINDOW_SIZE_DEFAULT = '15m';
+export const LOG_EXTRACTION_MAX_TIME_WINDOW_SIZE_DEFAULT = '1d';
 // Max total raw log documents to process per task run; 0 = no cap
```

2. Start ES and Kibana with the following feature flags:

```
uiSettings.overrides:
  securitySolution:entityStoreEnableV2: true

xpack.securitySolution.enableExperimental:
  - entityAnalyticsEntityStoreV2
  - entityAnalyticsWatchlistEnabled
  - entityAnalyticsNewHomePageEnabled
  - leadGenerationEnabled
  - entityAnalyticsMlJobBehaviorMaintainer    ---->> !!! NEW FEATURE FLAG FOR THIS PR !!!
```

3. Use this script to populate some data:
https://gist.github.com/ymao1/d35d356f090e23c746055446cc21fba0. NOTE!!:
You may need to modify the Kibana URL if you're using a different base
path or SSL

You will need to also download these scripts that are referenced by the
above script.
- Rare region data:
https://gist.github.com/ymao1/3f8d1214928b5c27aa505a20b7f2425d
- High login count:
https://gist.github.com/ymao1/fbbdbcf7552455fd155ee52ffcddf67a

4. Verify the maintainer is started in Dev Tools

```
GET kbn:/internal/security/entity_store/entity_maintainers?apiVersion=2
```

Response should include the new `ml-anomaly-detection-jobs` maintainer
and the status should be `started`

```
{
  "maintainers": [
    {
      "id": "ml-anomaly-detection-jobs",
      "taskStatus": "started",
      "interval": "1d",
      "description": "Entity Analytics ML Anomaly Detection Maintainer",
      "nextRunAt": "2026-05-19T12:30:27.957Z",
      "minLicense": "platinum",
      "customState": {},
      "runs": 1,
      "lastSuccessTimestamp": "2026-05-18T12:30:30.117Z",
      "lastErrorTimestamp": null
    },
  ]
}
```

5. Manually run the maintainer

```
POST kbn:/internal/security/entity_store/entity_maintainers/run/ml-anomaly-detection-jobs?apiVersion=2
```

You should see this info log when the maintainer is done:

```
[2026-05-19T17:45:00.929-04:00][INFO ][plugins.securitySolution.ml-anomaly-detection-jobs-default] Maintainer run completed in 2570ms
```

6. After the maintainer runs, you should see some entities populated
with behavior data

```
GET .entities.v2.latest.security_default-00001/_search
{
    "query": {
        "bool": {
            "filter": [
              {
                "exists": {
                  "field": "entity.behaviors.anomaly_job_ids"
                }
              }
            ]
        }
    }
}
```

and you should see entries in the details index

```
GET .entity_analytics.ml-ad-jobs-latest-default/_search
```

---------

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

Author: ymao1

.buildkite/ftr-manifests/ftr_security_serverless_configs.yml

@@ -129,6 +129,7 @@ enabled:
   - x-pack/solutions/security/test/security_solution_api_integration/test_suites/genai/conversations/trial_license_complete_tier/configs/serverless.config.ts
   - x-pack/solutions/security/test/security_solution_api_integration/test_suites/entity_analytics/risk_engine/trial_license_complete_tier/configs/serverless.config.ts
   - x-pack/solutions/security/test/security_solution_api_integration/test_suites/entity_analytics/risk_score_maintainer/trial_license_complete_tier/configs/serverless.config.ts
+  - x-pack/solutions/security/test/security_solution_api_integration/test_suites/entity_analytics/ml_anomaly_detection_maintainer/trial_license_complete_tier/configs/serverless.config.ts
   - x-pack/solutions/security/test/security_solution_api_integration/test_suites/entity_analytics/entity_resolution/trial_license_complete_tier/configs/serverless.config.ts
   - x-pack/solutions/security/test/security_solution_api_integration/test_suites/entity_analytics/risk_engine/basic_license_essentials_tier/configs/serverless.config.ts
   - x-pack/solutions/security/test/security_solution_api_integration/test_suites/entity_analytics/entity_store/trial_license_complete_tier/configs/serverless.config.ts

.buildkite/ftr-manifests/ftr_security_stateful_configs.yml

@@ -100,6 +100,7 @@ enabled:
   - x-pack/solutions/security/test/security_solution_api_integration/test_suites/genai/conversations/trial_license_complete_tier/configs/ess.config.ts
   - x-pack/solutions/security/test/security_solution_api_integration/test_suites/entity_analytics/risk_engine/trial_license_complete_tier/configs/ess.config.ts
   - x-pack/solutions/security/test/security_solution_api_integration/test_suites/entity_analytics/risk_score_maintainer/trial_license_complete_tier/configs/ess.config.ts
+  - x-pack/solutions/security/test/security_solution_api_integration/test_suites/entity_analytics/ml_anomaly_detection_maintainer/trial_license_complete_tier/configs/ess.config.ts
   - x-pack/solutions/security/test/security_solution_api_integration/test_suites/entity_analytics/entity_resolution/trial_license_complete_tier/configs/ess.config.ts
   - x-pack/solutions/security/test/security_solution_api_integration/test_suites/entity_analytics/risk_engine/basic_license_essentials_tier/configs/ess.config.ts
   - x-pack/solutions/security/test/security_solution_api_integration/test_suites/entity_analytics/entity_store/trial_license_complete_tier/configs/ess.config.ts

x-pack/solutions/security/plugins/security_solution/common/experimental_features.ts

@@ -246,6 +246,11 @@ export const allowedExperimentalValues = Object.freeze({
    */
   entityAnalyticsEntityStoreV2: true,
 
+  /**
+   * Enables entity ML job behavior maintainer
+   */
+  entityAnalyticsMlJobBehaviorMaintainer: false,
+
   /**
    * Enables the deprecated prebuilt rules UI
    * Release: 9.4

x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/maintainers/behaviors/ml_anomaly_detection/__snapshots__/fetch_baseline_behavior.test.ts.snap

@@ -0,0 +1,35 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+export const ML_AD_MAINTAINER_ID = 'ml-anomaly-detection-jobs';
+export const ML_AD_MAINTAINER_INTERVAL = '1d';
+export const ML_AD_MAINTAINER_TIMEOUT = '10m';
+
+export const ML_AD_JOB_ENTITY_TYPES = ['user', 'host'] as const;
+
+// Window of anomaly records to inspect each run.
+export const ML_AD_LOOKBACK = '90d';
+
+// Safety check to prevent infinite loops in maintainer run
+export const MAX_ALLOWED_ITERS = 10000;
+
+// Page size when iterating entities from the entity store.
+export const ENTITY_PAGE_SIZE = 200;
+
+// Page size for paginating anomaly search results.
+export const ANOMALY_SEARCH_PAGE_SIZE = 1000;
+
+// Number of source documents to capture
+export const TOP_SOURCE_HITS = 3;
+
+// Number of baseline buckets to retain per anomaly
+export const BASELINE_BUCKET_SIZE = 3;
+
+export const ML_AD_DETAILS_INDEX_BASE = '.entity_analytics.ml-ad-jobs-latest';
+
+export const getMlAdDetailsIndexName = (namespace: string): string =>
+  `${ML_AD_DETAILS_INDEX_BASE}-${namespace}`;

x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/maintainers/behaviors/ml_anomaly_detection/constants.ts

@@ -0,0 +1,86 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+import { elasticsearchServiceMock, loggingSystemMock } from '@kbn/core/server/mocks';
+import { ensureMlAdDetailsDataStream, ML_AD_DETAILS_MAPPING } from './details_index';
+import { getMlAdDetailsIndexName } from './constants';
+
+const esClient = elasticsearchServiceMock.createScopedClusterClient().asCurrentUser;
+let logger: ReturnType<typeof loggingSystemMock.createLogger>;
+
+describe('ensureMlAdDetailsDataStream', () => {
+  beforeEach(() => {
+    jest.clearAllMocks();
+    logger = loggingSystemMock.createLogger();
+  });
+
+  it('creates the index template and data stream when it does not exist', async () => {
+    esClient.indices.putIndexTemplate.mockResolvedValue({} as never);
+    esClient.indices.createDataStream.mockResolvedValue({} as never);
+
+    const dataStream = getMlAdDetailsIndexName('default');
+    const result = await ensureMlAdDetailsDataStream({ esClient, logger, namespace: 'default' });
+
+    expect(result).toBe(dataStream);
+    expect(esClient.indices.putIndexTemplate).toHaveBeenCalledWith(
+      expect.objectContaining({
+        index_patterns: [expect.stringContaining('ml-ad-jobs-latest')],
+        data_stream: {},
+        template: expect.objectContaining({
+          mappings: ML_AD_DETAILS_MAPPING,
+          lifecycle: { data_retention: '90d' },
+        }),
+      })
+    );
+    expect(esClient.indices.createDataStream).toHaveBeenCalledWith({ name: dataStream });
+  });
+
+  it('uses the namespace to build the data stream name', async () => {
+    esClient.indices.putIndexTemplate.mockResolvedValue({} as never);
+    esClient.indices.createDataStream.mockResolvedValue({} as never);
+
+    const result = await ensureMlAdDetailsDataStream({ esClient, logger, namespace: 'my-space' });
+
+    expect(result).toBe(getMlAdDetailsIndexName('my-space'));
+    expect(esClient.indices.createDataStream).toHaveBeenCalledWith({
+      name: getMlAdDetailsIndexName('my-space'),
+    });
+  });
+
+  describe('error handling', () => {
+    it('swallows resource_already_exists_exception from concurrent creation', async () => {
+      esClient.indices.putIndexTemplate.mockResolvedValue({} as never);
+      esClient.indices.createDataStream.mockRejectedValue(
+        new Error('resource_already_exists_exception: data_stream already exists')
+      );
+
+      const result = await ensureMlAdDetailsDataStream({ esClient, logger, namespace: 'default' });
+
+      expect(result).toBe(getMlAdDetailsIndexName('default'));
+      expect(logger.warn).not.toHaveBeenCalled();
+    });
+
+    it('logs and swallows template creation errors', async () => {
+      esClient.indices.putIndexTemplate.mockRejectedValue(new Error('cluster_block_exception'));
+
+      const result = await ensureMlAdDetailsDataStream({ esClient, logger, namespace: 'default' });
+
+      expect(result).toBe(getMlAdDetailsIndexName('default'));
+      expect(logger.warn).toHaveBeenCalledWith(expect.stringContaining('cluster_block_exception'));
+    });
+
+    it('logs and swallows unexpected data stream creation errors', async () => {
+      esClient.indices.putIndexTemplate.mockResolvedValue({} as never);
+      esClient.indices.createDataStream.mockRejectedValue(new Error('cluster_block_exception'));
+
+      const result = await ensureMlAdDetailsDataStream({ esClient, logger, namespace: 'default' });
+
+      expect(result).toBe(getMlAdDetailsIndexName('default'));
+      expect(logger.warn).toHaveBeenCalledWith(expect.stringContaining('cluster_block_exception'));
+    });
+  });
+});