Skip to content

Commit 5fb23c3

Browse files
maximpnnickpeihl
maximpn
authored and
nickpeihl
committed
[Security Solution] Add FTR tests for prebuilt rules OOM testing (elastic#236891)
**Partially addresses:** elastic#188090 ## Summary This PR contains FTR tests designed to expose potential Out of Memory (OOM) issues in Kibana when performing memory-intensive operations related to **Detection Prebuilt Rules**. The tests open a possibility to test against any Prebuilt Rules fleet package version including prerelease packages. ## Running test instructions - Set up QAF by following to the [instructions](https://docs.elastic.dev/appex-qa/qaf/getting-started) (internal) - Place the following Elastic Cloud plan in `~/.qaf/config/cloud_plans/prebuilt_rules_oom_testing.yml` ```yaml --- name: {{ deployment_name }} settings: autoscaling_enabled: {{ autoscaling_enabled }} metadata: system_owned: false resources: elasticsearch: - region: {{ region }} settings: dedicated_masters_threshold: 6 plan: cluster_topology: - zone_count: 1 elasticsearch: node_attributes: data: hot instance_configuration_id: gcp.es.datahot.n2.68x10x45 node_roles: - master - ingest - remote_cluster_client - data_hot - transform - data_content id: hot_content size: value: 1024 resource: memory elasticsearch: version: {{ stack_version }} deployment_template: id: gcp-storage-optimized ref_id: main-elasticsearch enterprise_search: [] kibana: - elasticsearch_cluster_ref_id: main-elasticsearch region: {{ region }} plan: cluster_topology: - instance_configuration_id: gcp.kibana.n2.68x32x45 zone_count: 1 size: value: 1024 resource: memory kibana: version: {{ stack_version }} ref_id: main-kibana ``` - Create an ECH deployment by running the following command ```bash qaf elastic-cloud deployments create --stack-version 9.3.0 --version-validation --deployment-name prebuilt-rules-oom-test-9.3.0 --environment production --no-autoscaling --no-sso --region gcp-us-west2 --plan prebuilt_rules_oom_testing ``` - Run the tests by running the following command ```bash qaf kibana ftr run-config --ec-deployment-name prebuilt-rules-oom-test-9.3.0 --kibana-repo-root <kibana-root> <kibana-root>/x-pack/solutions/security/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/oom_testing/configs/ess_basic_license.config.ts ``` where `<kibana-root>` is the absolute path to the Kibana's root folder.
1 parent 9e7fb91 commit 5fb23c3

6 files changed

Lines changed: 276 additions & 0 deletions

File tree

‎.buildkite/ftr_security_stateful_configs.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,9 @@ disabled:
3636
- x-pack/solutions/security/test/security_solution_api_integration/test_suites/detections_response/rules_management/configs/ess/rules_management.basic.config.ts
3737
- x-pack/solutions/security/test/security_solution_api_integration/test_suites/detections_response/rules_management/configs/ess/rules_management.trial.config.ts
3838

39+
# Detection Rules Management OOM Testing
40+
- x-pack/solutions/security/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/oom_testing/configs/ess_basic_license.config.ts
41+
3942
defaultQueue: 'n2-4-spot'
4043
enabled:
4144
- x-pack/solutions/security/test/security_solution_api_integration/test_suites/detections_response/detection_engine/actions/trial_license_complete_tier/configs/ess.config.ts

‎x-pack/solutions/security/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/oom_testing/README.md‎

Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
## Kibana Security Solution Prebuilt Rules OOM Testing
2+
3+
This folder contains **FTR tests** (Functional Test Runner) designed to expose potential Out of Memory (OOM) issues in Kibana when performing memory-intensive operations related to **Security Solution Prebuilt Rules**.
4+
5+
The tests focus on identifying prebuilt rules package size maximum limit, potential memory leaks or excessive memory usage during operations such as:
6+
7+
- Installing or upgrading prebuilt rule packages
8+
9+
- Installing or upgrading prebuilt rules by using prebuilt rule assets in the package
10+
11+
- Performing related bulk actions
12+
13+
### 💡 Purpose
14+
15+
The goal of these tests is to proactively detect memory exhaustion scenarios by simulating real-world conditions where Kibana operates under constrained memory environments. This helps improve the resilience and stability of the Security Solution features.
16+
17+
### ⚙️ Test Environment Setup
18+
19+
To effectively reproduce OOM-related behavior, the deployment should be created in Elastic Cloud with 1GB RAM limit for the Kibana instance. Elasticsearch instance isn't so important in that testing a reasonable 1GB RAM instance provide sufficient performance for the testing. ML and Integration instances as well as cold and frozen tier Elasticsearch nodes aren't required. An example Elastic Cloud configuration applicable for internal testing framework QAF (QA Framework) looks like the following
20+
21+
```yaml
22+
---
23+
name: '{{ deployment_name }}'
24+
settings:
25+
autoscaling_enabled: '{{ autoscaling_enabled }}'
26+
metadata:
27+
system_owned: false
28+
resources:
29+
elasticsearch:
30+
- region: '{{ region }}'
31+
settings:
32+
dedicated_masters_threshold: 6
33+
plan:
34+
cluster_topology:
35+
- zone_count: 1
36+
elasticsearch:
37+
node_attributes:
38+
data: hot
39+
instance_configuration_id: gcp.es.datahot.n2.68x10x45
40+
node_roles:
41+
- master
42+
- ingest
43+
- remote_cluster_client
44+
- data_hot
45+
- transform
46+
- data_content
47+
id: hot_content
48+
size:
49+
value: 1024
50+
resource: memory
51+
elasticsearch:
52+
version: '{{ stack_version }}'
53+
deployment_template:
54+
id: gcp-storage-optimized
55+
ref_id: main-elasticsearch
56+
enterprise_search: []
57+
kibana:
58+
- elasticsearch_cluster_ref_id: main-elasticsearch
59+
region: '{{ region }}'
60+
plan:
61+
cluster_topology:
62+
- instance_configuration_id: gcp.kibana.n2.68x32x45
63+
zone_count: 1
64+
size:
65+
value: 1024
66+
resource: memory
67+
kibana:
68+
version: '{{ stack_version }}'
69+
user_settings_yaml: |-
70+
xpack.securitySolution.prebuiltRulesPackageVersion: '<package-version>'
71+
ref_id: main-kibana
72+
```
73+
74+
Where `<package-version>` is the package to be installed, e.g. `8.17.4`.
75+
76+
With QAF the config provided above should be placed in `~/.qaf/config/cloud_plans/prebuilt_rules_oom_testing.yml`.
77+
78+
> **_NOTE:_** Make sure to use Cloud First Testing (CFT) regions when creating the testing deployment as these regions support Kibana configuration options like `xpack.securitySolution.prebuiltRulesPackageVersion` or `xpack.fleet.registryUrl` otherwise the choice of configuration options will be restricted to those listed on the [General settings in Kibana page](https://www.elastic.co/docs/reference/kibana/configuration-reference/general-settings) and having the "C" icon. The CFT regions are `gcp-us-west2` and `aws-eu-west-1`.
79+
80+
### 🧪 Running the Tests
81+
82+
#### Locally
83+
84+
The easiest way to run the tests locally is using our internal testing framework QAF. Please follow the steps to prepare the environment
85+
86+
- Set up QAF by following to the [instructions](https://docs.elastic.dev/appex-qa/qaf/getting-started) (internal)
87+
- Place the plan configuration provided in [Test Environment Setup](#⚙️-test-environment-setup) in `~/.qaf/config/cloud_plans/prebuilt_rules_oom_testing.yml`
88+
- Create an ECH deployment by running the following command
89+
90+
```bash
91+
qaf elastic-cloud deployments create --stack-version <stack-version> --version-validation --deployment-name <deployment-name> --environment production --no-autoscaling --no-sso --region gcp-us-west2 --plan prebuilt_rules_oom_testing
92+
```
93+
94+
where `<stack-version>` specifies which Elastic Stack version should be deployed (unreleased versions will be deployed from snapshots as a **-SNAPSHOT** version) and `<deployment-name>` specifies a deployment alias required later on to run the tests.
95+
96+
For example a command to create `9.2` Elastic Stack would be
97+
98+
```bash
99+
qaf elastic-cloud deployments create --stack-version 9.2.0 --version-validation --deployment-name prebuilt-rules-oom-test-9.2.0 --environment production --no-autoscaling --no-sso --region gcp-us-west2 --plan prebuilt_rules_oom_testing
100+
```
101+
102+
- Run the tests by running the following command
103+
104+
```bash
105+
qaf kibana ftr run-config --ec-deployment-name <deployment-name> --kibana-repo-root <kibana-root> <kibana-root>/x-pack/solutions/security/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/oom_testing/configs/ess_basic_license.config.ts
106+
```
107+
108+
where `<kibana-root>` is the absolute path to the Kibana's root folder and `<deployment-name>` is the deployment name used in `qaf elastic-cloud deployments create` to create an Elastic Stack deployment.
109+
110+
### 📁 Test Structure
111+
112+
The tests in this folder include scenarios that:
113+
114+
- Perform repeated or concurrent installations of prebuilt rules
115+
116+
- Trigger large API requests to heavy endpoints (e.g., rule package installation endpoints)
117+
118+
- Each test is expected to either succeed within the memory limit or trigger a recoverable failure. An unexpected crash or unhandled OOM error indicates a potential memory issue.
119+
120+
### ✅ Expected Outcomes
121+
122+
- **Pass**: Kibana completes all operations without OOM crashes.
123+
124+
- **Fail**: Kibana terminates with a fatal OOM error or becomes unresponsive.

‎x-pack/solutions/security/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/oom_testing/configs/ess_basic_license.config.ts‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
/*
2+
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
3+
* or more contributor license agreements. Licensed under the Elastic License
4+
* 2.0; you may not use this file except in compliance with the Elastic License
5+
* 2.0.
6+
*/
7+
8+
import type { FtrConfigProviderContext } from '@kbn/test';
9+
10+
export default async function ({ readConfigFile }: FtrConfigProviderContext) {
11+
const functionalConfig = await readConfigFile(
12+
require.resolve('../../../configs/ess/rules_management.basic.config')
13+
);
14+
15+
const testConfig = {
16+
...functionalConfig.getAll(),
17+
testFiles: [require.resolve('..')],
18+
junit: {
19+
reportName: 'Rules Management - Prebuilt Rules OOM Testing - ESS Basic License',
20+
},
21+
};
22+
23+
return testConfig;
24+
}

‎x-pack/solutions/security/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/oom_testing/index.ts‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
/*
2+
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
3+
* or more contributor license agreements. Licensed under the Elastic License
4+
* 2.0; you may not use this file except in compliance with the Elastic License
5+
* 2.0.
6+
*/
7+
8+
import type { FtrProviderContext } from '../../../../../ftr_provider_context';
9+
10+
export default ({ loadTestFile }: FtrProviderContext): void => {
11+
describe('Rules Management - Prebuilt Rules OOM Testing - ESS Basic License', function () {
12+
this.tags('skipFIPS');
13+
loadTestFile(require.resolve('./install_prebuilt_rules'));
14+
});
15+
};

‎x-pack/solutions/security/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/oom_testing/install_prebuilt_rules/index.ts‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
/*
2+
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
3+
* or more contributor license agreements. Licensed under the Elastic License
4+
* 2.0; you may not use this file except in compliance with the Elastic License
5+
* 2.0.
6+
*/
7+
8+
import type { FtrProviderContext } from '../../../../../../ftr_provider_context';
9+
10+
export default ({ loadTestFile }: FtrProviderContext): void => {
11+
loadTestFile(require.resolve('./install_prebuilt_rules'));
12+
};

‎x-pack/solutions/security/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/oom_testing/install_prebuilt_rules/install_prebuilt_rules.ts‎

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
/*
2+
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
3+
* or more contributor license agreements. Licensed under the Elastic License
4+
* 2.0; you may not use this file except in compliance with the Elastic License
5+
* 2.0.
6+
*/
7+
8+
import expect from 'expect';
9+
import {
10+
ELASTIC_HTTP_VERSION_HEADER,
11+
X_ELASTIC_INTERNAL_ORIGIN_REQUEST,
12+
} from '@kbn/core-http-common';
13+
import {
14+
ENDPOINT_PACKAGE_NAME,
15+
PREBUILT_RULES_PACKAGE_NAME,
16+
} from '@kbn/security-solution-plugin/common/detection_engine/constants';
17+
import {
18+
GET_PREBUILT_RULES_STATUS_URL,
19+
PERFORM_RULE_INSTALLATION_URL,
20+
REVIEW_RULE_INSTALLATION_URL,
21+
} from '@kbn/security-solution-plugin/common/api/detection_engine';
22+
import type { FtrProviderContext } from '../../../../../../ftr_provider_context';
23+
import {
24+
deleteEndpointFleetPackage,
25+
deletePrebuiltRulesFleetPackage,
26+
} from '../../../../utils/rules/prebuilt_rules/delete_fleet_packages';
27+
import { deleteAllRules, waitFor } from '../../../../../../config/services/detections_response';
28+
29+
export default ({ getService }: FtrProviderContext): void => {
30+
const es = getService('es');
31+
const supertest = getService('supertest');
32+
const log = getService('log');
33+
const retryService = getService('retry');
34+
const detectionsApi = getService('detectionsApi');
35+
36+
describe('@ess @serverless @skipInServerlessMKI Install from mocked prebuilt rule assets', () => {
37+
beforeEach(async () => {
38+
await deleteAllRules(supertest, log);
39+
40+
await deletePrebuiltRulesFleetPackage({ supertest, es, log, retryService });
41+
await deleteEndpointFleetPackage({ supertest, es, log, retryService });
42+
43+
await waitFor(
44+
async () => {
45+
const { body: prebuiltRulesStatus } = await supertest
46+
.get(GET_PREBUILT_RULES_STATUS_URL)
47+
.set(ELASTIC_HTTP_VERSION_HEADER, '1')
48+
.set(X_ELASTIC_INTERNAL_ORIGIN_REQUEST, 'kibana')
49+
.expect(200);
50+
51+
return (
52+
prebuiltRulesStatus.stats.num_prebuilt_rules_installed === 0 &&
53+
prebuiltRulesStatus.stats.num_prebuilt_rules_to_install === 0
54+
);
55+
},
56+
'waitForIndexesRefreshAfterPackagesDeletion',
57+
log
58+
);
59+
});
60+
61+
it('install prebuilt rules from a package', async () => {
62+
const { body: bootstrapPrebuiltRulesResponse } = await detectionsApi
63+
.bootstrapPrebuiltRules()
64+
.expect(200);
65+
66+
expect(bootstrapPrebuiltRulesResponse).toMatchObject({
67+
packages: expect.arrayContaining([
68+
expect.objectContaining({
69+
name: PREBUILT_RULES_PACKAGE_NAME,
70+
}),
71+
expect.objectContaining({
72+
name: ENDPOINT_PACKAGE_NAME,
73+
}),
74+
]),
75+
});
76+
77+
const { body: reviewPrebuiltRulesForInstallationResponse } = await supertest
78+
.post(REVIEW_RULE_INSTALLATION_URL)
79+
.set('kbn-xsrf', 'true')
80+
.set(ELASTIC_HTTP_VERSION_HEADER, '1')
81+
.set(X_ELASTIC_INTERNAL_ORIGIN_REQUEST, 'kibana')
82+
.send()
83+
.expect(200);
84+
85+
expect(reviewPrebuiltRulesForInstallationResponse.rules.length).toBeGreaterThan(0);
86+
87+
const { body: installPrebuiltRulesResponse } = await supertest
88+
.post(PERFORM_RULE_INSTALLATION_URL)
89+
.set('kbn-xsrf', 'true')
90+
.set(ELASTIC_HTTP_VERSION_HEADER, '1')
91+
.set(X_ELASTIC_INTERNAL_ORIGIN_REQUEST, 'kibana')
92+
.send({ mode: 'ALL_RULES' })
93+
.expect(200);
94+
95+
expect(installPrebuiltRulesResponse.summary.succeeded).toBeGreaterThan(0);
96+
});
97+
});
98+
};

0 commit comments

Comments
 (0)