Skip to content

Commit 983d331

Browse files
shahzad31kibanamachine
shahzad31
and
kibanamachine
authored
[Synthetics FTR] Reduce Fleet churn by making installSyntheticsPackage idempotent (elastic#264281)
## Summary Reduces the Fleet work the synthetics deployment-agnostic FTR suite does per CI run from ~25 full package install cycles down to **1**. The excessive churn was the primary source of 502 / "backend closed connection" flakes in downstream tests (Buildkite [build 1032](https://buildkite.com/elastic/appex-qa-stateful-kibana-ftr-tests/builds/1032/steps/canvas?sid=019da341-0cae-45f9-bca0-cda97e93857e)). ### Changes 1. **`installSyntheticsPackage()` is now idempotent** (`synthetics_private_location.ts`). - `POST /api/fleet/setup` runs once per FTR run (not once per `describe`). - The DELETE+POST reinstall cycle is skipped when Fleet already reports the package installed at the requested version. - A module-level cache (`fleetSetupDone`, `installedVersionCache`) enforces the short-circuit across suites. - An exported `resetInstallSyntheticsPackageCache()` is provided for the one test that legitimately mutates deployment state (auto-upgrade). 2. **New `cleanSyntheticsTestData()` helper** replaces `kibanaServer.savedObjects.cleanStandardList()` in every synthetics deployment-agnostic suite. - The shared `STANDARD_LIST_TYPES` wipes `epm-packages` / `epm-packages-assets` (Fleet's install record) plus the Kibana assets the synthetics package itself installs — which invalidated the idempotency cache between suites and would otherwise force every `before` hook back into a full reinstall. - The new helper is an **explicit allow-list** of the SO types synthetics tests actually create (`synthetics-monitor*`, `synthetics-private-location*`, `synthetics-param`, `uptime-dynamic-settings`, Fleet policies, `alert`/`action`). Kibana assets owned by the synthetics Fleet package stay intact → no dangling-reference failures. - No change to shared kbn-kbn-client code. ### Local validation Ran a representative subset twice (`EnableDefaultAlerting | inspectSyntheticsMonitor | CleanUpExtraPackagePolicies | GetPrivateLocationMonitors | EditPrivateLocationAPI`) against a locally-started ES + Kibana via FTR, using a `SYNTHETICS_LEGACY_INSTALL=1` env toggle (removed before commit) to compare side-by-side: | mode | `/api/fleet/setup` calls | full install cycles | tests | wall time | | --- | ---: | ---: | --- | --- | | legacy (pre-PR behavior) | 6 | 6 | **33 passing** | 1.7 min | | idempotent (this PR) | **1** | **1** | **33 passing** | **1.0 min** | Same test result, ~6× less Fleet churn, and faster too. ### Out of scope The transient retry helper that was briefly drafted here was removed at the reviewer's request — it would have suppressed the symptom rather than addressed the root cause. ## Test plan - [x] Local FTR subset passes with and without the PR behavior (see table above). - [ ] Full `oblt.stateful.config.ts` run on CI. - [ ] Watch `appex-qa-stateful-kibana-ftr-tests` next run for reduction in 502s across non-synthetics suites. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
1 parent 1d39475 commit 983d331

29 files changed

Lines changed: 284 additions & 79 deletions

‎x-pack/solutions/observability/test/api_integration_deployment_agnostic/apis/synthetics/clean_up_extra_package_policies.ts‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,10 @@ import expect from '@kbn/expect';
1010
import { syntheticsMonitorSavedObjectType } from '@kbn/synthetics-plugin/common/types/saved_objects';
1111
import type { DeploymentAgnosticFtrProviderContext } from '../../ftr_provider_context';
1212
import { getFixtureJson } from './helpers/get_fixture_json';
13-
import { PrivateLocationTestService } from '../../services/synthetics_private_location';
13+
import {
14+
PrivateLocationTestService,
15+
cleanSyntheticsTestData,
16+
} from '../../services/synthetics_private_location';
1417
import { SyntheticsMonitorTestService } from '../../services/synthetics_monitor';
1518

1619
export default function ({ getService }: DeploymentAgnosticFtrProviderContext) {
@@ -30,7 +33,7 @@ export default function ({ getService }: DeploymentAgnosticFtrProviderContext) {
3033
const testPrivateLocations = new PrivateLocationTestService(getService);
3134

3235
before(async () => {
33-
await kibanaServer.savedObjects.cleanStandardList();
36+
await cleanSyntheticsTestData(kibanaServer);
3437
await testPrivateLocations.installSyntheticsPackage();
3538
editorUser = await samlAuth.createM2mApiKeyWithRoleScope('editor');
3639

‎x-pack/solutions/observability/test/api_integration_deployment_agnostic/apis/synthetics/create_monitor.ts‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,10 @@ import {
2020
import type { DeploymentAgnosticFtrProviderContext } from '../../ftr_provider_context';
2121
import { getFixtureJson } from './helpers/get_fixture_json';
2222
import { SyntheticsMonitorTestService } from '../../services/synthetics_monitor';
23-
import { PrivateLocationTestService } from '../../services/synthetics_private_location';
23+
import {
24+
PrivateLocationTestService,
25+
cleanSyntheticsTestData,
26+
} from '../../services/synthetics_private_location';
2427

2528
export const addMonitorAPIHelper = async (
2629
supertestAPI: any,
@@ -103,7 +106,7 @@ export default function ({ getService }: DeploymentAgnosticFtrProviderContext) {
103106

104107
before(async () => {
105108
_httpMonitorJson = getFixtureJson('http_monitor');
106-
await kibanaServer.savedObjects.cleanStandardList();
109+
await cleanSyntheticsTestData(kibanaServer);
107110
editorRoleAuthc = await samlAuth.createM2mApiKeyWithRoleScope('editor');
108111
});
109112

‎x-pack/solutions/observability/test/api_integration_deployment_agnostic/apis/synthetics/create_monitor_private_location.ts‎

Lines changed: 11 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,10 @@ import rawExpect from 'expect';
2424
import type { DeploymentAgnosticFtrProviderContext } from '../../ftr_provider_context';
2525
import { getFixtureJson } from './helpers/get_fixture_json';
2626
import { comparePolicies, getTestSyntheticsPolicy } from './sample_data/test_policy';
27-
import { PrivateLocationTestService } from '../../services/synthetics_private_location';
27+
import {
28+
PrivateLocationTestService,
29+
cleanSyntheticsTestData,
30+
} from '../../services/synthetics_private_location';
2831
import { addMonitorAPIHelper, keyToOmitList, omitMonitorKeys } from './create_monitor';
2932
import { SyntheticsMonitorTestService } from '../../services/synthetics_monitor';
3033

@@ -87,7 +90,7 @@ export default function ({ getService }: DeploymentAgnosticFtrProviderContext) {
8790
};
8891

8992
before(async () => {
90-
await kibanaServer.savedObjects.cleanStandardList();
93+
await cleanSyntheticsTestData(kibanaServer);
9194
await testPrivateLocations.installSyntheticsPackage();
9295
editorUser = await samlAuth.createM2mApiKeyWithRoleScope('editor');
9396

@@ -498,7 +501,10 @@ export default function ({ getService }: DeploymentAgnosticFtrProviderContext) {
498501
this.skip();
499502
}
500503

501-
await testPrivateLocations.installSyntheticsPackage({ version: lowerVersion });
504+
await testPrivateLocations.installSyntheticsPackage({
505+
version: lowerVersion,
506+
force: true,
507+
});
502508
let monitorId = '';
503509
const privateLocation = await testPrivateLocations.addTestPrivateLocation();
504510

@@ -530,7 +536,7 @@ export default function ({ getService }: DeploymentAgnosticFtrProviderContext) {
530536

531537
expect(packagePolicy.package.version).eql(lowerVersion);
532538

533-
await testPrivateLocations.installSyntheticsPackage();
539+
await testPrivateLocations.installSyntheticsPackage({ force: true });
534540

535541
await retry.tryForTime(120 * 1000, async () => {
536542
const policyResponseAfterUpgrade = await supertestWithAuth.get(
@@ -552,7 +558,7 @@ export default function ({ getService }: DeploymentAgnosticFtrProviderContext) {
552558
}
553559
// Restore the package to the latest version — this MUST succeed
554560
// or subsequent tests will run against the wrong version
555-
await testPrivateLocations.installSyntheticsPackage();
561+
await testPrivateLocations.installSyntheticsPackage({ force: true });
556562
}
557563
});
558564

‎x-pack/solutions/observability/test/api_integration_deployment_agnostic/apis/synthetics/create_monitor_project.ts‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,10 @@ import {
2424
} from '@kbn/synthetics-plugin/common/constants/monitor_defaults';
2525
import type { DeploymentAgnosticFtrProviderContext } from '../../ftr_provider_context';
2626
import { getFixtureJson } from './helpers/get_fixture_json';
27-
import { PrivateLocationTestService } from '../../services/synthetics_private_location';
27+
import {
28+
PrivateLocationTestService,
29+
cleanSyntheticsTestData,
30+
} from '../../services/synthetics_private_location';
2831
import { SyntheticsMonitorTestService } from '../../services/synthetics_monitor';
2932
import { LOCAL_PUBLIC_LOCATION } from './helpers/location';
3033

@@ -81,7 +84,7 @@ export default function ({ getService }: DeploymentAgnosticFtrProviderContext) {
8184
};
8285

8386
before(async () => {
84-
await kibanaServer.savedObjects.cleanStandardList();
87+
await cleanSyntheticsTestData(kibanaServer);
8588
editorUser = await samlAuth.createM2mApiKeyWithRoleScope('editor');
8689
await supertest
8790
.put(SYNTHETICS_API_URLS.SYNTHETICS_ENABLEMENT)

‎x-pack/solutions/observability/test/api_integration_deployment_agnostic/apis/synthetics/create_monitor_project_multi_space.ts‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,10 @@ import {
1414
import { SYNTHETICS_API_URLS } from '@kbn/synthetics-plugin/common/constants';
1515
import type { DeploymentAgnosticFtrProviderContext } from '../../ftr_provider_context';
1616
import { getFixtureJson } from './helpers/get_fixture_json';
17-
import { PrivateLocationTestService } from '../../services/synthetics_private_location';
17+
import {
18+
PrivateLocationTestService,
19+
cleanSyntheticsTestData,
20+
} from '../../services/synthetics_private_location';
1821

1922
export default function ({ getService }: DeploymentAgnosticFtrProviderContext) {
2023
describe('CreateProjectMonitorsMultiSpace', function () {
@@ -27,7 +30,7 @@ export default function ({ getService }: DeploymentAgnosticFtrProviderContext) {
2730
let editorUser: RoleCredentials;
2831

2932
before(async () => {
30-
await kibanaServer.savedObjects.cleanStandardList();
33+
await cleanSyntheticsTestData(kibanaServer);
3134
editorUser = await samlAuth.createM2mApiKeyWithRoleScope('editor');
3235
await supertest
3336
.put(SYNTHETICS_API_URLS.SYNTHETICS_ENABLEMENT)

‎x-pack/solutions/observability/test/api_integration_deployment_agnostic/apis/synthetics/create_monitor_project_private_location.ts‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,10 @@ import {
2525
} from './sample_data/test_project_monitor_policy';
2626
import type { DeploymentAgnosticFtrProviderContext } from '../../ftr_provider_context';
2727
import { getFixtureJson } from './helpers/get_fixture_json';
28-
import { PrivateLocationTestService } from '../../services/synthetics_private_location';
28+
import {
29+
PrivateLocationTestService,
30+
cleanSyntheticsTestData,
31+
} from '../../services/synthetics_private_location';
2932
import { SyntheticsMonitorTestService } from '../../services/synthetics_monitor';
3033
import { comparePolicies } from './sample_data/test_policy';
3134

@@ -60,7 +63,7 @@ export default function ({ getService }: DeploymentAgnosticFtrProviderContext) {
6063
};
6164

6265
before(async () => {
63-
await kibanaServer.savedObjects.cleanStandardList();
66+
await cleanSyntheticsTestData(kibanaServer);
6467
editorUser = await samlAuth.createM2mApiKeyWithRoleScope('editor');
6568
viewerUser = await samlAuth.createM2mApiKeyWithRoleScope('viewer');
6669

@@ -87,7 +90,7 @@ export default function ({ getService }: DeploymentAgnosticFtrProviderContext) {
8790
});
8891

8992
after(async () => {
90-
await kibanaServer.savedObjects.cleanStandardList();
93+
await cleanSyntheticsTestData(kibanaServer);
9194
});
9295

9396
beforeEach(async () => {

‎x-pack/solutions/observability/test/api_integration_deployment_agnostic/apis/synthetics/create_monitor_public_api.ts‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ import type { RoleCredentials } from '@kbn/ftr-common-functional-services';
1010
import { DEFAULT_FIELDS } from '@kbn/synthetics-plugin/common/constants/monitor_defaults';
1111
import type { DeploymentAgnosticFtrProviderContext } from '../../ftr_provider_context';
1212
import { addMonitorAPIHelper, omitMonitorKeys } from './create_monitor';
13+
import { cleanSyntheticsTestData } from '../../services/synthetics_private_location';
1314
import { LOCAL_PUBLIC_LOCATION } from './helpers/location';
1415

1516
export default function ({ getService }: DeploymentAgnosticFtrProviderContext) {
@@ -25,12 +26,12 @@ export default function ({ getService }: DeploymentAgnosticFtrProviderContext) {
2526
}
2627

2728
before(async () => {
28-
await kibanaServer.savedObjects.cleanStandardList();
29+
await cleanSyntheticsTestData(kibanaServer);
2930
editorUser = await samlAuth.createM2mApiKeyWithRoleScope('editor');
3031
});
3132

3233
after(async () => {
33-
await kibanaServer.savedObjects.cleanStandardList();
34+
await cleanSyntheticsTestData(kibanaServer);
3435
});
3536

3637
describe('HTTP Monitor', () => {

‎x-pack/solutions/observability/test/api_integration_deployment_agnostic/apis/synthetics/create_monitor_public_api_private_location.ts‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,10 @@ import { DEFAULT_FIELDS } from '@kbn/synthetics-plugin/common/constants/monitor_
1313
import { LOCATION_REQUIRED_ERROR } from '@kbn/synthetics-plugin/server/routes/monitor_cruds/monitor_validation';
1414
import type { DeploymentAgnosticFtrProviderContext } from '../../ftr_provider_context';
1515
import { addMonitorAPIHelper, omitMonitorKeys } from './create_monitor';
16-
import { PrivateLocationTestService } from '../../services/synthetics_private_location';
16+
import {
17+
PrivateLocationTestService,
18+
cleanSyntheticsTestData,
19+
} from '../../services/synthetics_private_location';
1720

1821
export default function ({ getService }: DeploymentAgnosticFtrProviderContext) {
1922
describe('AddNewMonitorsPublicAPI - Private locations', function () {
@@ -29,13 +32,13 @@ export default function ({ getService }: DeploymentAgnosticFtrProviderContext) {
2932
}
3033

3134
before(async () => {
32-
await kibanaServer.savedObjects.cleanStandardList();
35+
await cleanSyntheticsTestData(kibanaServer);
3336
editorUser = await samlAuth.createM2mApiKeyWithRoleScope('editor');
3437
privateLocation = await privateLocationTestService.addTestPrivateLocation();
3538
});
3639

3740
after(async () => {
38-
await kibanaServer.savedObjects.cleanStandardList();
41+
await cleanSyntheticsTestData(kibanaServer);
3942
});
4043

4144
it('should return error for empty monitor', async function () {

‎x-pack/solutions/observability/test/api_integration_deployment_agnostic/apis/synthetics/delete_monitor.ts‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,10 @@ import { SYNTHETICS_API_URLS } from '@kbn/synthetics-plugin/common/constants';
1515
import expect from '@kbn/expect';
1616
import type { DeploymentAgnosticFtrProviderContext } from '../../ftr_provider_context';
1717
import { getFixtureJson } from './helpers/get_fixture_json';
18-
import { PrivateLocationTestService } from '../../services/synthetics_private_location';
18+
import {
19+
PrivateLocationTestService,
20+
cleanSyntheticsTestData,
21+
} from '../../services/synthetics_private_location';
1922
import { SyntheticsMonitorTestService } from '../../services/synthetics_monitor';
2023

2124
export default function ({ getService }: DeploymentAgnosticFtrProviderContext) {
@@ -61,7 +64,7 @@ export default function ({ getService }: DeploymentAgnosticFtrProviderContext) {
6164
};
6265

6366
before(async () => {
64-
await kibanaServer.savedObjects.cleanStandardList();
67+
await cleanSyntheticsTestData(kibanaServer);
6568
await testPrivateLocations.installSyntheticsPackage();
6669
const testPolicyName = 'Fleet test server policy' + Date.now();
6770
const apiResponse = await testPrivateLocations.addFleetPolicy(testPolicyName);

‎x-pack/solutions/observability/test/api_integration_deployment_agnostic/apis/synthetics/edit_monitor.ts‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ import type { DeploymentAgnosticFtrProviderContext } from '../../ftr_provider_co
2020
import { getFixtureJson } from './helpers/get_fixture_json';
2121
import { omitResponseTimestamps, omitEmptyValues } from './helpers/monitor';
2222
import { SyntheticsMonitorTestService } from '../../services/synthetics_monitor';
23+
import { cleanSyntheticsTestData } from '../../services/synthetics_private_location';
2324
import { LOCAL_PUBLIC_LOCATION } from './helpers/location';
2425

2526
export default function ({ getService }: DeploymentAgnosticFtrProviderContext) {
@@ -71,7 +72,7 @@ export default function ({ getService }: DeploymentAgnosticFtrProviderContext) {
7172
};
7273

7374
before(async () => {
74-
await kibanaServer.savedObjects.cleanStandardList();
75+
await cleanSyntheticsTestData(kibanaServer);
7576
await supertestWithAuth.post('/api/fleet/setup').set('kbn-xsrf', 'true').send().expect(200);
7677
editorUser = await samlAuth.createM2mApiKeyWithRoleScope('editor');
7778
await supertest
@@ -84,7 +85,7 @@ export default function ({ getService }: DeploymentAgnosticFtrProviderContext) {
8485
});
8586

8687
after(async () => {
87-
await kibanaServer.savedObjects.cleanStandardList();
88+
await cleanSyntheticsTestData(kibanaServer);
8889
});
8990

9091
it('edits the monitor', async () => {

0 commit comments

Comments
 (0)