migrate som / ml / monitoring ui tests to scout by dmlemeshko · Pull Request #271394 · elastic/kibana

dmlemeshko · 2026-05-27T09:00:43Z

Migration Summary

ML: x-pack/platform/test/functional/apps/ml/short_tests/feature_controls

FTR file	Scout spec	Layer
ml_security.ts (5 role/navlink scenarios)	x-pack/platform/plugins/shared/ml/test/scout/api/tests/feature_controls.spec.ts	API (capabilities)
ml_spaces.ts (enabled space → app renders; disabled → 404)	x-pack/platform/plugins/shared/ml/test/scout/ui/tests/feature_controls.spec.ts + same API spec (per-space capabilities)	UI + API

Parity: full. Browser-only smoke (mlAppPageOverview render + raw 404 JSON on disabled space) kept in UI; capabilities/navlink contracts moved to API.

Stack Monitoring: x-pack/platform/test/functional/apps/monitoring/feature_controls

FTR file	Scout spec	Layer
monitoring_security.ts (admin sees app, base:all no navlink)	x-pack/platform/plugins/private/monitoring/test/scout/ui/tests/feature_controls.spec.ts	UI
monitoring_spaces.ts (enabled/disabled space)	same UI spec	UI

Parity: partial — documented in the spec header.

Migrated: monitoringAppContainer renders for admin (default + enabled space), 404 JSON in disabled space, base:all user does NOT see sidebar link.
Not migrated (intentional gap): monitoring_user alone → forbidden and monitoring_user + kibana_admin → enable denied. Both require the reserved_monitoring ES application privilege, which Scout's setCustomRole cannot provision (the second case was already skipCloud in FTR).

Saved Objects Management: x-pack/platform/test/functional/apps/saved_objects_management/feature_controls

FTR file	Scout spec	Layer
security.ts (5 role/capability scenarios)	src/platform/plugins/shared/saved_objects_management/test/scout/api/tests/feature_controls.spec.ts	API
Same file (browser-only button visibility + appNotFound routing)	src/platform/plugins/shared/saved_objects_management/test/scout/ui/tests/feature_controls.spec.ts	UI

Parity: full. Pure capability/_find contracts at API, per-role button visibility & appNotFound routing at UI.

OSS Saved Objects Management: src/platform/test/functional/apps/saved_objects_management

FTR file	Scout spec	Layer
inspect_saved_objects.ts > "renders the envelope"	covered by ui/tests/spaces_integration.spec.ts + existing api/tests/bulk_get.spec.ts	UI + API
inspect_saved_objects.ts > "allows to delete a saved object"	src/platform/plugins/shared/saved_objects_management/test/scout/ui/tests/inspect.spec.ts	UI (only confirm-modal flow)
show_relationships.ts (invalid relationships)	covered by existing api/tests/relationships_invalid.spec.ts (no new spec needed)	API
spaces_integration.ts	src/platform/plugins/shared/saved_objects_management/test/scout/ui/tests/spaces_integration.spec.ts	UI

x-pack SOM: x-pack/platform/test/functional/apps/saved_objects_management

FTR file	Scout spec	Layer
import_saved_objects_between_versions.ts (3 imports: 7.13, 7.14, 8.0)	API spec for counts + UI render-smoke	API + UI
multi_space_import.ts (default-space + non-default-space import)	merged into same api/tests/imports.spec.ts	API
Copy-to-space flow	src/platform/plugins/shared/saved_objects_management/test/scout/ui/tests/imports.spec.ts (merged with 7.13 render-smoke)	UI

Parity: full. Import counts moved to API (no browser); browser-only checks (rendered dashboard, copy flyout summary counters) stay in UI.

Net CI impact

2 dedicated FTR configs removed (no more per-suite server starts):

x-pack/platform/test/functional/apps/ml/short_tests no longer loads feature_controls.
x-pack/platform/test/functional/apps/monitoring/index.group1.js no longer loads feature_controls.
x-pack/platform/test/functional/apps/saved_objects_management/config.ts — entirely removed.
src/platform/test/functional/apps/saved_objects_management/config.ts — entirely removed.

dmlemeshko · 2026-05-27T09:02:58Z

/flaky scoutConfig:x-pack/platform/plugins/shared/ml/test/scout/api/playwright.config.ts:20

dmlemeshko · 2026-05-27T09:03:12Z

/flaky scoutConfig:x-pack/platform/plugins/shared/ml/test/scout/ui/playwright.config.ts:20

kibanamachine · 2026-05-27T09:03:15Z

Flaky Test Runner

✅ Build triggered - kibana-flaky-test-suite-runner#12466

x-pack/platform/plugins/shared/ml/test/scout/api/playwright.config.ts x20

kibanamachine · 2026-05-27T09:03:28Z

Flaky Test Runner

✅ Build triggered - kibana-flaky-test-suite-runner#12467

x-pack/platform/plugins/shared/ml/test/scout/ui/playwright.config.ts x20

kibanamachine · 2026-05-27T09:28:15Z

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#12467

[✅] x-pack/platform/plugins/shared/ml/test/scout/ui/playwright.config.ts (--arch stateful --domain classic): 20/20 tests passed.

see run history

… into migrate-som-and-ml-ftr-to-scout

dmlemeshko · 2026-05-27T10:38:57Z

/flaky scoutConfig:x-pack/platform/plugins/shared/ml/test/scout/api/playwright.config.ts:20

kibanamachine · 2026-05-27T10:39:13Z

Flaky Test Runner

✅ Build triggered - kibana-flaky-test-suite-runner#12471

x-pack/platform/plugins/shared/ml/test/scout/api/playwright.config.ts x20

kibanamachine · 2026-05-27T11:05:36Z

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#12471

[❌] x-pack/platform/plugins/shared/ml/test/scout/api/playwright.config.ts (--arch stateful --domain classic): 0/20 tests passed.

see run history

dmlemeshko · 2026-05-27T14:52:43Z

/flaky scoutConfig:x-pack/platform/plugins/shared/ml/test/scout/api/playwright.config.ts:20

kibanamachine · 2026-05-27T14:53:04Z

Flaky Test Runner

✅ Build triggered - kibana-flaky-test-suite-runner#12476

x-pack/platform/plugins/shared/ml/test/scout/api/playwright.config.ts x20

qn895 · 2026-05-27T14:58:25Z

+
+import { tags } from '@kbn/scout';
+import { expect } from '@kbn/scout/api';
+import { apiTest, COMMON_HEADERS, CUSTOM_ROLES } from '../fixtures';


Just curious, why we are exporting apiTest in ../fixtures instead of a direct import from kbn/scout?

good point, currently we don't extend kbn/scout apiTest fixture with any custom logic, so we can important directly from kbn/scout

No need to change for this PR, just curious. But good to know apiTest can be extended with custom logic. Is that different from mlApiTest?

mlApiTest is the one extended for this plugin and it should be used in tests. I will push update.

qn895

LGTM 🎉

kibanamachine · 2026-05-27T15:22:46Z

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#12476

[❌] x-pack/platform/plugins/shared/ml/test/scout/api/playwright.config.ts (--arch stateful --domain classic): 0/20 tests passed.

see run history

… into migrate-som-and-ml-ftr-to-scout

ElenaStoeva

Changes lgtm

… into migrate-som-and-ml-ftr-to-scout

pickypg

LG Stack Monitoring as long as it passes :)

kibanamachine · 2026-05-27T23:25:00Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: f3fe50c
Build duration: 58 mins

Failed CI Steps

Test Failures

[job] [logs] FTR Configs #163 / alerting api integration security and spaces enabled - Group 2 Alerting and Actions Telemetry test telemetry "before each" hook for "should retrieve telemetry data in the expected format"
[job] [logs] Jest Tests #1 / AssetDocumentTab should select json tab when clicked
[job] [logs] Scout Lane #47 - serverless-observability_complete / default / local-serverless-observability_complete - Hosts Page - Empty State - should show onboarding page when no data is present
[job] [logs] Scout Lane #10 - stateful-classic / default / local-stateful-classic - Saved query menu — CRUD (Discover) - save, load, update, save-as-new, delete via the popover

Metrics [docs]

✅ unchanged

History

💔 Build #449085 failed 79ede96
💛 Build #448799 was flaky 4712260
💔 Build #448777 failed 29696ae
💔 Build #448699 failed 7694226

kibanamachine · 2026-05-28T08:47:00Z

Starting backport for target branches: 8.19, 9.3, 9.4

https://github.com/elastic/kibana/actions/runs/26564547780

kibanamachine · 2026-05-28T08:58:58Z

💔 All backports failed

Status	Branch	Result
❌	8.19	Backport failed because of merge conflicts
❌	9.3	Backport failed because of merge conflicts
❌	9.4	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 271394

Questions ?

Please refer to the Backport tool documentation

…sts4 * commit '56de32d81a3c90f159477cdecc8914612ea45332': (27 commits) [One Workflow] Serial polling for execution UI to prevent overlapping API requests (elastic#271072) migrate som / ml / monitoring ui tests to scout (elastic#271394) Upgrade @opentelemetry/host-metrics 0.38.2 → 0.38.3 (elastic#271613) Bump fast-xml-builder to 1.2.0 (elastic#270595) remove redundant top-level after hooks from FTR suites (elastic#271312) [Entity Store] Fix before-all timeout in CSV upload FTR suite and unskip (elastic#269798) [Security Solution] Enable @kbn/change-history feature flag (elastic#270091) (fix) OBS BI - exclude rollup indices from otel telemetry collection plugin (elastic#271453) [Synthetics] All monitors errors page !! (elastic#211205) [One Workflow] managed workflows - restrict mutations (elastic#270896) [Fleet] Fix namespace index templates labels (elastic#271465) [Security solution] Remove stale GenAI ownership (elastic#271593) [kbn-ui] Move skill to proper folder (elastic#270542) [ResponseOps][Alerting V2] Response descriptions are generic and non-specific (elastic#271176) [ES|QL] Dot-prefixed sources validation (elastic#271263) [Scout] Update test config manifests (elastic#271606) [CI] Remove shallow checkout for rspack verify step (elastic#271331) [GH Workflow] Add Node Version (elastic#271578) [ResponseOps] Synchronously invalidate rule API keys on DELETE via `invalidate_api_key_now` (elastic#267556) [* as Code] Remove CM from visualize library and refactor add from library flyout (elastic#269399) ...

kibanamachine · 2026-05-29T08:49:19Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 271394 locally
cc: @dmlemeshko

### Migration Summary 1. ML: `x-pack/platform/test/functional/apps/ml/short_tests/feature_controls` FTR file | Scout spec | Layer -- | -- | -- ml_security.ts (5 role/navlink scenarios) | x-pack/platform/plugins/shared/ml/test/scout/api/tests/feature_controls.spec.ts | API (capabilities) ml_spaces.ts (enabled space → app renders; disabled → 404) | x-pack/platform/plugins/shared/ml/test/scout/ui/tests/feature_controls.spec.ts + same API spec (per-space capabilities) | UI + API Parity: **full**. Browser-only smoke (mlAppPageOverview render + raw 404 JSON on disabled space) kept in UI; capabilities/navlink contracts moved to API. 2. Stack Monitoring: `x-pack/platform/test/functional/apps/monitoring/feature_controls` FTR file | Scout spec | Layer -- | -- | -- monitoring_security.ts (admin sees app, base:all no navlink) | x-pack/platform/plugins/private/monitoring/test/scout/ui/tests/feature_controls.spec.ts | UI monitoring_spaces.ts (enabled/disabled space) | same UI spec | UI Parity: **partial** — documented in the spec header. Migrated: monitoringAppContainer renders for admin (default + enabled space), 404 JSON in disabled space, base:all user does NOT see sidebar link. Not migrated (intentional gap): `monitoring_user` alone → forbidden and `monitoring_user + kibana_admin` → enable denied. Both require the reserved_monitoring ES application privilege, which Scout's setCustomRole cannot provision (the second case was already skipCloud in FTR). 3. Saved Objects Management: `x-pack/platform/test/functional/apps/saved_objects_management/feature_controls` FTR file | Scout spec | Layer -- | -- | -- security.ts (5 role/capability scenarios) | src/platform/plugins/shared/saved_objects_management/test/scout/api/tests/feature_controls.spec.ts | API Same file (browser-only button visibility + appNotFound routing) | src/platform/plugins/shared/saved_objects_management/test/scout/ui/tests/feature_controls.spec.ts | UI Parity: **full**. Pure capability/_find contracts at API, per-role button visibility & appNotFound routing at UI. 4. OSS Saved Objects Management: `src/platform/test/functional/apps/saved_objects_management` FTR file | Scout spec | Layer -- | -- | -- inspect_saved_objects.ts > "renders the envelope" | covered by ui/tests/spaces_integration.spec.ts + existing api/tests/bulk_get.spec.ts | UI + API inspect_saved_objects.ts > "allows to delete a saved object" | src/platform/plugins/shared/saved_objects_management/test/scout/ui/tests/inspect.spec.ts | UI (only confirm-modal flow) show_relationships.ts (invalid relationships) | covered by existing api/tests/relationships_invalid.spec.ts (no new spec needed) | API spaces_integration.ts | src/platform/plugins/shared/saved_objects_management/test/scout/ui/tests/spaces_integration.spec.ts | UI 5. x-pack SOM: `x-pack/platform/test/functional/apps/saved_objects_management` FTR file | Scout spec | Layer -- | -- | -- import_saved_objects_between_versions.ts (3 imports: 7.13, 7.14, 8.0) | API spec for counts + UI render-smoke | API + UI multi_space_import.ts (default-space + non-default-space import) | merged into same api/tests/imports.spec.ts | API Copy-to-space flow | src/platform/plugins/shared/saved_objects_management/test/scout/ui/tests/imports.spec.ts (merged with 7.13 render-smoke) | UI Parity: **full**. Import counts moved to API (no browser); browser-only checks (rendered dashboard, copy flyout summary counters) stay in UI. ### Net CI impact 2 dedicated FTR configs removed (no more per-suite server starts): - x-pack/platform/test/functional/apps/ml/short_tests no longer loads feature_controls. - x-pack/platform/test/functional/apps/monitoring/index.group1.js no longer loads feature_controls. - x-pack/platform/test/functional/apps/saved_objects_management/config.ts — entirely removed. - src/platform/test/functional/apps/saved_objects_management/config.ts — entirely removed. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

kibanamachine · 2026-06-01T09:48:58Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 271394 locally
cc: @dmlemeshko

migrate som / ml / monitoring ui tests to scout

7694226

dmlemeshko added release_note:skip Skip the PR/issue when compiling release notes backport:all-open Backport to all branches that could still receive a release labels May 27, 2026

kibanamachine added 3 commits May 27, 2026 09:08

Changes from node scripts/lint_ts_projects --fix

1ff4b64

Changes from node scripts/regenerate_moon_projects.js --update

c0a1a1f

Changes from node scripts/eslint_all_files --no-cache --fix

1d2e596

dmlemeshko added 2 commits May 27, 2026 12:37

cleanup

8974657

Merge remote-tracking branch 'origin/migrate-som-and-ml-ftr-to-scout'…

29696ae

… into migrate-som-and-ml-ftr-to-scout

dmlemeshko added 2 commits May 27, 2026 13:11

use toStrictEqual

b4afb94

Merge branch 'main' into migrate-som-and-ml-ftr-to-scout

4712260

dmlemeshko marked this pull request as ready for review May 27, 2026 12:30

dmlemeshko requested review from a team as code owners May 27, 2026 12:30

qn895 reviewed May 27, 2026

View reviewed changes

qn895 approved these changes May 27, 2026

View reviewed changes

csr approved these changes May 27, 2026

View reviewed changes

dmlemeshko and others added 3 commits May 27, 2026 17:42

use mlApiTest and predefined roles

0856e47

Merge remote-tracking branch 'origin/migrate-som-and-ml-ftr-to-scout'…

e9a9f5a

… into migrate-som-and-ml-ftr-to-scout

Changes from node scripts/eslint_all_files --no-cache --fix

79ede96

ElenaStoeva approved these changes May 27, 2026

View reviewed changes

dmlemeshko added 2 commits May 28, 2026 00:25

fix linting error

c77eafb

Merge remote-tracking branch 'origin/migrate-som-and-ml-ftr-to-scout'…

f3fe50c

… into migrate-som-and-ml-ftr-to-scout

pickypg approved these changes May 27, 2026

View reviewed changes

dmlemeshko enabled auto-merge (squash) May 28, 2026 06:56

dokmic approved these changes May 28, 2026

View reviewed changes

dmlemeshko merged commit cb5cacd into elastic:main May 28, 2026
31 checks passed

kibanamachine added the v9.5.0 label May 28, 2026

kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label May 29, 2026

Conversation

dmlemeshko commented May 27, 2026 • edited by kibanamachine Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Migration Summary

Net CI impact

Uh oh!

dmlemeshko commented May 27, 2026

Uh oh!

dmlemeshko commented May 27, 2026

Uh oh!

kibanamachine commented May 27, 2026

Flaky Test Runner

✅ Build triggered - kibana-flaky-test-suite-runner#12466

Uh oh!

kibanamachine commented May 27, 2026

Flaky Test Runner

✅ Build triggered - kibana-flaky-test-suite-runner#12467

Uh oh!

kibanamachine commented May 27, 2026

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#12467

Uh oh!

dmlemeshko commented May 27, 2026

Uh oh!

kibanamachine commented May 27, 2026

Flaky Test Runner

✅ Build triggered - kibana-flaky-test-suite-runner#12471

Uh oh!

kibanamachine commented May 27, 2026

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#12471

Uh oh!

dmlemeshko commented May 27, 2026

Uh oh!

kibanamachine commented May 27, 2026

Flaky Test Runner

✅ Build triggered - kibana-flaky-test-suite-runner#12476

Uh oh!

qn895 May 27, 2026

Choose a reason for hiding this comment

Uh oh!

dmlemeshko May 27, 2026

Choose a reason for hiding this comment

Uh oh!

qn895 May 27, 2026

Choose a reason for hiding this comment

Uh oh!

dmlemeshko May 27, 2026

Choose a reason for hiding this comment

Uh oh!

dmlemeshko May 27, 2026

Choose a reason for hiding this comment

Uh oh!

qn895 left a comment

Choose a reason for hiding this comment

Uh oh!

kibanamachine commented May 27, 2026

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#12476

Uh oh!

ElenaStoeva left a comment

Choose a reason for hiding this comment

Uh oh!

pickypg left a comment

Choose a reason for hiding this comment

Uh oh!

kibanamachine commented May 27, 2026

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

History

Uh oh!

Uh oh!

kibanamachine commented May 28, 2026

Uh oh!

kibanamachine commented May 28, 2026

💔 All backports failed

Manual backport

Questions ?

dmlemeshko commented May 27, 2026 •

edited by kibanamachine

Loading