You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
/src/platform/test/functional/fixtures/kbn_archiver/saved_search.json@elastic/kibana-core# Assigned per only use: https://github.com/elastic/kibana/blob/main/test/interpreter_functional/test_suites/run_pipeline/esaggs.ts#L100
2292
-
/src/platform/test/functional/fixtures/kbn_archiver/saved_objects_management/show_relationships.json@elastic/kibana-core# Assigned per only use: https://github.com/elastic/kibana/blob/main/test/functional/apps/saved_objects_management/show_relationships.ts#L20
/src/platform/test/functional/fixtures/kbn_archiver/saved_objects_management/edit_saved_object.json@elastic/kibana-core# Assigned per only use: https://github.com/elastic/kibana/blob/main/test/functional/apps/saved_objects_management/inspect_saved_objects.ts#L40
Copy file name to clipboardExpand all lines: .github/workflows/failed-test-investigator.md
+22-54Lines changed: 22 additions & 54 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -87,20 +87,9 @@ Investigate a failed-test issue, classify the failure, and propose a fix when ap
87
87
-**`issues` trigger**: use the triggering issue (non-PR, labeled `failed-test`).
88
88
-**`workflow_dispatch`**: use issue `${{ github.event.inputs.issue_number }}`. Fetch it explicitly before analysis, and post the final comment there.
89
89
90
-
## Where did the test run?
91
-
92
-
The test's **target** (e.g. `local-stateful-classic`, `cloud-serverless-security_complete`) tells you where it ran:
93
-
94
-
-**`cloud-*`** — ran against a real Elastic Cloud project (serverless) or deployment (stateful). Pipeline names: `appex-qa-{serverless|stateful}-kibana-{ftr|scout}-tests`.
95
-
-**`local-*`** — ran on the agent's local machine. `kibana-on-merge` and `kibana-pull-request` are local (no Elastic Cloud API calls), so the environment is more stable and less prone to network/env flakiness.
96
-
97
90
## Investigate
98
91
99
-
1. Read the issue title, body, labels, and all comments.
100
-
2. Parse test metadata if present: location (test file path), config path, code owners, target.
101
-
3. Look at all the failures reported in the issue. The very same test could have been failing with different error messages, for different reasons, on different pipelines, and on different branches.
102
-
4. Inspect the relevant test file and nearby helpers/fixtures. For Scout, start from the reported location; otherwise infer from the title.
103
-
5. Check recent git history and blame on the test file and related product code.
92
+
Investigate the test failure(s) using the `flaky-test-investigator` skill.
104
93
105
94
Every conclusion must cite specific evidence. Do not guess.
106
95
@@ -150,53 +139,32 @@ No other side-effects beyond posting the comment and updating the label.
150
139
151
140
## Comment format
152
141
153
-
Post exactly one comment with two main parts:
154
-
155
-
-**Visible section**: a very concise summary that would inform a developer with a quick glance. Highlight main findings. Keep it high-signal and to the point.
156
-
-**Collapsed `<details>` section**: full long-form context for the downstream auto-fix agent (and any human who wants to audit the call).
157
-
158
-
The visible section is a _distillation_ of the collapsed one. Do not repeat content verbatim across both: the visible bullets summarize, the collapsed block holds the full evidence the summary was derived from.
159
-
160
-
### Visible (top), in this order:
161
-
162
-
1.**One-line bold headline** stating the result kind and one identifying detail. Consistent with `classification` but not templated. Example: `**Likely test-design fix** — missing waitForAlertsToPopulate() in building_block_alerts.spec.ts`.
163
-
164
-
2.**A 3–5 sentence prose paragraph** (no headings, no bullets) covering: what broke and where (name the test file/name), the most likely root cause, and any evidence-backed author attribution with `@username` so they get notified on first read.
165
-
166
-
3.**One-line action hint**: the proposed fix, recommended action, or missing evidence. Skip if the paragraph already covers it.
167
-
168
-
4.**Findings bullets** — exactly these four, in this order, with one concrete value each. Downstream tooling parses these directly; preserve keys, casing, and `` - `key`: value `` shape:
5.**Suspected root cause** — 2–4 short bullets, each tied to a specific piece of evidence. Skip the section entirely when `classification` is `external` or `inconclusive` and there is nothing concrete to assert.
176
-
177
-
6.**Key references** — at most 3 Markdown links: the failing test file, the failing CI run, and the implicated commit (when one exists). Skip any of the three that are not applicable; skip the section entirely when none apply.
178
-
179
-
### Collapsed (`<details>`):
180
-
181
-
This section is the full context for agents and humans to dive deep into the findings. Verify all information. Wrap it in a single `<details>` block. The blank lines around `</summary>` and `</details>` are required for the inner markdown to render.
182
-
183
-
```
184
-
<details>
185
-
<summary>See full details</summary>
142
+
Post exactly one comment. Keep the visible portion very short and easy to read:
186
143
187
-
#### Full root-cause analysis
144
+
1.**One-line bold headline** stating the result kind and one identifying detail.
145
+
2.**Diagnosis** (≤5 concise bullet points): what broke and where, the most likely root cause.
146
+
3.**Next steps** (≤5 concise bullet points).
188
147
189
-
The long-form version of the visible "Suspected root cause" bullets. Walk through the evidence chain step by step. Cite the specific log lines, stack frames, blame results, or related PRs that led to the conclusion.
148
+
Put the full `flaky-test-investigator` skill output inside a collapsed `<details><summary>Investigation details</summary> ... </details>` block (not in the visible portion). Open the block with a `#### Findings` subsection containing exactly these four bullets in this order — downstream tooling parses them, so preserve keys, casing, and `` - `key`: value `` shape. These bullets must live **inside `<details>`**, never in the visible portion:
A complete list of the evidence consulted: issue comments, file paths, commits, CI runs, blame output, related PRs. Each item should be a Markdown link, not a bare path or SHA.
155
+
The skill's "Reporting" subsections should also be inside the collapsible section:
194
156
195
-
#### Suggested patch
157
+
- What the test does
158
+
- What failed and when
159
+
- Where it ran
160
+
- Root cause hypothesis
161
+
- Evidence
162
+
- Failure screenshot
163
+
- Recommended next step
164
+
- Open questions
196
165
197
-
Only when justified by the evidence: a small diff-style snippet showing the suggested edit. Include the exact file, function, assertion, wait condition, fixture, selector, API, or behavior to change. Omit this section entirely when no defensible patch can be proposed.
166
+
Blank lines around `</summary>` and `</details>` are required for the inner markdown to render.
198
167
199
-
</details>
200
-
```
168
+
End the comment with this footer line (verbatim, on its own line after the `</details>` block):
201
169
202
-
Use `####` headings inside the details block (not `###`) so they nest below the comment's own structure. Any of the three subsections may be omitted when there is nothing meaningful to put in it.
170
+
`<sup>AI-generated, share feedback in [#appex-qa](https://elastic.slack.com/archives/C04HT4P1YS3)</sup>`
Copy file name to clipboardExpand all lines: docs/extend/scout.md
+3-2Lines changed: 3 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -12,6 +12,7 @@ Scout is Kibana’s **modern UI and API test framework** built on [Playwright](h
12
12
-[Best practices](./scout/best-practices.md) (see also [UI](./scout/ui-best-practices.md) and [API](./scout/api-best-practices.md) test best practices)
13
13
-[UI testing](./scout/ui-testing.md)
14
14
-[API testing](./scout/api-testing.md)
15
+
-[Migrate tests to Scout](./scout/migrate-tests.md)
15
16
16
17
## Scout benefits [scout-main-features]
17
18
@@ -57,7 +58,7 @@ We welcome contributions to one of the Scout packages.
57
58
| Is reusable but scoped to a solution | In the solution Scout package (for example `@kbn/scout-security`, `@kbn/scout-oblt`, `@kbn/scout-search`) | Solution workflows and domain-specific helpers |
58
59
| Is specific to one plugin or package | In your plugin or package’s `test/scout` directory | Components specific to your plugin or package only |
59
60
60
-
## Need help?
61
+
## Need help?[need-help]
61
62
62
63
-**Internal (Elasticians)**: reach out to the AppEx QA team in the `#kibana-scout` Slack channel for guidance.
63
64
@@ -95,7 +96,7 @@ Existing FTR tests continue to run, and teams can migrate them to Scout incremen
95
96
96
97
#### Q: Should I migrate every FTR test to Scout? [scout-faq-migrate-ftr-to-scout]
97
98
98
-
_It, depends_: [pick the right test type](./scout/best-practices.md#pick-the-right-test-type) before migrating a test.
99
+
_It depends_: [pick the right test type](./scout/best-practices.md#pick-the-right-test-type) before migrating a test. Refer to our [Migrate tests to Scout](./scout/migrate-tests.md) guide.
99
100
100
101
#### Q: Does Scout support feature flags? [scout-faq-feature-flags]
| A platform feature that works everywhere |[`tags.deploymentAgnostic`](#scout-deployment-tags-deployment-agnostic)|
61
+
| A solution feature |`tags.stateful.<solution>` + `tags.serverless.<solution>` (use `.complete` when the solution has tiers) |
62
+
| Behavior specific to one serverless project tier | The explicit tier tag, e.g. `tags.serverless.security.essentials` or `tags.serverless.observability.logs_essentials`|
63
+
| Behavior that only exists on stateful |`tags.stateful.<solution>` alone |
64
+
65
+
::::::{warning}
66
+
Don't reach for `tags.deploymentAgnostic` from a solution module. It runs your test across every solution and is expensive — use explicit per-deployment tags instead. See the [`tags.deploymentAgnostic` note](#scout-deployment-tags-deployment-agnostic).
67
+
::::::
68
+
54
69
## Common shortcuts [scout-deployment-tags-shortcuts]
When using `feature_flags.overrides`, the keys must match the feature flag IDs registered by the owning plugin. Overrides bypass variation and type validation, so ensure the values match what the consuming code expects.
91
91
92
-
## Custom server configs (reach out to AppEx QA first) [scout-feature-flags-custom-servers]
92
+
## Custom server configs [scout-feature-flags-custom-servers]
93
93
94
94
Some settings — such as those used during the plugin `setup` lifecycle (e.g., HTTP route registration) — cannot be changed at runtime and must be present when Kibana starts. For these cases Scout supports **custom server configuration sets** that manage a local Kibana process.
0 commit comments