Skip to content

Commit 6c3621f

Browse files
committed
drop hardcoded artifact paths
1 parent 2502029 commit 6c3621f

3 files changed

Lines changed: 6 additions & 36 deletions

File tree

.agents/skills/flaky-test-investigator/SKILL.md

Lines changed: 2 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -42,8 +42,6 @@ For every failure, try to retrieve:
4242
- **Server logs** (`kibana.log`, `elasticsearch.log` when present). Cross-reference the failure timestamp with any errors in the logs — a server-side 500 or unexpected warning is strong evidence the failure is a product bug, not a test bug.
4343
- **Full session trace** when the framework supports it (Scout / Playwright). Lets you scrub through every step, locator query, network call, and DOM snapshot.
4444

45-
How to actually find and download each artifact type is framework-specific — see "Retrieve failure artifacts" below.
46-
4745
Things to specifically check in the artifacts before forming a root-cause hypothesis:
4846

4947
- **Did the expected element render at all?** If yes and the selector missed it → flaky selector (Tier 2 fix territory). If no → real rendering / race / data issue (Tier 1 territory).
@@ -53,27 +51,9 @@ Things to specifically check in the artifacts before forming a root-cause hypoth
5351

5452
If artifacts are not available (expired, not uploaded, no `read_artifacts` token), say so in the report rather than fabricating a hypothesis. "Screenshot would have resolved this; not available" is a valid open question.
5553

56-
### Retrieve failure artifacts
57-
58-
The standard recipe is **list → filter by path → download by ID**, always scoped to the failed job's UUID. Two Buildkite gotchas to know about first:
59-
60-
- **Failed-attempt jobs are hidden by default.** `/builds/<n>` returns only the latest attempt; append `?include_retried_jobs=true` to find the original failing job (the one cited in `failed-test` comments). `retried` and `retried_in_job_id` link the two.
61-
- **Per-job artifacts use a different endpoint than build-wide artifacts.** If a build retried to green, failure artifacts only live on the failed job's listing (`bk artifacts list <build> -p <pipeline> --job-uuid <jobId>`). Don't conclude "no screenshot uploaded" until you've checked there.
62-
63-
**Scout** (`@kbn/scout-reporting`, not standard Playwright output — `playwright-report/`, `trace.zip`, and video are NOT published):
64-
65-
- `.scout/reports/scout-playwright-test-failures-<runId>/test-failures-summary.json` — maps test name → HTML report. Start here.
66-
- `.scout/reports/scout-playwright-test-failures-<runId>/<testId>.html` — self-contained: error, stdout, embedded screenshot. Usually sufficient on its own.
67-
- `.scout/reports/scout-playwright-test-failures-<runId>/scout-failures-<runId>.ndjson` — one record per failure (`id` = `<testId>`, `owner`, `location`, `error.*`) for programmatic use.
68-
- `**/.scout/test-artifacts/<test-slug>/test-failed-<N>.png` — plain Playwright screenshot; the PNG doesn't carry `<testId>`, so correlate via spec path.
69-
70-
**FTR** (a single content `<hash>` links every artifact for one failure):
71-
72-
- `target/test_failures/<jobId>_<hash>.{json,log,html}``.json` is source of truth; full Kibana/ES stdout lives in `system-out` (there is no separate `kibana.log`). Pull this first.
73-
- `<test-root>/screenshots/failure/*-<hash>.png` and `<test-root>/failure_debug/html/*-<hash>.html` — UI tests only; fetch only when the failure is UI-side.
74-
- `.es/*.log` — transport/cluster-shaped failures.
54+
### List failure artifacts
7555

76-
`target/test_failures/` is shared with Scout; filter by `.jobName` (e.g. `FTR Configs #90` vs `Scout Lane #12`) to keep only FTR. On Cloud FTR pipelines the layout differs: one self-contained HTML per failure at `<config-path-with-underscores>-<unix-timestamp>/html/<contentHash>.html` — no `target/test_failures/`, screenshot, or DOM artifacts.
56+
`bk artifacts list <build> -p <pipeline> --job-uuid <jobId>` returns a JSON listing of every artifact uploaded for the failing job. Pass `--job-uuid <jobId>` for the failed attempt (without it, `bk` only returns the latest attempt and hides retried failures). If a build retried to green, failure artifacts only live on the failed job's listing; don't conclude "no screenshot" until you've scoped to the right job UUID.
7757

7858
### Understand the scope
7959

.github/CODEOWNERS

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3539,6 +3539,7 @@ x-pack/solutions/observability/plugins/synthetics/server/saved_objects/synthetic
35393539
/src/cli/ @elastic/kibana-operations
35403540
/src/cli_keystore/ @elastic/kibana-operations
35413541
/.github/workflows/ @elastic/kibana-operations
3542+
/.github/workflows/failed-test-investigator.md @elastic/kibana-operations @elastic/appex-qa
35423543
/.github/aw/ @elastic/kibana-operations
35433544
/.buildkite/ @elastic/kibana-operations
35443545
/moon.yml @elastic/kibana-operations

.github/workflows/failed-test-investigator.md

Lines changed: 3 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -143,28 +143,17 @@ Post exactly one comment. Keep the visible portion very short and easy to read:
143143

144144
1. **One-line bold headline** stating the result kind and one identifying detail.
145145
2. **Diagnosis** (≤5 concise bullet points): what broke and where, the most likely root cause.
146-
3. **Next steps** (≤5 concise bullet points).
146+
3. **Recommended next steps** (≤5 concise bullet points).
147147

148-
Put the full `flaky-test-investigator` skill output inside a collapsed `<details><summary>Investigation details</summary> ... </details>` block (not in the visible portion). Open the block with a `#### Findings` subsection containing exactly these four bullets in this order — downstream tooling parses them, so preserve keys, casing, and `` - `key`: value `` shape. These bullets must live **inside `<details>`**, never in the visible portion:
149-
150-
- `classification`: `test-design` | `test-environment` | `application` | `external` | `inconclusive`
151-
- `confidence`: `high` | `medium` | `low`
152-
- `test.type`: `scout` (if `scout-playwright` label) | `ftr` | `jest` | `unknown`
153-
- `test.file`: repo-relative path, or `unknown`
148+
Put the full `flaky-test-investigator` skill output inside a collapsed `<details><summary>Investigation details</summary> ... </details>` block (not in the visible portion).
154149

155150
The skill's "Reporting" subsections should also be inside the collapsible section:
156151

157152
- What the test does
158-
- What failed and when
159153
- Where it ran
160154
- Root cause hypothesis
161155
- Evidence
162-
- Failure screenshot
163-
- Recommended next step
156+
- Failure screenshot (omit this section if not available)
164157
- Open questions
165158

166159
Blank lines around `</summary>` and `</details>` are required for the inner markdown to render.
167-
168-
End the comment with this footer line (verbatim, on its own line after the `</details>` block):
169-
170-
`<sup>AI-generated, share feedback in [#appex-qa](https://elastic.slack.com/archives/C04HT4P1YS3)</sup>`

0 commit comments

Comments
 (0)