Skip to content

feat(ci): per-config failing-test names in FTR job annotation#271518

Draft
TamerlanG wants to merge 1 commit into
elastic:mainfrom
TamerlanG:ftr/job-annotation
Draft

feat(ci): per-config failing-test names in FTR job annotation#271518
TamerlanG wants to merge 1 commit into
elastic:mainfrom
TamerlanG:ftr/job-annotation

Conversation

@TamerlanG
Copy link
Copy Markdown
Contributor

Summary

Adds structured per-config test-failure details to the Buildkite FTR job annotation, without any shell temp-file plumbing.

How it works:

  • New ftr_annotation_helper.ts exports two functions:
    • snapshotJunitDir(dir) — records current *.xml paths to <dir>/.ftr_annotation_snapshot
    • collectNewFailedTestNames(dir) — diffs snapshot vs current XMLs, parses failures from new files only, deletes snapshot
  • scripts/ftr_annotation_helper.js wires up snapshot <dir> and list-new-failures <dir> CLI subcommands
  • ftr_configs.sh calls snapshot before each config run and list-new-failures in the failure branch to get per-config failing test names
  • ftr_job_annotation.sh contains write_job_annotation() which emits a Buildkite annotation with:
    • Per-config status table (passed / recovered / failed / still-failing / new-failure)
    • Per-config bulleted list of failing test names

Annotation example:

### FTR Configs — `my-group` (attempt 1)

| Config | Duration | Status |
| --- | --- | --- |
| `x-pack/test/foo/config.ts` | 4m 12s | **failed** |
| `x-pack/test/bar/config.ts` | 1m 3s | passed |

**Failing tests — `x-pack/test/foo/config.ts`:**

- Suite › my test name

Test plan

  • Unit tests: node scripts/jest packages/kbn-failed-test-reporter-cli/failed_tests_reporter/ftr_annotation_helper.test.ts (5 tests, all green)
  • CI: push to branch and verify annotation appears in a real FTR step with failing configs

Related

Companion to #269605 (smart retry). This PR adds the annotation layer on top of main independently, so it can land separately.

Adds a `node scripts/ftr_annotation_helper snapshot / list-new-failures`
CLI (backed by `ftr_annotation_helper.ts`) that diffs JUnit XML files
written during each config run, so the job annotation can show exactly
which tests failed per config without any shell temp-file plumbing.

- `snapshotJunitDir`: records current *.xml paths to `.ftr_annotation_snapshot`
- `collectNewFailedTestNames`: diffs snapshot vs current files, parses
  failures from new XMLs only, then deletes the snapshot file
- `ftr_configs.sh` calls `snapshot` before each FTR run and
  `list-new-failures` in the failure branch; builds per-config annotation
  rows (passed / recovered / failed / still-failing / new-failure) and
  emits a `write_job_annotation` summary at the end
@infra-vault-gh-plugin-prod
Copy link
Copy Markdown

🤖 Jobs for this PR can be triggered through checkboxes. 🚧

ℹ️ To trigger the CI, please tick the checkbox below 👇

  • Click to trigger kibana-pull-request for this PR!
  • Click to trigger kibana-deploy-project-from-pr for this PR!
  • Click to trigger kibana-deploy-cloud-from-pr for this PR!
  • Click to trigger kibana-entity-store-performance-from-pr for this PR!
  • Click to trigger kibana-storybooks-from-pr for this PR!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant