[CI] Opt-in FTR solution-selective testing via PR labels#271968
Closed
shahzad31 wants to merge 1 commit into
Closed
[CI] Opt-in FTR solution-selective testing via PR labels#271968shahzad31 wants to merge 1 commit into
shahzad31 wants to merge 1 commit into
Conversation
Reduce flaky-FTR noise on PRs whose changes are confined to a single solution. Two opt-in PR labels gate the behaviour: - ci:skip-unaffected-ftr-configs: drop FTR configs that belong to solutions the PR does not touch. - ci:soft-fail-unaffected-ftr-configs: keep those configs running but mark their failures non-blocking (warning annotations) so unrelated flakiness can't block the PR. Detection is deliberately conservative. The diff is only narrowed when every changed file (and every downstream dependent) is confined to one or more solutions' private code. Anything touching platform/shared modules, CI/test-infra, or FTR manifests — or a downstream platform consumer — runs the full, blocking suite exactly as today. On-merge builds always run the full suite. Implemented on top of the affected-packages module graph: kibana.jsonc `group` is now captured per module (getModuleGroup) and the downstream dependency graph is used to expand the affected set before deciding whether the change is solution-confined. Co-authored-by: Cursor <cursoragent@cursor.com>
shahzad31
added
release_note:skip
github-actions
Bot
added
the
author:actionable-obs
|
🤖 Jobs for this PR can be triggered through checkboxes. 🚧
ℹ️ To trigger the CI, please tick the checkbox below 👇
|
Contributor
Author
|
@tylersmalley @delanni i think seems feasible, at least solution specific skip/soft-fail for ftr configs , let me know what you think 🙏🏼 |
Contributor
💔 Build Failed
Failed CI Steps
Test Failures
Metrics [docs]
History |
Contributor
Author
|
we have a RFC in progress from Ops team |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
FTR configs are a frequent source of flaky, unrelated failures that block PRs. This adds opt-in, solution-level selective testing for FTR, gated behind two new PR labels, so a PR confined to one solution doesn't get blocked by another solution's flaky FTR configs.
Two labels (mutually usable,
skipwins if both are set):ci:skip-unaffected-ftr-configs— drop FTR configs that belong to solutions the PR doesn't touch.ci:soft-fail-unaffected-ftr-configs— keep those configs running, but make their failures non-blocking (warning annotations) instead of failing the build.Safety model (conservative by design)
The diff is only narrowed when every changed file — and every downstream dependent — is confined to one or more solutions' private code. The full, blocking suite runs exactly as today whenever the change touches:
platform/ shared modules,platformor an unrecognized group.On-merge builds always run the full suite. Solutions are
visibility: privateand cannot depend on one another, so a diff that stays inside a single solution's code cannot break another solution — that invariant is what makes this safe.How it works
group(fromkibana.jsonc) is now captured per module (getModuleGroup) in the affected-packages graph.selective_ftr.tsresolves the affected solution set from the diff + transitive downstream graph, bailing to a full run on anything cross-cutting.pick_test_group_run_ordereither reduces the enabled FTR config set (skip) or emits aftr_soft_fail_configs.jsonartifact (soft-fail).ftr_configs.shtreats soft-fail configs as non-blocking per-config.Dry-run impact (40 recent merged PRs)
For a typical solution-confined PR, ~40% of the FTR suite's blocking surface (the other solutions' configs) is removed or made non-blocking. No diff touching platform/shared/CI was ever narrowed in the sample.
Test plan
node .buildkitejest — 95 tests).ci:skip-unaffected-ftr-configsand confirm other solutions' FTR configs are dropped.ci:soft-fail-unaffected-ftr-configsand confirm other solutions' configs run but are non-blocking.platform/shared/CI still runs the full blocking suite.Made with Cursor