Repository navigation
Failing test: Dispatcher - dispatches fire actions for non-suppressed episodes and suppress actions for suppressed ones when alert episodes have user actions #300336
Description
A test failed on a tracked branch
Scout Test Details:
| Field | Value |
|---|---|
| Test ID | 98e07605e6e2555-5c049b2017b1c88 |
| Target | cloud-serverless-observability_complete |
| Location | x-pack/platform/plugins/shared/alerting_v2/test/scout/engine_dispatcher/api/tests/dispatcher.spec.ts |
| Duration | 11.33s |
| Module | alertingVTwo (plugin) |
| Config path | x-pack/platform/plugins/shared/alerting_v2/test/scout/engine_dispatcher/api/playwright.config.ts |
| Code Owners | elastic/rna-project-team |
Error: expect(received).toHaveLength(expected)
Expected length: 5
Received length: 10
Received array: [{"@timestamp": "2026-10-24T12:36:28.127841730Z", "action_type": "fire", "actor": {"type": "internal"}, "group_hash": "rule-003-series-2", "last_series_event_timestamp": "2026-10-24T12:33:32.673Z", "reason": "dispatched by policy np-1", "rule_id": "rule-003", "source": "internal", "space_id": "default"}, {"@timestamp": "2026-10-24T12:36:28.127995982Z", "action_type": "fire", "actor": {"type": "internal"}, "group_hash": "rule-003-series-1", "last_series_event_timestamp": "2026-10-24T12:33:42.673Z", "reason": "dispatched by policy np-1", "rule_id": "rule-003", "source": "internal", "space_id": "default"}, {"@timestamp": "2026-10-24T12:36:28.128035009Z", "action_type": "fire", "actor": {"type": "internal"}, "group_hash": "rule-004-series-3", "last_series_event_timestamp": "2026-10-24T12:33:42.673Z", "reason": "dispatched by policy np-1", "rule_id": "rule-004", "source": "internal", "space_id": "default"}, {"@timestamp": "2026-10-24T12:36:28.128081905Z", "action_type": "fire", "actor": {"type": "internal"}, "group_hash": "rule-003-series-2", "last_series_event_timestamp": "2026-10-24T12:33:42.673Z", "reason": "dispatched by policy np-1", "rule_id": "rule-003", "source": "internal", "space_id": "default"}, {"@timestamp": "2026-10-24T12:36:28.128109652Z", "action_type": "fire", "actor": {"type": "internal"}, "group_hash": "rule-004-series-1", "last_series_event_timestamp": "2026-10-24T12:33:42.673Z", "reason": "dispatched by policy np-1", "rule_id": "rule-004", "source": "internal", "space_id": "default"}, {"@timestamp": "2026-10-24T12:36:28.128134663Z", "action_type": "fire", "actor": {"type": "internal"}, "group_hash": "rule-004-series-2", "last_series_event_timestamp": "2026-10-24T12:33:42.673Z", "reason": "dispatched by policy np-1", "rule_id": "rule-004", "source": "internal", "space_id": "default"}, {"@timestamp": "2026-10-24T12:36:28.128159147Z", "action_type": "fire", "actor": {"type": "internal"}, "group_hash": "rule-002-series-1", "last_series_event_timestamp": "2026-10-24T12:33:42.673Z", "reason": "dispatched by policy np-1", "rule_id": "rule-002", "source": "internal", "space_id": "default"}, {"@timestamp": "2026-10-24T12:36:28.128184656Z", "action_type": "fire", "actor": {"type": "internal"}, "group_hash": "rule-005-series-1", "last_series_event_timestamp": "2026-10-24T12:33:42.673Z", "reason": "dispatched by policy np-1", "rule_id": "rule-005", "source": "internal", "space_id": "default"}, {"@timestamp": "2026-10-24T12:36:28.128208057Z", "action_type": "fire", "actor": {"type": "internal"}, "group_hash": "rule-005-series-2", "last_series_event_timestamp": "2026-10-24T12:33:42.673Z", "reason": "dispatched by policy np-1", "rule_id": "rule-005", "source": "internal", "space_id": "default"}, {"@timestamp": "2026-10-24T12:36:28.128231310Z", "action_type": "fire", "actor": {"type": "internal"}, "group_hash": "rule-001-series-1", "last_series_event_timestamp": "2026-10-24T12:33:42.673Z", "reason": "dispatched by policy np-1", "rule_id": "rule-001", "source": "internal", "space_id": "default"}]
at x-pack/platform/plugins/shared/alerting_v2/test/scout/engine_dispatcher/api/tests/dispatcher.spec.ts:809:35
First failure: appex-qa-serverless-kibana-scout-tests - main
Test history: passes and failures over time
Activity
github-actions commentedon Oct 24, 2026
Test needs an update — events are seeded before the user actions the dispatcher must see
A dispatcher tick landed ~455 ms into the test, in the gap between
ruleEvents.seed(events)andalertActionsEvents.seed(userActions), so all 10 episodes dispatched asfireandexpect(fireActions).toHaveLength(5)received 10.Details
Proposed fix
Seed the user actions before the rule events, so no dispatcher tick can ever observe episodes without their suppression state. Both arrays are pure data built from the same
baseTime, so this is a pure statement reorder — move theruleEvents.seed(events)call below thealertActionsEvents.seed(userActions)call in each test:dispatcher.spec.tsruleEvents.seed(events)to afteralertActionsEvents.seed(userActions)(:732)alertActionsEvents.seed(userActions)(:963)This is the ordering already used and documented at :1001 in this same file — "Seed the ack first: the dispatcher only picks up episodes from
.rule-events, so it can never evaluate episode-2 without the ack already in place." A tick in the reordered gap sees user actions but no events, and dispatches nothing.The reorder is safe for dedup: the watermark at
queries.ts:62only takesmax(last_series_event_timestamp)overfire/suppress/unmatched, so pre-seedingack/unack/snooze/deactivaterows cannot advance it or cause the series to be skipped.Separately, the
waitForAtLeast(10, { actionTypes: ['fire', 'suppress'] })gate is composition-blind — 10 wrong rows satisfy it just as well as 10 right ones — so it reports the race as an assertion failure rather than a timeout. Worth tightening to the expected split once the ordering is fixed, but it is not the cause.Root cause & evidence
The dispatcher is a free-running singleton task; the test never gates it, and
queries.tsreads.rule-eventsand.alert-actionsin one ES|QL statement, so each tick evaluates whatever state is committed at that instant. Both seeds userefresh: true, which makes each one individually visible — and therefore makes the window between them observable.The timestamps pin the tick inside that window. Reconstructing
baseTimefrom the reported rows:last_series_event_timestamp: 2026-10-24T12:33:42.673ZiseventTs(165), i.e.baseTime − 165s, givingbaseTime = 12:36:27.673. All 10 rows were written between12:36:28.127841730and12:36:28.128231310— within 390 μs of each other (a single batch), ~455 ms after the test started.Every received row carries
reason: "dispatched by policy np-1"— the generic policy-dispatch reason — and not one carriesack,snooze, ordeactivate, confirming the tick saw an empty.alert-actionsrather than mis-evaluating the suppression rules. The set is also exactly the 10 expected series/episode rows, with the 5 that should have beensuppressflipped tofire.The failure is terminal, not slow: those 10 rows advance the dedup watermark above every seeded event timestamp, so
last_fired < @timestampexcludes all 10 series on every later tick. No additional waiting could have recovered the correct composition — which is why this surfaces as a hard count mismatch.Additional context
fire); not lane/neighbor pollution (beforeEachwipes both streams, and all 10 rows carry this test's ownbaseTime-derived timestamps); not a timeout or an under-budget poll (the gate was satisfied, then the assertion failed on composition).node scripts/scout run-tests --config x-pack/platform/plugins/shared/alerting_v2/test/scout/engine_dispatcher/api/playwright.config.ts. Note a green run is weak evidence for this class — the race only fires on an unlucky tick alignment, so review the seed ordering directly.Tip
Marked "AI-fixable". Add
ai:fix-flakyto request a fix now; otherwise it will be requested automatically if the test fails again.kibanamachine commentedon Oct 25, 2026
New failure for "cloud-serverless-observability_complete" target: appex-qa-serverless-kibana-scout-tests - main
Test history: passes and failures over time
New error message:
github-actions commentedon Oct 25, 2026
✅ Flaky Test Fixer completed successfully!
github-actions commentedon Oct 25, 2026
➡️ A fix PR is ready for review: #300354
It seeds the dispatcher test's user actions before its rule events, closing the window where a tick saw 10 episodes with an empty
.alert-actionsand fired all of them.kibanamachine commentedon Oct 25, 2026
New failure for "cloud-serverless-observability_complete" target: appex-qa-serverless-kibana-scout-tests - main
Test history: passes and failures over time
New error message: