Failing test: default index_pattern API (legacy index pattern api) - does not override existing default without force flag

[kibanamachine]

A test failed on a tracked branch

Scout Test Details:

Field	Value
Test ID	d63ff6050f7b781-18fd78397d3cca2
Target	cloud-stateful-classic
Location	src/platform/plugins/shared/data_views/test/scout/api/tests/index_patterns/default_index_pattern.spec.ts
Duration	0.23s
Module	dataViews (plugin)
Config path	src/platform/plugins/shared/data_views/test/scout/api/playwright.config.ts
Code Owners	elastic/kibana-data-discovery

Error: expect(received).toBe(expected) // Object.is equality

Expected: "default-id-1777165988421-0.5249244759020981"
Received: "default-id-1777165988489-0.1024818726242811"
    at src/platform/plugins/shared/data_views/test/scout/api/tests/index_patterns/default_index_pattern.spec.ts:109:46

First failure: appex-qa-stateful-kibana-scout-tests - main

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[kibanamachine]

New failure for "cloud-stateful-classic" target: appex-qa-stateful-kibana-scout-tests - main

New error message:

Error: expect(received).toBe(expected) // Object.is equality

Expected: "default-id-1777727282291-0.6034903718182163"
Received: "default-id-1777727282149-0.7760231437140086"

[kibanamachine]

New failure for "cloud-stateful-classic" target: appex-qa-stateful-kibana-scout-tests - main

New error message:

Error: expect(received).toBe(expected) // Object.is equality

Expected: "default-id-1777857089792-0.8438693071427112"
Received: "59e7d2da-16b0-4b3b-ad47-2719d70279a2"

[kibanamachine]

New failure for "cloud-stateful-classic" target: appex-qa-stateful-kibana-scout-tests - main

New error message:

Error: expect(received).toBe(expected) // Object.is equality

Expected: "default-id-1778548352565-0.1362332725691836"
Received: "ac2272b9-d2fa-4aaf-b4c2-7874e645d7f5"

[kibanamachine]

New failure for "cloud-stateful-classic" target: appex-qa-stateful-kibana-scout-tests - main

New error message:

Error: expect(received).toBe(expected) // Object.is equality

Expected: "default-id-1778635490505-0.8383775343175288"
Received: "default-id-1778635490339-0.6736624965184721"

[github-actions]

Investigation Summary

This test was previously closed by PR #268370 (commit 1b5e94e, merged 2026-05-12) — which added expect.poll and a unique per-suite space. However, the issue was reopened on 2026-05-13 with a new failure in build 262, so the fix did not fully eliminate flakiness. The remaining failures look consistent with a real race in the server-side UI Settings layer that backs data_views/default: writes use refresh: false and reads flow through a process-wide cache, so the guard in setDefault can read stale state and let a non-force POST overwrite the existing default.

Flakiness Finding

• classification: code
• confidence: medium

Suspected Root Cause

• The server guard in data_views.ts:585-589 is non-atomic: if (force || !(await this.getDefaultId())) { await this.config.set(...) }. The getDefaultId → config.set window is a classic TOCTOU.
• UiSettingsService injects a process-wide sharedUserProvidedCache with NAMESPACED_CACHE_TTL = 10_000 into every scoped client (ui_settings_service.ts:66). Cache reads/dedup happen in ui_settings_client_common.ts:49-82.
• UI Settings writes call the SO client with { refresh: false }. Cache is invalidated locally, but an in-flight or immediately-following read can repopulate the cache with the pre-write ES snapshot before the index refresh. The stale empty result then lives for up to 10s in the namespace.
• Result: when test 2 fires the second POST (no force) right after the first POST + first poll, the handler's getDefaultId() can legitimately return null (cache repopulated from un-refreshed ES) — so the guard collapses and the override is written. Polling cannot recover from this: the second poll permanently observes the overwritten value.
• The earlier UUID-shaped "Received" values in build 244 and build 260 are consistent with the auto-default logic in getDefaultDataViewId (data_views.ts:1431-1451), which silently overwrites defaultIndex to a real data view id whenever the existing id doesn't resolve to an existing pattern — another source of concurrent writes to the same UI setting.
• Recent context: PR #263146 touched data view caching but did not change setDefault/UI Settings. cc PR author @AlexGPlay since this is the same area as the recent fix attempt.

Proposed Fix

Do both: harden the test against the underlying race, and (preferably) fix the race at the source.

1. Server-side root-cause fix (preferred). Either make setDefault honor force/no-force atomically by reading and writing through the SO client with refresh: 'wait_for' (or a conditional update), or stop relying on cached getDefaultId() inside the guard. Change setDefault in data_views.ts:

 setDefault = async (id: string | null, force = false) => {
-  if (force || !(await this.getDefaultId())) {
-    await this.config.set(DEFAULT_DATA_VIEW_ID, id);
-  }
+  if (force) {
+    await this.config.set(DEFAULT_DATA_VIEW_ID, id);
+    return;
+  }
+  // Force a fresh read past the 10s shared UI-settings cache, e.g. by
+  // exposing a bypassCache option on the wrapper, or by gating on a
+  // saved-object-level conditional update so the check and write are atomic.
 }

The straightforward variant is to add a bypassCache: true read path on the UiSettingsServerToCommon wrapper (ui_settings_wrapper.ts) that reads directly from the SO client (skipping sharedUserProvidedCache), and use it inside the no-force branch. Pair the underlying UI-settings write with refresh: 'wait_for' for this setting.

2. Test-side hardening (cheap, ships now). After the override POST, wait a beat and re-poll, or assert stability across N polls so a momentary write-then-rollback doesn't mask the race. In default_index_pattern.spec.ts (and the parallel default_data_view.spec.ts):

   // Try to override the default index pattern without the force flag
   const overrideResponse = await apiClient.post(scopedPath(defaultPath), {
     ...
     body: { [serviceKeyId]: newId() },
   });
   expect(overrideResponse).toHaveStatusCode(200);

+  // Wait past the UI settings shared-cache TTL so a stale getDefaultId()
+  // can't have caused a silent override that has since rolled out.
+  await new Promise((r) => setTimeout(r, 11_000));

   // Verify that the default index pattern is still the same
   await expectDefaultIndexPattern(apiClient, defaultId);

An 11s sleep is ugly; an alternative is to issue several GETs in a row and assert all return defaultId. The point is to defeat a write that may be temporarily masked by the cache.

If the team would rather not fix the server-side race right now, a minimal mitigation is to add await new Promise(r => setTimeout(r, 1500)) between the two POSTs in test 2 (and the parallel default_data_view.spec.ts) so the ES refresh window can close before the no-force POST.

Evidence Used

• Issue body + new-failure comments (Buildkite builds 227, 241, 244, 260, 262).
• Test file: default_index_pattern.spec.ts and the parallel suite default_data_view.spec.ts.
• Server route: default_data_view.ts.
• Server logic: data_views.ts setDefault/getDefaultId and auto-default getDefaultDataViewId.
• UI Settings cache + write semantics: ui_settings_service.ts:66, namespaced_cache.ts:19, ui_settings_client_common.ts:49-117, 183.
• Recent fix attempt: PR #268370 / commit 1b5e94e by @AlexGPlay (added polling and unique spaces; did not address the UI-settings race).

Generated by Failed Test Investigator for issue #265664 · ● 3M · ◷