Configurable refresh interval for FTR

[dgieselaar]

The refresh interval of Elasticsearch is by default 1s in stateful, and 5s in stateless. This means that our FTR spend quite some time waiting on refreshes. To speed up these tests, we can consider configuring the refresh interval to be much lower - e.g. 5-50ms, which should yield significant speedups, with a risk of introducing race condition related failures.

[elasticmachine]

Pinging @elastic/obs-entities (Team:obs-entities)

[klacabane]

Reference PR #232545

[klacabane]

Tests that are still failing, and that I could not reproduce locally since they either succeed or fail differently during a setup step

• [job] [logs] FTR Configs 71 / Serverless Observability - Deployment-agnostic API integration tests Observability Alerting Custom Threshold rule GROUP_BY - FIRED Rule creation should be active
• [job] [logs] FTR Configs 62 / apis home apis sample data apis install in the other space dates should load elasticsearch index containing sample data with dates relative to current time - Fails locally with index_not_found_exception: no such index [kibana_sample_data_flights] it depends on a previous test to install index - PASSES LOCALLY
• [job] [logs] FTR Configs 62 / apis home apis sample data apis install in the other space dates should load elasticsearch index containing sample data with dates relative to now parameter - PASSES LOCALLY
• [job] [logs] FTR Configs 3 / alerting api integration security and spaces enabled - Group 2 Connectors webhook action OAuth2 client credentials should refresh the token once the previous one has expired - PASSES LOCALLY
• [job] [logs] FTR Configs 6 / EQL execution logic API ess serverless Alert Suppression for EQL rules skipInServerless sequence queries with suppression duration suppresses alerts across two rule executions when the suppression duration exceeds the rule interval - PASSES LOCALLY - Appears to be an unlucky failure https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/9485
• [job] [logs] FTR Configs 123 / Detection Engine - Value lists APIs Value Lists - Items APIs ess patch_list_items_migrations patch list items legacy list items index (list created before migration to data stream) should patch list item that was created in legacy index and migrated through LIST_INDEX request - Passes after disabling refresh_interval override
• [job] [logs] FTR Configs 91 / InfraOps App Metrics UI Infrastructure Source Configuration with metrics present can change the metric indices to a remote cluster when connection does not exist - Had to fix an assertion, got the same failure on main not sure why it's not skipped/fixed

[klacabane]

I've started flaky test runs for tests that pass locally

• FAILS - apis home apis sample data apis install in the other space dates should load elasticsearch index containing sample data with dates relative to now parameter - https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/9484
• SUCCEEDS - EQL execution logic API ess serverless Alert Suppression for EQL rules skipInServerless sequence queries with suppression duration suppresses alerts across two rule executions when the suppression duration exceeds the rule interval - https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/9485
• FAILS - Serverless Observability - Deployment-agnostic API integration tests Observability Alerting Custom Threshold rule GROUP_BY - FIRED Rule creation should be active - https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/9486
• FAILS - Connectors webhook action OAuth2 client credentials should refresh the token once the previous one has expired - https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/9487

[klacabane]

New flaky test runs after overriding configs

• Observability Alerting Custom Threshold rule GROUP_BY - FIRED Rule creation should be active - https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/9489
• should load elasticsearch index containing sample data with dates relative to current time - https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/9490
• Connectors webhook action OAuth2 client credentials should refresh the token once the previous one has expired - https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/9491

[klacabane]

FROM scout-test-events-*
  | WHERE reporter.type == "ftr"
  | WHERE buildkite.build.number IN (349805,349808)
  | WHERE test_run.status == "passed" AND event.action == "run-end"
  | EVAL duration = test_run.duration, id = test_run.config.file.path
  | EVAL is_left = CASE(buildkite.build.number == 349805, true, false)
  | EVAL left_duration = CASE(is_left, duration, NULL), right_duration = CASE(is_left, NULL, duration)
  | STATS avg_left_duration = AVG(left_duration), avg_right_duration = AVG(right_duration) BY id
  | STATS total_left_duration = SUM(avg_left_duration), total_right_duration = SUM(avg_right_duration)
  | EVAL total_left_duration = total_left_duration / 1000 / 60, total_right_duration = total_right_duration  / 1000 / 60
  | EVAL total_left_duration = ROUND(total_left_duration), total_right_duration = ROUND(total_right_duration)
  | EVAL diff = ((total_left_duration - total_right_duration) / (total_right_duration) * 100)

reports a ~16% runtime decrease

[klacabane]

Seeing failures of src/platform/test/functional/apps/discover/esql/config.ts #9 / discover/esql discover esql columns should correctly set fields when initial query returns no results in CI and in flaky test runner recently, but it does not look related to the refresh_interval as we can see the test failing more often in our metrics