[ES|QL] Validation tests agains Elasticsearch by bartoval · Pull Request #270445 · elastic/kibana

bartoval · 2026-05-21T15:20:46Z

Summary

I kept the logic intentionally basic— there’s no file autodetection or other “smart” behavior.

To test it locally:

yarn test:jest_integration --config src/platform/packages/shared/kbn-esql-language/jest.integration.config.js

bartoval · 2026-05-21T15:22:26Z

/ci

coderabbitai · 2026-05-21T15:22:51Z

Caution

Review failed

An error occurred during the review process. Please try again later.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

🛠️ Update Documentation

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

coderabbitai · 2026-05-21T15:22:52Z

Caution

Review failed

An error occurred during the review process. Please try again later.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

🛠️ Update Documentation

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

bartoval · 2026-05-21T16:54:11Z

/ci

bartoval · 2026-05-21T19:07:07Z

/ci

bartoval · 2026-05-26T13:37:09Z

/ci

bartoval · 2026-05-26T13:46:01Z

/ci

bartoval · 2026-05-27T15:50:41Z

/ci

bartoval · 2026-05-28T06:38:25Z

I'm not closing this task yet because I still have a couple of open points to clarify:

src/platform/packages/shared/kbn-esql-language/src/language/validation/validation.test.ts is currently a monolith that generates the JSON consumed by the FRT tests. In theory, this behavior is already covered by FRT, but if the intention is to remove those FRT tests, then this coverage needs to be added here. This opens up a couple more modifcations that I will eventually address in another pr.
Right now, we cover false positives — cases where Kibana reports an error but Elasticsearch actually accepts the query, effectively blocking the user unnecessarily.
I think it makes sense not to cover false negatives -> The user can execute the query and will receive the error directly from Elasticsearch.
I’m using the default timeout preset, which is jest.setTimeout(10 * 60 * 1000) (10 minutes). If we want a lower timeout for this test, I’ll need to set it explicitly.
src/platform/packages/shared/kbn-test/jest_integration_node/jest-preset.js -> src/platform/packages/shared/kbn-test/src/jest/setup/after_env.integration.js
I created these suites so they can be reused for both integration and unit tests, mainly for convenience since the set of files is more or less fixed and doesn’t really change over time. When a new file is added, it just needs to be imported manually into the suite. Since that’s a minimal effort, I preferred to avoid adding extra abstractions unnecessarily.

infra-vault-gh-plugin-prod · 2026-05-28T07:02:49Z

Pinging @elastic/kibana-esql (Team:ESQL)

stratoula

@bartoval we also have these src/platform/test/api_integration/apis/esql so having some there and the other here is a bit confusing.

I suggest to have one place to put the integration tests and a common testing strategy. Can you take a look holistically?

bartoval · 2026-05-29T11:16:12Z

@bartoval we also have these src/platform/test/api_integration/apis/esql so having some there and the other here is a bit confusing.

I suggest to have one place to put the integration tests and a common testing strategy. Can you take a look holistically?

I removed the FRTs and the JSON file generated from validation.test.ts for these tests.
I partially migrated validation.test.ts into dedicated test files. This way, the tests follow the same strategy. I already did this as a pilot for sources and command.
I’ll migrate the rest in another PR because handling the whole migration here would become difficult.

stratoula

I like where it goes but I would like us to be more strategic:

we need to optimize to not make CI slower

stratoula · 2026-06-01T06:13:04Z

+    await esqlEnv.setupIndicesPolicies();
+  });
+
+  afterAll(async () => {


afterAll(async () => { try { await esqlEnv?.cleanup(); } finally { await esqlEnv?.integrationEnv.stop(); } });

stratoula · 2026-06-01T06:14:35Z

+        if (type === 'cartesian_shape') {
+          esType = 'shape';
+        }
+        if (type === 'aggregate_metric_double') {


Is this ok to map it as double? It is an object https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/aggregate-metric-double

stratoula · 2026-06-01T06:15:33Z

+      { ignore: [404] }
+    );
+    for (const policy of policies) {
+      await es.enrich.deletePolicy({ name: policy.name }, { ignore: [404] });


We are deleting the policies but not the backing indices.

stratoula · 2026-06-01T06:21:54Z

+      // Integration suites use this hook to compare the same validation run with ES.
+      // Unit suites leave it undefined, so their behavior stays unchanged.
+      const promise = validateQuery(query, cb).then(async (result) => {
+        await afterValidate?.({


I think we need to do some optimizations before merging this Val. With all the efforts done to improve the CI time I am afraid that having these run in every PR might make the situation worse especially when we run this sequentially.

Some ideas:

Batch the ES queries

Only check queries with non-syntax errors

kibanamachine · 2026-06-01T06:34:59Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: cb72f2a
Build duration: 54 mins

Failed CI Steps

Test Failures

[job] [logs] Scout Lane #11 - stateful-classic / default / local-stateful-classic - Saved query menu — CRUD (Discover) - save, load, update, save-as-new, delete via the popover
[job] [logs] Scout Lane #12 - stateful-classic / default / local-stateful-classic - UptimeIntegrationDeprecation - returns true when non-managed synthetics policies exist

Metrics [docs]

✅ unchanged

History

💚 Build #450241 succeeded 85496fa
💛 Build #449330 was flaky 278bb7d
💛 Build #449069 was flaky eb56889
💔 Build #448194 failed a58f9b5
💔 Build #448184 failed 1221080
💛 Build #446507 was flaky 231d877

cc @bartoval

bartoval · 2026-06-01T06:37:03Z

I don't cover the false negative => The user can execute the query and will receive the error directly from Elasticsearch.
One reason I didn't optimize is that test ran fast. Where did you see any slowdowns?

stratoula · 2026-06-01T06:53:40Z

The user can execute the query and will receive the error directly from Elasticsearch.

My bad Valerio, I got confused from one comment and thought we are not doing this

Where did you see any slowdowns?

I didnt run them, from the code review it seems as doing this sequentially will make CI slower (?) So if we could batch them and make it even faster would be great. Unless the @elastic/appex-qa is ok with it. Not sure which is the process here but I wouldnt want to have complains about us making the CI slower in each PR 😅

validation tests again ES

4747623

bartoval self-assigned this May 21, 2026

bartoval added release_note:skip Skip the PR/issue when compiling release notes backport:skip This PR does not require backporting Feature:ES|QL ES|QL related features in Kibana Team:ESQL ES|QL related features in Kibana t// v9.5.0 labels May 21, 2026

bartoval changed the title ~~[ES|QL] validation tests agains Elaticsearch~~ [ES|QL] Validation tests agains Elaticsearch May 21, 2026

Changes from node scripts/regenerate_moon_projects.js --update

f773273

bartoval changed the title ~~[ES|QL] Validation tests agains Elaticsearch~~ [ES|QL] Validation tests agains Elasticsearch May 21, 2026

refactor

1b3af55

bartoval force-pushed the validation-tests-es branch from a79793e to 1b3af55 Compare May 21, 2026 16:00

Merge branch 'main' into validation-tests-es

dfd9f85

fix ci errors

231d877

bartoval and others added 2 commits May 26, 2026 15:34

Run client-side validator live in integration tests

0e3ccb1

Merge branch 'main' into validation-tests-es

1221080

fix typo

a58f9b5

bartoval and others added 4 commits May 27, 2026 06:48

Add validation Jest integration coverage

4e80467

Merge branch 'main' into validation-tests-es

521eb55

refactor

d5d8506

wiring unit tests

6e8e64c

bartoval force-pushed the validation-tests-es branch from 9839573 to 6e8e64c Compare May 27, 2026 15:42

Merge branch 'main' into validation-tests-es

eb56889

cleanup

c137f37

Merge branch 'main' into validation-tests-es

278bb7d

bartoval marked this pull request as ready for review May 28, 2026 07:02

bartoval requested a review from a team as a code owner May 28, 2026 07:02

stratoula reviewed May 29, 2026

View reviewed changes

bartoval added 2 commits May 29, 2026 12:30

remove JSON generation from validation.test.ts and FRT tests

c472c97

pilot migration of validation.tests.ts

4fc78f3

bartoval requested a review from a team as a code owner May 29, 2026 11:07

Changes from node scripts/lint_ts_projects --fix

ff5ef4c

kibanamachine and others added 3 commits May 29, 2026 11:18

Changes from node scripts/regenerate_moon_projects.js --update

8270329

avoid validation redoundancy in file names

85496fa

Merge branch 'main' into validation-tests-es

cb72f2a

stratoula reviewed Jun 1, 2026

View reviewed changes

Conversation

bartoval commented May 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

bartoval commented May 21, 2026

Uh oh!

coderabbitai Bot commented May 21, 2026

Review failed

Uh oh!

coderabbitai Bot commented May 21, 2026

Review failed

Uh oh!

bartoval commented May 21, 2026

Uh oh!

bartoval commented May 21, 2026

Uh oh!

bartoval commented May 26, 2026

Uh oh!

bartoval commented May 26, 2026

Uh oh!

bartoval commented May 27, 2026

Uh oh!

bartoval commented May 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

infra-vault-gh-plugin-prod Bot commented May 28, 2026

Uh oh!

stratoula left a comment

Choose a reason for hiding this comment

Uh oh!

bartoval commented May 29, 2026

Uh oh!

stratoula left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stratoula Jun 1, 2026

Choose a reason for hiding this comment

Uh oh!

stratoula Jun 1, 2026

Choose a reason for hiding this comment

Uh oh!

stratoula Jun 1, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

stratoula Jun 1, 2026

Choose a reason for hiding this comment

Uh oh!

kibanamachine commented Jun 1, 2026

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

History

Uh oh!

bartoval commented Jun 1, 2026

Uh oh!

stratoula commented Jun 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

bartoval commented May 21, 2026 •

edited

Loading

bartoval commented May 28, 2026 •

edited

Loading

stratoula left a comment •

edited

Loading

stratoula commented Jun 1, 2026 •

edited

Loading