Skip to content

[Vis skill] Fixes ES|QL escaping drift during visualization config generation#272493

Merged
mbondyra merged 1 commit into
elastic:mainfrom
mbondyra:vis_skill_esql_fix
Jun 4, 2026
Merged

[Vis skill] Fixes ES|QL escaping drift during visualization config generation#272493
mbondyra merged 1 commit into
elastic:mainfrom
mbondyra:vis_skill_esql_fix

Conversation

@mbondyra

@mbondyra mbondyra commented Jun 3, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes ES|QL escaping drift during visualization config generation.

What Was Not Working

generate_esql produced the correct query for a GROK pattern:

| GROK message "%{IP} - - \\[%{DATA}\\] \"%{WORD:method} %{DATA}\""

But when the visualization config was generated, the query embedded in visualization.data_source.query lost one escaping level for the bracket regex:

| GROK message "%{IP} - - \[%{DATA}\] \"%{WORD:method} %{DATA}\""

That happened because the prompt asked the model to manually place raw ES|QL inside a JSON string. For ES|QL containing backslashes and quotes, JSON serialization needs an additional escaping layer. The model-generated visualization looked correct visually, but after JSON parsing it no longer matched the authoritative ES|QL.

Description

This change ensures visualization configs preserve the ES|QL query produced by the graph. The prompt now embeds the ES|QL query as a JSON-encoded string, preventing raw interpolation from dropping escape levels for patterns like GROK.
Before:
Screenshot 2026-06-03 at 14 25 37

After:
Screenshot 2026-06-03 at 14 25 31

Test Plan

  • node scripts/jest x-pack/platform/packages/shared/agent-builder/agent-builder-tools-base/visualization/graph_lens.test.ts
  • node scripts/eslint x-pack/platform/packages/shared/agent-builder/agent-builder-tools-base/visualization/graph_lens.ts x-pack/platform/packages/shared/agent-builder/agent-builder-tools-base/visualization/graph_lens.test.ts x-pack/platform/packages/shared/agent-builder/agent-builder-tools-base/visualization/prompts.ts

Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

  • Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
  • Documentation was added for features that require explanation or tutorials
  • Unit or functional tests were updated or added to match the most common scenarios
  • If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
  • This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The release_note:breaking label should be applied in these situations.
  • Flaky Test Runner was used on any tests changed
  • The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines
  • Review the backport guidelines and apply applicable backport:* labels.

Identify risks

Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss.

Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging.

@mbondyra mbondyra added the Team:Presentation Presentation Team for Dashboard, Input Controls, and Canvas t// label Jun 3, 2026
@mbondyra mbondyra marked this pull request as ready for review June 3, 2026 12:26
@mbondyra mbondyra requested a review from a team as a code owner June 3, 2026 12:26
@infra-vault-gh-plugin-prod

Copy link
Copy Markdown

Pinging @elastic/kibana-presentation (Team:Presentation)

@kibanamachine

Copy link
Copy Markdown
Contributor

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

  • [job] [logs] FTR Configs #116 / Discover discover in space Storing search sessions in space Saves and restores a session
  • [job] [logs] Scout Lane #9 - stateful-classic / default / local-stateful-classic - UptimeIntegrationDeprecation - returns true when non-managed synthetics policies exist

Metrics [docs]

✅ unchanged

@mbondyra mbondyra merged commit 5bbfcc2 into elastic:main Jun 4, 2026
106 checks passed
@kibanamachine

Copy link
Copy Markdown
Contributor

Starting backport for target branches: 9.4

https://github.com/elastic/kibana/actions/runs/26940432770

@mbondyra mbondyra deleted the vis_skill_esql_fix branch June 4, 2026 08:37
@kibanamachine

Copy link
Copy Markdown
Contributor

💔 All backports failed

Status Branch Result
9.4 Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 272493

Questions ?

Please refer to the Backport tool documentation

@kibanamachine kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Jun 8, 2026
@kibanamachine

Copy link
Copy Markdown
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 272493 locally
cc: @mbondyra

3 similar comments
@kibanamachine

Copy link
Copy Markdown
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 272493 locally
cc: @mbondyra

@kibanamachine

Copy link
Copy Markdown
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 272493 locally
cc: @mbondyra

@kibanamachine

Copy link
Copy Markdown
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 272493 locally
cc: @mbondyra

tfcmarques pushed a commit to tfcmarques/kibana that referenced this pull request Jun 11, 2026
…neration (elastic#272493)

## Summary

Fixes ES|QL escaping drift during visualization config generation. 

## What Was Not Working
`generate_esql` produced the correct query for a GROK pattern:

```esql
| GROK message "%{IP} - - \\[%{DATA}\\] \"%{WORD:method} %{DATA}\""
```

But when the visualization config was generated, the query embedded in
`visualization.data_source.query` lost one escaping level for the
bracket regex:

```esql
| GROK message "%{IP} - - \[%{DATA}\] \"%{WORD:method} %{DATA}\""
```

That happened because the prompt asked the model to manually place raw
ES|QL inside a JSON string. For ES|QL containing backslashes and quotes,
JSON serialization needs an additional escaping layer. The
model-generated visualization looked correct visually, but after JSON
parsing it no longer matched the authoritative ES|QL.

## Description
This change ensures visualization configs preserve the ES|QL query
produced by the graph. The prompt now embeds the ES|QL query as a
JSON-encoded string, preventing raw interpolation from dropping escape
levels for patterns like `GROK`.
Before:
<img width="639" height="854" alt="Screenshot 2026-06-03 at 14 25 37"
src="https://github.com/user-attachments/assets/bd339e4a-57e6-4b5a-a1fd-6e7f606027a6"
/>

After:
<img width="666" height="687" alt="Screenshot 2026-06-03 at 14 25 31"
src="https://github.com/user-attachments/assets/c06d7dd0-862c-464a-9433-841422c9fbab"
/>


## Test Plan
- `node scripts/jest
x-pack/platform/packages/shared/agent-builder/agent-builder-tools-base/visualization/graph_lens.test.ts`
- `node scripts/eslint
x-pack/platform/packages/shared/agent-builder/agent-builder-tools-base/visualization/graph_lens.ts
x-pack/platform/packages/shared/agent-builder/agent-builder-tools-base/visualization/graph_lens.test.ts
x-pack/platform/packages/shared/agent-builder/agent-builder-tools-base/visualization/prompts.ts`


### Checklist

Check the PR satisfies following conditions. 

Reviewers should verify this PR satisfies this list as well.

- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
- [ ] Review the [backport
guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)
and apply applicable `backport:*` labels.

### Identify risks

Does this PR introduce any risks? For example, consider risks like hard
to test bugs, performance regression, potential of data loss.

Describe the risk, its severity, and mitigation for each identified
risk. Invite stakeholders and evaluate how to proceed before merging.

- [ ] [See some risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)
- [ ] ...
@kibanamachine

Copy link
Copy Markdown
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 272493 locally
cc: @mbondyra

8 similar comments
@kibanamachine

Copy link
Copy Markdown
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 272493 locally
cc: @mbondyra

@kibanamachine

Copy link
Copy Markdown
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 272493 locally
cc: @mbondyra

@kibanamachine

Copy link
Copy Markdown
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 272493 locally
cc: @mbondyra

@kibanamachine

Copy link
Copy Markdown
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 272493 locally
cc: @mbondyra

@kibanamachine

Copy link
Copy Markdown
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 272493 locally
cc: @mbondyra

@kibanamachine

Copy link
Copy Markdown
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 272493 locally
cc: @mbondyra

@kibanamachine

Copy link
Copy Markdown
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 272493 locally
cc: @mbondyra

@kibanamachine

Copy link
Copy Markdown
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 272493 locally
cc: @mbondyra

logeekal pushed a commit to logeekal/kibana that referenced this pull request Jun 25, 2026
…neration (elastic#272493)

## Summary

Fixes ES|QL escaping drift during visualization config generation. 

## What Was Not Working
`generate_esql` produced the correct query for a GROK pattern:

```esql
| GROK message "%{IP} - - \\[%{DATA}\\] \"%{WORD:method} %{DATA}\""
```

But when the visualization config was generated, the query embedded in
`visualization.data_source.query` lost one escaping level for the
bracket regex:

```esql
| GROK message "%{IP} - - \[%{DATA}\] \"%{WORD:method} %{DATA}\""
```

That happened because the prompt asked the model to manually place raw
ES|QL inside a JSON string. For ES|QL containing backslashes and quotes,
JSON serialization needs an additional escaping layer. The
model-generated visualization looked correct visually, but after JSON
parsing it no longer matched the authoritative ES|QL.

## Description
This change ensures visualization configs preserve the ES|QL query
produced by the graph. The prompt now embeds the ES|QL query as a
JSON-encoded string, preventing raw interpolation from dropping escape
levels for patterns like `GROK`.
Before:
<img width="639" height="854" alt="Screenshot 2026-06-03 at 14 25 37"
src="https://github.com/user-attachments/assets/bd339e4a-57e6-4b5a-a1fd-6e7f606027a6"
/>

After:
<img width="666" height="687" alt="Screenshot 2026-06-03 at 14 25 31"
src="https://github.com/user-attachments/assets/c06d7dd0-862c-464a-9433-841422c9fbab"
/>


## Test Plan
- `node scripts/jest
x-pack/platform/packages/shared/agent-builder/agent-builder-tools-base/visualization/graph_lens.test.ts`
- `node scripts/eslint
x-pack/platform/packages/shared/agent-builder/agent-builder-tools-base/visualization/graph_lens.ts
x-pack/platform/packages/shared/agent-builder/agent-builder-tools-base/visualization/graph_lens.test.ts
x-pack/platform/packages/shared/agent-builder/agent-builder-tools-base/visualization/prompts.ts`


### Checklist

Check the PR satisfies following conditions. 

Reviewers should verify this PR satisfies this list as well.

- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
- [ ] Review the [backport
guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)
and apply applicable `backport:*` labels.

### Identify risks

Does this PR introduce any risks? For example, consider risks like hard
to test bugs, performance regression, potential of data loss.

Describe the risk, its severity, and mitigation for each identified
risk. Invite stakeholders and evaluate how to proceed before merging.

- [ ] [See some risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)
- [ ] ...
@kibanamachine

Copy link
Copy Markdown
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 272493 locally
cc: @mbondyra

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport missing Added to PRs automatically when the are determined to be missing a backport. backport:version Backport to applied version labels release_note:fix Team:Presentation Presentation Team for Dashboard, Input Controls, and Canvas t// v9.4.3 v9.5.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants