Commit b50f897
Add esql method to Saved Objects repository and client (elastic#255866)
## Summary
Fixes elastic/kibana-team#2946
- Adds `esql` method to `ISavedObjectsRepository` and
`SavedObjectsClientContract` for executing ES|QL queries against saved
objects
- Injects space/namespace security filtering via
`EsqlQueryRequest.filter` using the same `getNamespacesBoolFilter()`
pattern as `search`
- Strips encrypted attribute values from ES|QL results (replaces with
`null`)
- Includes unit tests, tutorial documentation, and example plugin route
- Uses `@kbn/esql-language` `esql` tagged template with named param
syntax (`${{ name: value }}`) for injection-safe query construction
## Test plan
- [ ] Unit tests pass: `yarn test:jest
src/core/packages/saved-objects/api-server-internal/src/lib/apis/esql.test.ts`
- [ ] Typecheck passes for affected packages
- [ ] Manual verification: start Kibana with example plugins, call
`/api/saved_objects_example/_esql_query`
- [ ] Verify namespace security: queries in space A should not return
results from space B
- [ ] Verify encrypted attributes are stripped in response columns
<details>
<summary>PRD: ES|QL Search for Saved Objects</summary>
# PRD: ES|QL Search for Saved Objects
## Introduction
Add an `esql` method to the Saved Objects client and repository,
following the same pattern as the existing `search` method ([PR
elastic#239432](elastic#239432)). Like `search`,
the `esql` method avoids the translation/indirection layer of the `find`
method to give consumers access to the full power of ES|QL's processing
commands. The consumer specifies saved object types (as with `search`
and `find`), and the system handles index resolution (via
`getIndicesForTypes`) and security filtering (via
`EsqlQueryRequest.filter`). The consumer writes only the ES|QL
processing pipeline (everything after `FROM`), keeping index names as an
implementation detail they never need to know.
### Prior Art
- **`find` method:** Exposes a saved-object-specific data model that
translates queries and documents. This indirection requires rewriting
the whole API surface of Elasticsearch.
- **`search` method** ([PR
elastic#239432](elastic#239432)): Avoids this
translation to directly support the full ES `_search` DSL. Security is
maintained by injecting a `namespacesBoolFilter` via
`mergeUserQueryWithNamespacesBool`. Includes a
[tutorial](elastic#241235) and [security
best practices
documentation](elastic#246064).
- **`esql` method (this PRD):** Repeats the `search` pattern but for
ES|QL queries via Elasticsearch's `esql.query` API.
## Goals
- Add `esql` method to `ISavedObjectsRepository` and
`SavedObjectsClientContract`
- Accept an ES|QL processing `pipeline` (everything after FROM) — the
`FROM` clause is auto-generated from the `type` parameter via
`getIndicesForTypes()`
- Inject space/namespace security filtering via the
`EsqlQueryRequest.filter` parameter (includes type restriction —
consumers don't need `WHERE type ==`)
- Enforce RBAC authorization using the same
`securityExtension.authorizeFind` pattern as `search`
- Handle hidden types consistently with `search` and `find`
- Return raw ES|QL responses (`EsqlQueryResponse`) without translation
- Document usage, security risks, and the `@kbn/esql-language` `esql`
tagged template composer for injection prevention
## User Stories
### US-001: Define `SavedObjectsEsqlOptions` types and add `esql` to
`ISavedObjectsRepository`
**Description:** As a Kibana core developer, I want to define the ES|QL
types and add an `esql` method to the saved objects repository interface
so that plugins can execute ES|QL queries against saved objects with
security enforced.
**Acceptance Criteria:**
- [ ] New type file at `api-server/src/apis/esql.ts` defining
`SavedObjectsEsqlOptions` and `SavedObjectsEsqlResponse`
- [ ] `SavedObjectsEsqlOptions` extends `Omit<estypes.EsqlQueryRequest,
'format' | 'columnar' | 'delimiter' | 'query'>`, adding:
- `type: string | string[]` — saved object types to query (used for
index resolution and security)
- `namespaces: string[]` — spaces to query within
- `pipeline: string` — the ES|QL processing pipeline (everything after
FROM, e.g., `'| KEEP dashboard.title | LIMIT 100'`)
- `metadata?: string[]` — optional METADATA fields on the FROM clause
(e.g., `['_id', '_source']`)
- [ ] `SavedObjectsEsqlResponse` is a type alias for
`estypes.EsqlQueryResponse` (aliased from `EsqlEsqlResult` in ES client
types)
- [ ] `ISavedObjectsRepository` interface (in
`api-server/src/saved_objects_repository.ts`) has a new `esql` method
with signature: `esql(options: SavedObjectsEsqlOptions):
Promise<SavedObjectsEsqlResponse>`
- [ ] Types are exported from `api-server/src/apis/index.ts` barrel and
`api-server/index.ts`
- [ ] Typecheck/lint passes
**Implementation notes:**
- The `query` field from `EsqlQueryRequest` is omitted because the full
query is constructed internally by combining `FROM <indices>` (resolved
from `type` via `getIndicesForTypes()`) with the consumer's `pipeline`
- Also omits `format`, `columnar`, and `delimiter` (not relevant for
saved objects)
- `estypes.EsqlQueryResponse` is internally aliased to `EsqlEsqlResult`
in the ES client types
- The `filter` field on `EsqlQueryRequest` is `QueryDslQueryContainer` —
this is where security filters are injected (includes type restriction,
so consumers don't need `WHERE type ==` in their pipeline)
### US-002: Add `esql` to `SavedObjectsClientContract`, mocks, and
repository stub
**Description:** As a Kibana plugin developer, I want the `esql` method
available on the saved objects client so that I can use it from route
handlers via `ctx.core.savedObjects.client.esql()`.
**Acceptance Criteria:**
- [ ] `SavedObjectsClientContract` interface (in
`api-server/src/saved_objects_client.ts`) has the `esql` method with
JSDoc warning about security risks (matching `search` method's remarks)
- [ ] `SavedObjectsClient` implementation (in
`api-server-internal/src/saved_objects_client.ts`) delegates to the
repository's `esql` method
- [ ] Repository mock (`api-server-mocks/src/repository.mock.ts`)
includes `esql` as a jest mock
- [ ] Internal repository mock
(`api-server-internal/src/mocks/repository.mock.ts`) includes `esql` as
a jest mock — **there are TWO repository mock files that must stay in
sync**
- [ ] Client mock (`api-server-mocks/src/saved_objects_client.mock.ts`)
includes `esql` as a jest mock
- [ ] `SavedObjectsRepository` class gets a **stub** `esql` method
(throws "not implemented") to satisfy the interface — the real
implementation is wired in US-004
- [ ] Typecheck/lint passes
**Implementation notes:**
- `SavedObjectsClient` in `api-server-internal` simply delegates to the
repository, following the pattern of all other methods
- Adding a method to `ISavedObjectsRepository` requires updating the
`SavedObjectsRepository` class immediately (even with a stub) to pass
typecheck — the stub is replaced in US-004
### US-003: Implement `performEsql` in the repository internals
**Description:** As a Kibana core developer, I want the internal
implementation of the `esql` method to execute ES|QL queries via the
Elasticsearch client while injecting space and type security filters and
auto-generating the FROM clause.
**Acceptance Criteria:**
- [ ] New `performEsql` function in
`api-server-internal/src/lib/apis/esql.ts`, following the pattern of
`performSearch` in `search.ts`
- [ ] Accepts a `PerformEsqlParams` object with `options:
SavedObjectsEsqlOptions` and `rawClient: ElasticsearchClient` (see
implementation notes)
- [ ] Validates `options.namespaces` is not empty (throws
`BadRequestError` if empty)
- [ ] Validates `options.type` contains at least one allowed type
(returns empty response `{ columns: [], values: [] }` if none match
`allowedTypes`)
- [ ] Resolves indices from types via
`commonHelper.getIndicesForTypes(types)` and constructs the `FROM
<indices>` clause (with optional `METADATA` fields if `options.metadata`
is provided)
- [ ] Constructs the full ES|QL query by concatenating the FROM clause
with the consumer's `pipeline`
- [ ] Validates that `pipeline` does not start with a source command
(`FROM`, `ROW`, `SHOW`, `METRICS`) — throws `BadRequestError` if it does
- [ ] Uses `spacesExtension.getSearchableNamespaces()` to resolve
authorized namespaces; returns empty response on 403 Boom errors
- [ ] Uses `securityExtension.authorizeFind()` to check RBAC, supporting
`partially_authorized` via `typeToNamespacesMap`
- [ ] Constructs the namespaces bool filter via
`getNamespacesBoolFilter()` (imported from `../search`) — this filter
already includes type restriction
- [ ] Injects the filter into `EsqlQueryRequest.filter` — merging with
any user-provided filter using `{ bool: { must: [namespacesBoolFilter,
userFilter] } }`
- [ ] Calls `rawClient.esql.query()` with the constructed query and
filter
- [ ] Returns the raw `EsqlQueryResponse`
- [ ] Exported from `apis/index.ts` barrel file
- [ ] Typecheck/lint passes
**Implementation notes:**
- **Critical:** `RepositoryEsClient` only wraps top-level ES methods
(`search`, `get`, `bulk`, etc.) — it does NOT expose `esql.query()`. The
raw `ElasticsearchClient` must be passed separately via the `rawClient`
parameter. It's available in the repository as `this.options.client`
- Index resolution uses `commonHelper.getIndicesForTypes(types)` — same
as `search` and `find`. The FROM clause is constructed as `FROM
${indices.join(', ')}` (with `METADATA ${metadata.join(', ')}` appended
if specified)
- The consumer's `pipeline` is appended to the FROM clause. If the
pipeline starts with `|`, the full query is `FROM <indices>
${pipeline}`. If it doesn't start with `|`, prepend a space.
- `getNamespacesBoolFilter` and the filter-merging pattern can be reused
directly from `search.ts`
- The namespace bool filter already restricts to the specified types —
consumers don't need `WHERE type ==` in their pipeline
- ES|QL uses `filter` field (not `query`) for injecting security filters
— this aligns with how `getNamespacesBoolFilter` works
### US-004: Wire `esql` into the repository implementation
**Description:** As a Kibana core developer, I want the repository class
to delegate `esql` calls to the `performEsql` function with the correct
execution context.
**Acceptance Criteria:**
- [ ] `SavedObjectsRepository.esql()` method (in
`api-server-internal/src/lib/repository.ts`) replaces the stub with a
call to `performEsql({ options, rawClient: this.options.client },
this.apiExecutionContext)`
- [ ] Typecheck/lint passes
**Implementation notes:**
- Raw ES client is available via `this.options.client` in the repository
(constructor stores options as `private readonly`)
- Prettier enforces single-line for short function calls — don't split
across multiple lines
### US-005: Strip/redact encrypted attributes from ES|QL results
**Description:** As a Kibana core developer, I want encrypted saved
object attributes to be stripped from ES|QL query results so that
secrets are never exposed in tabular responses.
**Acceptance Criteria:**
- [ ] Add `getEncryptedAttributes(type: string): ReadonlySet<string> |
undefined` to `ISavedObjectsEncryptionExtension` interface (in
`server/src/extensions/encryption.ts`)
- [ ] Implement `getEncryptedAttributes` on
`SavedObjectsEncryptionExtension` class (in `encrypted_saved_objects`
plugin) — delegates to `EncryptedSavedObjectsService`
- [ ] Implement `getEncryptedAttributes` on
`EncryptedSavedObjectsService` — returns the `ReadonlySet` from the
type's definition stored in the private `Map`
- [ ] Update all three encryption extension mocks: `api-server-mocks`,
`api-server-internal/mocks`, and ESO plugin service mocks
- [ ] Add `stripEncryptedColumns` function to `performEsql` that nulls
encrypted column values
- [ ] ES|QL columns for SO attributes use the pattern
`<type>.<attribute_name>` — match against this pattern to identify
encrypted columns
- [ ] Replace values in encrypted columns with `null` (preserving column
structure so row indices remain stable)
- [ ] If no encryption extension exists or all requested types are
non-encryptable, skip post-processing entirely (fast path)
- [ ] Works correctly for mixed-type queries where only some types have
encrypted attributes
- [ ] Document in the tutorial that encrypted fields are stripped (not
decrypted) because ES|QL returns tabular data and AAD reconstruction
from columns/values is not feasible
- [ ] Typecheck/lint passes
**Implementation notes:**
- **Critical:** `ISavedObjectsEncryptionExtension` did NOT have
`getEncryptedAttributes` — it must be added to the core interface AND
implemented in the ESO plugin
- `EncryptedSavedObjectsService` stores type definitions in a private
`Map`; `getEncryptedAttributes` returns the `ReadonlySet` from the
definition
- Three encryption mock files need updating:
`api-server-mocks/src/saved_objects_extensions.mock.ts`,
`api-server-internal/src/mocks/saved_objects_extensions.mock.ts`, and
`encrypted_saved_objects/server/crypto/encrypted_saved_objects_service.mocks.ts`
### US-006: Write unit tests for `performEsql`
**Description:** As a Kibana core developer, I want comprehensive unit
tests so that I can verify the FROM clause generation, security
filtering, authorization, encryption stripping, and error handling of
the `esql` method.
**Acceptance Criteria:**
- [ ] Test file at `api-server-internal/src/lib/apis/esql.test.ts`
- [ ] Tests that the FROM clause is auto-generated from `type` via
`getIndicesForTypes()`
- [ ] Tests that `metadata` fields are appended to the FROM clause
(e.g., `FROM .kibana METADATA _id`)
- [ ] Tests that `pipeline` is appended after the FROM clause
- [ ] Tests that a pipeline starting with a source command (`FROM`,
`ROW`, `SHOW`, `METRICS`) throws `BadRequestError`
- [ ] Tests that namespaces filter is injected into
`EsqlQueryRequest.filter`
- [ ] Tests that user-provided `filter` is merged (not replaced) with
the security filter
- [ ] Tests that empty `namespaces` throws `BadRequestError`
- [ ] Tests that unauthorized types return empty response
- [ ] Tests that `partially_authorized` correctly restricts types to
authorized namespaces
- [ ] Tests that hidden types are filtered from allowed types
- [ ] Tests that encrypted attribute columns have their values replaced
with `null`
- [ ] Tests that non-encrypted columns are returned unmodified
- [ ] Typecheck/lint passes
**Implementation notes:**
- **Critical:** Must test through the `SavedObjectsRepository` class
(not call `performEsql` directly) because `getNamespacesBoolFilter` from
`../search` doesn't resolve correctly in direct Jest imports
- Follow the pattern of `search.test.ts`: use `@ts-expect-error` to
access private constructor
- Use `client.esql.query.mockResolvedValue()` to mock ES|QL responses
- Verify the constructed `query` string passed to `client.esql.query()`
includes the correct FROM clause + pipeline
- The ES client TypeScript types define `EsqlESQLParam` as `FieldValue |
FieldValue[]` (positional), but ES also supports named params as
`Record<string, unknown>` at runtime (the types are incomplete). The
`@kbn/esql-language` composer produces named params.
### US-007: Write tutorial documentation
**Description:** As a Kibana plugin developer, I want a tutorial doc so
that I can understand how to use the `esql` method, including the
pipeline concept, security model, and the `esql` tagged template
composer for avoiding injection attacks.
**Acceptance Criteria:**
- [ ] New tutorial at `dev_docs/tutorials/saved_objects_esql.mdx`
(following `saved_objects_search.mdx` pattern)
- [ ] Explains the relationship between `find`, `search`, and `esql`
methods
- [ ] Explains the `pipeline` concept: consumers write only the ES|QL
processing pipeline (everything after FROM), the system handles index
resolution from `type` and security filtering. No `FROM` clause or
`WHERE type ==` needed.
- [ ] Shows a basic usage example in a route handler (using
`savedObjectsClient.esql()` with `pipeline`)
- [ ] Shows how to use `metadata` for `METADATA _id, _source` on the
FROM clause
- [ ] Explains how space and type security filtering is injected via the
`filter` parameter — emphasizes that the DSL filter already restricts to
the specified types
- [ ] Documents the `esql` tagged template from `@kbn/esql-language` as
the recommended way to build pipelines with user input — specifically
the **named param syntax** `${{ name: value }}` which creates proper
`?name` parameterized placeholders (code/data separation at the protocol
level). The plain `${value}` syntax only provides quoting-based
protection (inlines as escaped literal) which is less safe.
- [ ] Shows how to extract pipeline and params from the composer: use
`.print()` for the pipeline string and `.getParams()` for the named
params, converting to the format expected by `SavedObjectsEsqlOptions`
- [ ] Documents the type gap: the composer's `.toRequest()` method
returns `{ filter?: unknown, params?: EsqlRequestParams }` which doesn't
match the ES client's `{ filter?: QueryDslQueryContainer, params?:
EsqlESQLParam[] }`. This is a TypeScript-level issue only — ES|QL named
params work at runtime. Bridge with a type assertion or by extracting
`.print()` + `.getParams()` separately
- [ ] Documents that encrypted fields are stripped (not decrypted) in
responses
- [ ] Links to the Elasticsearch ES|QL documentation
- [ ] Typecheck/lint passes (mdx builds)
**Implementation notes:**
- The `@kbn/esql-language` `esql` composer has two template hole modes —
document both clearly:
1. **`${{ name: value }}`** (named param shorthand) — creates a `?name`
placeholder in the query and stores the value in the params bag. This is
true parameterization: code and data are separated at the protocol
level. **This is the recommended approach for user input.**
2. **`${value}`** (plain synth hole) — inlines the value into the query
string as an escaped literal (strings get proper `"` quoting with `\`
escaping via `LeafPrinter.string()`). Provides quoting-based protection
but is NOT true parameterization. Acceptable for developer-controlled
values, not for user input.
- The composer's `.toRequest()` types have a gap with the ES client
types (`filter?: unknown` vs `QueryDslQueryContainer`, and
`EsqlRequestParams` includes `Record<string, unknown>` entries while
`EsqlESQLParam` is `FieldValue | FieldValue[]`). This is a
TypeScript-level issue only — Elasticsearch supports named params at
runtime. To bridge: either use a type assertion on `.toRequest()`, or
extract `.print()` for query and convert `.getParams()` to the params
array format manually.
- Existing codebase callers (e.g., `kbn-index-editor`) use the composer
purely for query string construction via `.print()` and `.pipe`, not via
`.toRequest()`. This is fine for developer-controlled values but doesn't
leverage parameterization.
- Note: since the consumer writes only the pipeline (not a full query
with FROM), the composer is used to build the pipeline string, not the
full query. The FROM clause is added by `performEsql` internally.
- When simplifying examples, remember to remove unused package
references from `tsconfig.json`
### US-008: Add example plugin route
**Description:** As a Kibana plugin developer, I want a working example
so that I can see the `esql` method used in a real route handler with
the pipeline API.
**Acceptance Criteria:**
- [ ] New route file
`examples/saved_objects/server/esql_example_routes.ts` (following
`search_example_routes.ts` pattern)
- [ ] Route registered in `examples/saved_objects/server/plugin.ts`
- [ ] Demonstrates the `esql` method with `type`, `namespaces`, and
`pipeline` — no FROM clause or WHERE type == in the pipeline
- [ ] Shows a simple pipeline example (e.g., `| KEEP dashboard.title |
SORT dashboard.title | LIMIT 100`)
- [ ] Shows usage of `@kbn/esql-language` `esql` tagged template with
named param syntax (`${{ name: value }}`) for safe pipeline construction
when interpolating user input
- [ ] Shows usage of `metadata` option for METADATA fields
- [ ] Includes error handling with `isResponseError` from
`@kbn/es-errors`
- [ ] Typecheck/lint passes
**Implementation notes:**
- Use the `esql` tagged template with `${{ name: value }}` named param
syntax for user input. Bridge the type gap by extracting `.print()` and
`.getParams()` separately (see US-007 implementation notes)
- Convert `.getParams()` (returns `Record<string, unknown>`) to the
`EsqlESQLParam[]` format expected by the ES client:
`Object.entries(params).map(([k, v]) => ({ [k]: v }))`
- The pipeline should NOT include FROM or WHERE type == — these are
handled automatically
### US-009: End-to-end integration tests for the esql method
**Description:** As a Kibana core developer, I want integration tests
that exercise the `esql` method against a real Elasticsearch instance so
that I can verify the full flow — index resolution, namespace security
filtering, type filtering, ES|QL query execution, and encrypted
attribute stripping — works end-to-end, not just in unit tests with
mocked clients.
**Acceptance Criteria:**
- [ ] New integration test file at
`src/core/server/integration_tests/saved_objects/service/lib/esql.test.ts`
(following the pattern of `search.test.ts`)
- [ ] Test setup: boot a real ES + Kibana root using `createTestServers`
and `createRootWithCorePlugins`, register a `test-type` with
`namespaceType: 'multiple'` and mappings for `name` (text) and `email`
(keyword)
- [ ] Seed data: bulk-create test documents across `default` namespace
and a second namespace (e.g., `namespaceA`)
- [ ] Test: basic ES|QL query returns expected columns and values for
the default namespace
- [ ] Test: namespace scoping — querying `namespaceA` returns only
documents from that namespace
- [ ] Test: pipeline commands work (e.g., `| WHERE`, `| SORT`, `|
LIMIT`, `| STATS COUNT(*)`)
- [ ] Test: pipeline starting with a source command (FROM, ROW) is
rejected with BadRequestError
- [ ] Test: `metadata` option (e.g., `metadata: ['_id']`) includes
metadata fields in the response
- [ ] Test: user-provided `filter` is merged with namespace filter
(results are filtered by both)
- [ ] All tests pass against a real Elasticsearch instance
- [ ] Typecheck/lint passes
**Implementation notes:**
- Follow the exact pattern of `search.test.ts`: use `createTestServers`
to start ES, `createRootWithCorePlugins` for the Kibana root,
`registerTypes` during setup,
`start.savedObjects.createInternalRepository()` for the repository
- The integration test config is auto-discovered from
`jest.integration.config.js` — run with `yarn test:jest_integration`
- ES|QL responses have `columns` (array of `{ name, type }`) and
`values` (2D array) — assert on both
- Use `setProxyInterrupt(null)` if reusing `repository_with_proxy_utils`
(as `search.test.ts` does), or omit if not needed
- ES|QL queries on saved objects use the `<type>.<field>` column naming
convention (e.g., `test-type.name`)
- The `createInternalRepository()` doesn't have spaces or security
extensions, so namespace filtering is purely filter-based in integration
tests (no RBAC paths). This is acceptable — RBAC paths are covered by
unit tests.
## Functional Requirements
- FR-1: Add `esql` method to `ISavedObjectsRepository` and
`SavedObjectsClientContract` interfaces
- FR-2: `SavedObjectsEsqlOptions` extends
`Omit<estypes.EsqlQueryRequest, 'format' | 'columnar' | 'delimiter' |
'query'>`, adding `type`, `namespaces`, `pipeline`, and optional
`metadata`
- FR-3: Auto-generate the `FROM <indices>` clause from `type` via
`commonHelper.getIndicesForTypes(types)`, appending `METADATA` fields if
specified, and concatenating the consumer's `pipeline`
- FR-4: Validate that `pipeline` does not start with a source command
(`FROM`, `ROW`, `SHOW`, `METRICS`) — throw `BadRequestError` if it does
- FR-5: Inject space/namespace security via `EsqlQueryRequest.filter`
using `getNamespacesBoolFilter()` — the filter already includes type
restriction (consumers don't need `WHERE type ==`)
- FR-6: Merge any user-provided `filter` with the security filter using
`{ bool: { must: [namespacesBoolFilter, userFilter] } }`
- FR-7: Execute via `rawClient.esql.query()` — NOT `RepositoryEsClient`
(which doesn't expose `esql.query()`)
- FR-8: Return `EsqlQueryResponse` (columns + values format) with
encrypted columns stripped
- FR-9: Filter hidden types from the `type` array, returning empty
response if no valid types remain
- FR-10: Support partial authorization via `typeToNamespacesMap` when
`securityExtension.authorizeFind` returns `partially_authorized`
- FR-11: Return empty response (not error) when user has no access to
any requested namespace or type
- FR-12: Post-process ES|QL results to identify columns matching
registered encrypted attributes (via
`encryptionExtension.isEncryptableType()` and new
`encryptionExtension.getEncryptedAttributes()`), replacing their values
with `null`
- FR-13: Document that the pipeline executes with `kibana_system` user
privileges and warn against injecting arbitrary user input into the
pipeline
## Non-Goals
- No HTTP route endpoint in this iteration — the method is added to the
repository/client for plugin consumption (plugins create their own
routes)
- No UI for composing or running ES|QL queries
- No saved/shareable query support
- No write operations (ES|QL is read-only)
- No decryption of encrypted attributes — ES|QL returns tabular data
where AAD (type + id + namespace + `attributesToIncludeInAAD` values)
cannot be reliably reconstructed per row. Encrypted columns are
stripped/nulled instead.
- No migration processing of results — ES|QL returns tabular
(columns/values) data, not raw saved object documents, so
`migrationHelper` processing from `search` does not apply
- No column mapping or renaming — users query against raw document
fields (e.g., `dashboard.title`, not `title`)
- No support for ES|QL source commands other than FROM (`ROW`, `SHOW`,
`METRICS`) — use the raw ES client for those
- No `esql.asyncQuery()` support — only synchronous `esql.query()` in
this iteration
## Technical Considerations
- **ES|QL `filter` parameter:** The `EsqlQueryRequest.filter` field
accepts a `QueryDslQueryContainer` that restricts documents before the
ES|QL query runs. This is the injection point for space/namespace
security — identical in effect to how `search` merges the namespace
filter into the `query` field. The filter includes type restriction via
`getNamespacesBoolFilter()`, so consumers don't need `WHERE type ==` in
their pipeline.
- **Index resolution from types:** Like `search` (which passes `index:
commonHelper.getIndicesForTypes(types)`), `performEsql` resolves indices
from the `type` parameter and auto-generates the `FROM <indices>`
clause. The consumer never specifies index names — they write only the
processing pipeline. This maintains the same abstraction as `search` and
`find`, where index names are an implementation detail.
- **RepositoryEsClient limitation:** `RepositoryEsClient` only wraps
top-level ES methods (`search`, `get`, `bulk`, etc.). It does NOT expose
namespaced methods like `esql.query()`. The `performEsql` function
requires the raw `ElasticsearchClient` (available as
`this.options.client` in the repository), passed via the `rawClient`
parameter in `PerformEsqlParams`.
- **Encrypted attribute stripping:** The `search` method post-processes
hits through `encryptionHelper` to decrypt individual saved objects
using AAD constructed from `stableStringify([namespace, type, id,
{aadAttributes}])`. ES|QL returns tabular data (columns + values) where
the per-row descriptor and AAD attributes cannot be reliably
reconstructed (users may omit required columns via `KEEP`/`DROP`).
Instead, `performEsql` post-processes the response to identify columns
matching registered encrypted attribute names (using the pattern
`<type>.<attribute_name>`) and replaces their values with `null`. A new
`getEncryptedAttributes` method must be added to
`ISavedObjectsEncryptionExtension` since it did not previously exist.
- **No migration processing:** ES|QL returns tabular data, not full
documents, so `migrationHelper` processing from `search` does not apply.
This is acceptable because ES|QL queries are power-user operations
against raw data.
- **Injection prevention:** Use the `@kbn/esql-language` `esql` tagged
template composer with **named param syntax** (`${{ name: value }}`) for
user input. This creates proper `?name` parameterized placeholders in
the query, separating code from data at the protocol level. The plain
`${value}` syntax only provides quoting-based protection (the synth
layer creates escaped string literals via `LeafPrinter.string()` — e.g.,
`O"Brien` → `"O\"Brien"`) which is acceptable for developer-controlled
values but NOT sufficient for untrusted user input. The composer's
`.toRequest()` has a TypeScript-level type gap with the ES client
(`filter?: unknown` vs `QueryDslQueryContainer`, and named param records
vs `EsqlESQLParam`), but this is a types-only issue — ES|QL named params
work at runtime. Bridge by extracting `.print()` + `.getParams()`
separately or using a type assertion.
- **`EsqlQueryRequest` type:** Imported from `@elastic/elasticsearch`
(`estypes.EsqlQueryRequest`). Key fields: `query` (string — constructed
internally from FROM + pipeline), `filter` (QueryDslQueryContainer),
`params` (`EsqlESQLParam[]` where `EsqlESQLParam = FieldValue |
FieldValue[]`). Note: the ES client TypeScript types define
`EsqlESQLParam` as `FieldValue | FieldValue[]` which doesn't include the
named param `Record<string, unknown>` format — but ES itself supports
named params at runtime. `SavedObjectsEsqlOptions` omits `format`,
`columnar`, `delimiter`, and `query` (replaced by `pipeline`).
- **Existing infrastructure:** Reuses `getNamespacesBoolFilter()`,
`spacesExtension.getSearchableNamespaces()`,
`securityExtension.authorizeFind()`, and
`commonHelper.getIndicesForTypes()` from the `search` implementation.
- **Testing constraints:** Unit tests must go through
`SavedObjectsRepository` (not call `performEsql` directly) because Jest
module resolution doesn't work with the `../search` relative import for
`getNamespacesBoolFilter`. Follow `search.test.ts` pattern using
`@ts-expect-error` to access the private constructor.
- **Mock files:** There are TWO repository mock files
(`api-server-mocks/src/repository.mock.ts` and
`api-server-internal/src/mocks/repository.mock.ts`) and THREE encryption
extension mock files that must all be kept in sync.
## Success Metrics
- Plugin developers can call `savedObjectsClient.esql()` with an ES|QL
pipeline and receive correct, space-scoped results without needing to
know index names or write type filters
- Security filtering is enforced identically to the `search` method — no
unauthorized cross-space data leakage
- Unit tests cover all authorization paths (fully authorized, partially
authorized, unauthorized)
- Tutorial documentation enables a developer to write their first query
without additional support
## Resolved Questions
- **Does the ES|QL API support an explicit `index` parameter?** No. The
data source must be specified in the `FROM` clause of the query string.
The implementation auto-generates the FROM clause from the `type`
parameter via `getIndicesForTypes()`, maintaining the same abstraction
as `search` and `find`.
- **Should encrypted saved object types be excluded entirely from ES|QL
queries?** No. Encrypted types are allowed, but columns matching
registered `attributesToEncrypt` are stripped (values replaced with
`null`). Decryption is not feasible because AAD reconstruction from
tabular data is unreliable.
- **Should we use `@kbn/esql-language` tagged template?** Yes —
specifically the named param syntax `${{ name: value }}` which creates
proper `?name` parameterized placeholders (true code/data separation).
The composer's `.toRequest()` has a TypeScript-level type gap with the
ES client types (`filter?: unknown` vs `QueryDslQueryContainer`, named
param `Record<string, unknown>` vs `EsqlESQLParam`), but this is
types-only — ES supports named params at runtime. Bridge by extracting
`.print()` + `.getParams()` separately or using a type assertion on
`.toRequest()`.
- **Should we support `esql.asyncQuery()`?** Not in this iteration —
only synchronous `esql.query()`.
- **Does `ISavedObjectsEncryptionExtension` have
`getEncryptedAttributes`?** No — it must be added as part of US-005,
along with the implementation in the ESO plugin.
## Open Questions
- How should the `search` tutorial be updated to cross-reference the new
`esql` tutorial?
## Change Log
### v2 — Recommend `@kbn/esql-language` composer for injection
prevention
After investigating the `@kbn/esql-language` `esql` tagged template
composer, we corrected the initial assessment that it was incompatible
with the ES client. The composer has two template hole modes:
1. **`${{ name: value }}`** (named param shorthand) — creates `?name`
placeholders in the query with values stored separately. This is true
parameterization (code/data separation at the protocol level).
**Recommended for user input.**
2. **`${value}`** (plain synth hole) — inlines the value as an escaped
literal via `LeafPrinter.string()`. Quoting-based protection only.
Acceptable for developer-controlled constants.
The `.toRequest()` method has a TypeScript-level type gap with the ES
client (`filter?: unknown` vs `QueryDslQueryContainer`, named param
`Record<string, unknown>` vs `EsqlESQLParam`), but this is types-only —
ES supports named params at runtime. Bridge by extracting `.print()` +
`.getParams()` separately or using `as unknown as` type assertion.
Also fixed the tutorial's incorrect claim that the index is controlled
server-side (it is not — ES|QL uses the FROM clause). Fixed the
positional params example which incorrectly used typed objects `{ type:
'keyword', value: ... }` instead of plain `FieldValue`.
**Files changed:** `dev_docs/tutorials/saved_objects_esql.mdx`,
`examples/saved_objects/server/esql_example_routes.ts`,
`examples/saved_objects/tsconfig.json`
### v3 — Replace `query` with `pipeline`, auto-generate FROM clause from
types
Design review identified that the v1 API broke the saved objects
abstraction: consumers had to write `FROM .kibana` (an implementation
detail) and redundantly filter `WHERE type == "dashboard"` (already
enforced by the DSL filter). This is inconsistent with `search` and
`find`, where consumers specify types as a dedicated parameter and never
see index names.
**Key changes:**
- **Rename `query` → `pipeline`** in `SavedObjectsEsqlOptions`. The
consumer writes only the processing pipeline (everything after FROM): `|
KEEP dashboard.title | SORT dashboard.title | LIMIT 100`
- **Auto-prepend `FROM <indices>`** using
`commonHelper.getIndicesForTypes(types)`, matching how `search` resolves
`index: commonHelper.getIndicesForTypes(types)`
- **Add optional `metadata` field** (`string[]`) to
`SavedObjectsEsqlOptions` for `METADATA _id, _source` on the FROM clause
- **Remove redundant `WHERE type ==`** from all examples — the DSL
filter already restricts to the specified types
- **Validate/reject queries starting with source commands** (FROM, ROW,
SHOW, METRICS) — if consumers need those, they should use the raw ES
client directly
**Design rationale:** The `esql` method provides a secure execution
context scoped to saved object types. The `type` param controls: (1)
index resolution → FROM clause, (2) security filter → DSL filter with
type+namespace restriction, (3) RBAC → authorizeFind, (4) encrypted attr
stripping → post-processing. FROM is derived plumbing, not a consumer
concern. The full power of ES|QL's processing commands (WHERE, KEEP,
DROP, EVAL, STATS, DISSECT, GROK, SORT, LIMIT, etc.) remains available.
This avoids the `find` anti-pattern of exposing a crippled subset of ES
features while maintaining the same mental model as `search`.
### v4 — Document `kibana_system` privilege model, do not block
ENRICH/LOOKUP
Investigation confirmed that ES|QL `ENRICH` can access data from enrich
policies whose source index the end user cannot read (requires
`monitor_enrich` or `manage_enrich` cluster privilege, which
`kibana_system` has). `LOOKUP JOIN` is not an escalation vector because
it requires `index.mode: "lookup"` on the target index.
However, this is not a novel vulnerability — it follows the same trust
model as all server-side Kibana code: the `kibana_system` user always
has elevated privileges, and developers are responsible for not leaking
data through those privileges. Blocking ENRICH/LOOKUP would be overly
restrictive; developers who use ENRICH are enriching from policies they
created with data appropriate for all users.
**Key changes:**
- **Do not block ENRICH or LOOKUP** — these are legitimate ES|QL
commands available to developers.
- **Document the privilege model** in the tutorial: the pipeline
executes as `kibana_system`, developers must never inject arbitrary user
input into the pipeline string.
- **Remove US-009** (was: block ENRICH/LOOKUP commands). Renumber US-010
→ US-009.
- **Add US-009** (was US-010) for end-to-end integration tests following
the `search.test.ts` pattern.
**Design rationale:** The developer controls the pipeline server-side.
If they use ENRICH, they are enriching from a policy whose data is
appropriate for all users who will see the results. The security
responsibility is on the developer to not pass through untrusted user
input that could include ENRICH — the same as any other query built with
the system user's privileges.
</details>
🤖 Generated with [Claude Code cli](https://claude.com/claude-code) and
Ralph loop
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* ES|QL query support for Saved Objects with namespace-aware filtering,
security checks, and client/repository APIs.
* Public API to query encrypted-attribute-aware results and to inspect
which attributes are encrypted.
* **Documentation**
* New tutorial demonstrating safe ES|QL usage, parameters, security
guidance, and examples.
* **Tests**
* Extensive unit and integration tests validating queries, namespaces,
encryption handling, and error cases.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Jeramy Soucy <jeramy.soucy@elastic.co>1 parent 856ac8a commit b50f897
23 files changed
Lines changed: 1396 additions & 0 deletions
File tree
- dev_docs/tutorials
- examples/saved_objects/server
- src/core
- packages/saved-objects
- api-server-internal/src
- lib
- apis
- mocks
- api-server-mocks/src
- api-server
- src
- apis
- server/src/extensions
- server/integration_tests/saved_objects/service/lib
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
| 19 | + | |
19 | 20 | | |
20 | 21 | | |
21 | 22 | | |
| |||
29 | 30 | | |
30 | 31 | | |
31 | 32 | | |
| 33 | + | |
32 | 34 | | |
33 | 35 | | |
34 | 36 | | |
| |||
0 commit comments