ES|QL dense vector field type support #126456

carlosdelest · 2025-04-08T11:24:48Z

Support dense_vector field type. This is the first step to allow kNN queries and having dense_vector as a first class citizen in ES|QL

This allows a mapping that has dense_vector like the following:

{
  "properties": {
    "id": {
      "type": "long"
    },
    "vector": {
      "type": "dense_vector",
      "similarity": "l2_norm"
    }
  }
}

To be retrieved via ES|QL:

FROM dense_vector
| KEEP id, vector
| SORT id

id   | vector
0    | [1.0, 2.0, 3.0]
1    | [4.0, 5.0, 6.0]

For now, just float element types are allowed. There will be a similar work in order to allow for byte and bit element types, but I wanted to review this implementation first to ensure it's in line with what we need.

Both indexed / not indexed types and synthetic source is supported.

Support for CSV tests has been added. For now CSV tests are simple, we can expand on these and also support additional operations on dense_vector field types in subsequent PRs. An integration test has been added to test extensively on different index options and doc storage structure.

dense_vector field type is under a feature flag, as this will require follow up work.

…ctor_support # Conflicts: # x-pack/plugin/esql-core/src/main/java/org/elasticsearch/xpack/esql/core/type/DataType.java # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/ResponseValueUtils.java # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/planner/LocalExecutionPlanner.java

…c fields

…multivalued fields

carlosdelest · 2025-04-08T11:32:30Z

server/src/main/java/org/elasticsearch/index/mapper/BlockDocValuesReader.java

@@ -504,6 +506,80 @@ public String toString() {
        }
    }

+    public static class DenseVectorBlockLoader extends DocValuesBlockLoader {


Added a BlockLoader for dense vectors, that uses FloatVectorValues to retrieve indexed vector data.

carlosdelest · 2025-04-08T11:33:28Z

server/src/main/java/org/elasticsearch/index/mapper/vectors/DenseVectorFieldMapper.java

+        @Override
+        public BlockLoader blockLoader(MappedFieldType.BlockLoaderContext blContext) {
+            if (elementType != ElementType.FLOAT) {
+                throw new UnsupportedOperationException("Only float dense vectors are supported for now");


We can work on this next, creating specific BlockLoaders for Byte and Bit field types.

carlosdelest · 2025-04-08T11:34:15Z

x-pack/plugin/esql/qa/testFixtures/src/main/java/org/elasticsearch/xpack/esql/CsvAssert.java

@@ -145,6 +146,10 @@ private static void assertMetadata(
                    // Type.asType translates all bytes references into keywords
                    continue;
                }
+                if (blockType == Type.DOUBLE && expectedType == DENSE_VECTOR) {
+                    // DENSE_VECTOR is internally represented as a double block


This could potentially change when we support byte and bit element types - we could create the appropriate blocks.

carlosdelest · 2025-04-08T11:35:31Z

x-pack/plugin/esql/qa/testFixtures/src/main/resources/mapping-default-incompatible.json

@@ -63,18 +63,7 @@
            "type" : "keyword"
        },
        "salary_change": {
-            "type": "float",


Changing CSV loading made this a problem, as there were parsing exceptions when trying to index float numeric data into integers. I didn't see a convenient way out of this and decided to remove as this field is not being tested.

If we can keep this unchanged, that will be great, this is a good example of nested fields.

It is just changed for the mapping-default-incompatible mapping, which was created to test some incompatible field mappings that did not include subfields. I'll try to give this another shot but it will require changes to the CSV loader or the dataset 😢

It is just changed for the mapping-default-incompatible mapping, which was created to test some incompatible field mappings that did not include subfields. I'll try to give this another shot but it will require changes to the CSV loader or the dataset 😢

Is it easier if we make another copy of employees's schema and data for dense_vector related tests?

Is it easier if we make another copy of employees's schema and data for dense_vector related tests?

The problem is that changing how the CSV tests load data impacted this dataset. Before this change, multivalues were being uploaded as arrays of strings, which is something we don't want to do for dense_vectors as that is not a format supported on the DenseVectorFieldMapper.

It seemed like too much work to change the actual dataset when that particular field is not actually used in the tests.

I'm open to other solutions here!

@fang-xing-esql are these fields used in other tests somehow?
I see this was actually added by @carlosdelest a while back #117555.
The employees_incompatible index that is set with this mapping is only used in match function/operator tests.
We don't modify any of those tests here, so this looks like a safe change to me.

…support' into feature/esql_dense_vector_support

… values

carlosdelest · 2025-05-06T08:44:28Z

Closing this PR, as the final approach will imply multiple field types for the different dense_vector element types.

…ctor_support # Conflicts: # x-pack/plugin/build.gradle # x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/data/BlockUtils.java # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/planner/PlannerUtils.java

…support' into feature/esql_dense_vector_support

carlosdelest · 2025-05-23T18:00:40Z

Reopening this, as we've decided to go for a single dense_vector field type. We can reevaluate the need for multiple dense_vector element types later.

carlosdelest · 2025-05-23T18:03:57Z

@ioanatia @ChrisHegarty @fang-xing-esql I intend to merge this PR on Monday to provide initial support for dense_vector float element types. Please comment if you have any issue with that.

Next steps will be to work on KNN query and similarity functions, and eventually provide support for byte and bit element types.

…support' into feature/esql_dense_vector_support

carlosdelest · 2025-05-26T12:59:28Z

@elasticmachine run docs-build

elasticsearchmachine · 2025-05-27T05:56:08Z

💔 Backport failed

Status	Branch	Result
❌	8.19	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 126456

(cherry picked from commit b759161) # Conflicts: # server/src/test/java/org/elasticsearch/index/mapper/vectors/DenseVectorFieldTypeTests.java # x-pack/plugin/esql-core/src/main/java/org/elasticsearch/xpack/esql/core/type/DataType.java # x-pack/plugin/esql/qa/server/single-node/src/javaRestTest/java/org/elasticsearch/xpack/esql/qa/single_node/RestEsqlIT.java # x-pack/plugin/esql/qa/testFixtures/src/main/java/org/elasticsearch/xpack/esql/CsvTestsDataLoader.java # x-pack/plugin/esql/qa/testFixtures/src/main/java/org/elasticsearch/xpack/esql/EsqlTestUtils.java # x-pack/plugin/esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/action/LookupJoinTypesIT.java # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/nulls/Coalesce.java # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plan/logical/join/Join.java

carlosdelest · 2025-05-27T07:05:02Z

💚 All backports created successfully

Status	Branch	Result
✅	8.19

Questions ?

Please refer to the Backport tool documentation

* ES|QL dense vector field type support (#126456) (cherry picked from commit b759161) # Conflicts: # server/src/test/java/org/elasticsearch/index/mapper/vectors/DenseVectorFieldTypeTests.java # x-pack/plugin/esql-core/src/main/java/org/elasticsearch/xpack/esql/core/type/DataType.java # x-pack/plugin/esql/qa/server/single-node/src/javaRestTest/java/org/elasticsearch/xpack/esql/qa/single_node/RestEsqlIT.java # x-pack/plugin/esql/qa/testFixtures/src/main/java/org/elasticsearch/xpack/esql/CsvTestsDataLoader.java # x-pack/plugin/esql/qa/testFixtures/src/main/java/org/elasticsearch/xpack/esql/EsqlTestUtils.java # x-pack/plugin/esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/action/LookupJoinTypesIT.java # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/nulls/Coalesce.java # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plan/logical/join/Join.java * [CI] Auto commit changes from spotless * Fix switch expression --------- Co-authored-by: elasticsearchmachine <[email protected]>

…es (#126150) (#128776) * Integration tests for LOOKUP JOIN over wider range of data types (#126150) This test suite tests the lookup join functionality in ESQL with various data types. For each pair of types being tested, it builds a main index called "index" containing a single document with as many fields as types being tested on the left of the pair, and then creates that many other lookup indexes, each with a single document containing exactly two fields: the field to join on, and a field to return. The assertion is that for valid combinations, the return result should exist, and for invalid combinations an exception should be thrown. If no exception is thrown, and no result is returned, our validation rules are not aligned with the internal behaviour (ie. a bug). Since the `LOOKUP JOIN` command requires the match field name to be the same between the main index and the lookup index, we will have field names that correctly represent the type of the field in the main index, but not the type of the field in the lookup index. This can be confusing, but it is important to remember that the field names are not the same as the types. (cherry picked from commit afc53a3) * fix compilng error * add missing part of the backport of #126456 * add missing part of the backport of #126456 --------- Co-authored-by: Craig Taverner <[email protected]>

carlosdelest added 14 commits April 7, 2025 11:02

Register dense vector field type

6439422

Add first version of BlockDocValuesReader for dense_vector

c7e48a0

Fixed BlockDocValuesReader to work with all dense_vector types

9e40cd4

Small test fixes

d84c8dd

Fix casting

9ff5ed8

Improve testing to add random indexing, and fix similarity

2754697

Add index = false support

d196be0

Spotless

975b4db

Synthetic source testing

ae9fa5f

Add CSV tests and necessary infra for dense_vector field type

9896adf

Make CSV test loader to use numbers when there are multivalued numeri…

dfa420e

…c fields

Fixed test mapping to avoid parsing errors when numbers are used for …

d983495

…multivalued fields

Avoid non-float dense vector field types for now

5707e25

elasticsearchmachine added the v9.1.0 label Apr 8, 2025

carlosdelest added Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) :Analytics/ES|QL AKA ESQL Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch :Search Relevance/Search Catch all for Search Relevance labels Apr 8, 2025

[CI] Auto commit changes from spotless

a8c8a6a

carlosdelest commented Apr 8, 2025

View reviewed changes

carlosdelest added 9 commits April 8, 2025 14:12

Fix test error when checking block loaders

d827c6d

Fix value generation for dense vectors

d8e139d

Merge remote-tracking branch 'carlosdelest/feature/esql_dense_vector_…

0539aff

…support' into feature/esql_dense_vector_support

Fix testing for dense vectors and block loaders

7bbe7ee

Remove dense_vector from CaseTests

7d1f8b7

Provide support in tests for creating constant blocks from collection…

8870fef

… values

Fix test error

ad84f13

Fix unsupported types test

2619fe6

Support synthetic source with non indexed vectors

b23d5a9

carlosdelest closed this May 6, 2025

carlosdelest added 3 commits May 23, 2025 19:55

Spotless

af965a1

Merge remote-tracking branch 'carlosdelest/feature/esql_dense_vector_…

53ec29c

…support' into feature/esql_dense_vector_support

carlosdelest reopened this May 23, 2025

carlosdelest and others added 5 commits May 23, 2025 20:17

Fix tests

60da0fe

Fix test suggested cast for joins

06cbea9

Merge branch 'main' into feature/esql_dense_vector_support

8f07e08

Merge branch 'main' into feature/esql_dense_vector_support

53bcbc7

[CI] Auto commit changes from spotless

fe82007

carlosdelest added auto-backport Automatically create backport pull requests when merged v8.19.0 labels May 26, 2025

carlosdelest added 2 commits May 26, 2025 11:11

Fix LookupJoinTypesIT

a63ca2c

Merge remote-tracking branch 'carlosdelest/feature/esql_dense_vector_…

c5d6ddc

…support' into feature/esql_dense_vector_support

ChrisHegarty approved these changes May 26, 2025

View reviewed changes

carlosdelest merged commit b759161 into elastic:main May 27, 2025
16 of 18 checks passed

elasticsearchmachine added the backport pending label May 27, 2025

carlosdelest mentioned this pull request May 27, 2025

[8.19] ES|QL dense vector field type support (#126456) #128495

Merged

fang-xing-esql added a commit to fang-xing-esql/Elasticsearch that referenced this pull request Jun 3, 2025

add missing part of the backport of elastic#126456

e75a94e

fang-xing-esql added a commit to fang-xing-esql/Elasticsearch that referenced this pull request Jun 3, 2025

add missing part of the backport of elastic#126456

825e299

alex-spies mentioned this pull request Jun 26, 2025

[CI] MixedClusterEsqlSpecIT failling on main->9.0 bwc tests #130085

Closed

ES|QL dense vector field type support #126456

ES|QL dense vector field type support #126456

Uh oh!

Conversation

carlosdelest commented Apr 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

carlosdelest commented May 6, 2025

Uh oh!

carlosdelest commented May 23, 2025

Uh oh!

carlosdelest commented May 23, 2025

Uh oh!

carlosdelest commented May 26, 2025

Uh oh!

Uh oh!

elasticsearchmachine commented May 27, 2025

💔 Backport failed

Uh oh!

carlosdelest commented May 27, 2025

💚 All backports created successfully

Questions ?

Uh oh!

Uh oh!

carlosdelest commented Apr 8, 2025 •

edited

Loading