ES|QL dense vector field type support #126456

carlosdelest · 2025-04-08T11:24:48Z

Support dense_vector field type. This is the first step to allow kNN queries and having dense_vector as a first class citizen in ES|QL

This allows a mapping that has dense_vector like the following:

{
  "properties": {
    "id": {
      "type": "long"
    },
    "vector": {
      "type": "dense_vector",
      "similarity": "l2_norm"
    }
  }
}

To be retrieved via ES|QL:

FROM dense_vector
| KEEP id, vector
| SORT id

id   | vector
0    | [1.0, 2.0, 3.0]
1    | [4.0, 5.0, 6.0]

For now, just float element types are allowed. There will be a similar work in order to allow for byte and bit element types, but I wanted to review this implementation first to ensure it's in line with what we need.

Both indexed / not indexed types and synthetic source is supported.

Support for CSV tests has been added. For now CSV tests are simple, we can expand on these and also support additional operations on dense_vector field types in subsequent PRs. An integration test has been added to test extensively on different index options and doc storage structure.

dense_vector field type is under a feature flag, as this will require follow up work.

…ctor_support # Conflicts: # x-pack/plugin/esql-core/src/main/java/org/elasticsearch/xpack/esql/core/type/DataType.java # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/ResponseValueUtils.java # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/planner/LocalExecutionPlanner.java

…c fields

…multivalued fields

carlosdelest · 2025-04-08T11:32:30Z

server/src/main/java/org/elasticsearch/index/mapper/BlockDocValuesReader.java

@@ -504,6 +506,80 @@ public String toString() {
        }
    }

+    public static class DenseVectorBlockLoader extends DocValuesBlockLoader {


Added a BlockLoader for dense vectors, that uses FloatVectorValues to retrieve indexed vector data.

carlosdelest · 2025-04-08T11:33:28Z

server/src/main/java/org/elasticsearch/index/mapper/vectors/DenseVectorFieldMapper.java

+        @Override
+        public BlockLoader blockLoader(MappedFieldType.BlockLoaderContext blContext) {
+            if (elementType != ElementType.FLOAT) {
+                throw new UnsupportedOperationException("Only float dense vectors are supported for now");


We can work on this next, creating specific BlockLoaders for Byte and Bit field types.

carlosdelest · 2025-04-08T11:34:15Z

x-pack/plugin/esql/qa/testFixtures/src/main/java/org/elasticsearch/xpack/esql/CsvAssert.java

@@ -145,6 +146,10 @@ private static void assertMetadata(
                    // Type.asType translates all bytes references into keywords
                    continue;
                }
+                if (blockType == Type.DOUBLE && expectedType == DENSE_VECTOR) {
+                    // DENSE_VECTOR is internally represented as a double block


This could potentially change when we support byte and bit element types - we could create the appropriate blocks.

carlosdelest · 2025-04-08T11:35:31Z

x-pack/plugin/esql/qa/testFixtures/src/main/resources/mapping-default-incompatible.json

@@ -63,18 +63,7 @@
            "type" : "keyword"
        },
        "salary_change": {
-            "type": "float",


Changing CSV loading made this a problem, as there were parsing exceptions when trying to index float numeric data into integers. I didn't see a convenient way out of this and decided to remove as this field is not being tested.

If we can keep this unchanged, that will be great, this is a good example of nested fields.

It is just changed for the mapping-default-incompatible mapping, which was created to test some incompatible field mappings that did not include subfields. I'll try to give this another shot but it will require changes to the CSV loader or the dataset 😢

It is just changed for the mapping-default-incompatible mapping, which was created to test some incompatible field mappings that did not include subfields. I'll try to give this another shot but it will require changes to the CSV loader or the dataset 😢

Is it easier if we make another copy of employees's schema and data for dense_vector related tests?

Is it easier if we make another copy of employees's schema and data for dense_vector related tests?

The problem is that changing how the CSV tests load data impacted this dataset. Before this change, multivalues were being uploaded as arrays of strings, which is something we don't want to do for dense_vectors as that is not a format supported on the DenseVectorFieldMapper.

It seemed like too much work to change the actual dataset when that particular field is not actually used in the tests.

I'm open to other solutions here!

@fang-xing-esql are these fields used in other tests somehow?
I see this was actually added by @carlosdelest a while back #117555.
The employees_incompatible index that is set with this mapping is only used in match function/operator tests.
We don't modify any of those tests here, so this looks like a safe change to me.

…support' into feature/esql_dense_vector_support

… values

carlosdelest · 2025-04-15T07:21:50Z

@ioanatia @fang-xing-esql I was able to use FLOAT blocks for dense_vector - changed that in cd462b8.

As Ioana noted, we're not publicly exposing float as a type here but dense_vector, so the change is doable as an implementation detail 👍 . Thanks!

…support' into feature/esql_dense_vector_support

ChrisHegarty

I'm happy to see FloatBlocks being used, but they have too much flexibility - all different dims per position and also nulls. Maybe we just need an easy way/utility to assert their correct shape?

.../esql/compute/src/main/java/org/elasticsearch/compute/lucene/ValuesSourceReaderOperator.java

ioanatia

there's an outstanding question from @ChrisHegarty that would be good to address but otherwise LGTM

…mensions

…ctor_support # Conflicts: # x-pack/plugin/esql/qa/testFixtures/src/main/java/org/elasticsearch/xpack/esql/CsvTestsDataLoader.java # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java

carlosdelest · 2025-04-21T11:05:06Z

I'm happy to see FloatBlocks being used, but they have too much flexibility - all different dims per position and also nulls. Maybe we just need an easy way/utility to assert their correct shape?

@ChrisHegarty , I renamed some methods to make clear that we're creating dense vectors instead of floats, and added some checks for them, on c951ee7.

I had to take some back on ba0a6b9, as I can't easily add a new float block builder given the sealed structure of builders. We may have to add a new Block type for dense_vector, maybe as a follow up?

LMKWYT

ioanatia · 2025-04-30T08:19:22Z

x-pack/plugin/esql/qa/testFixtures/src/main/resources/data/dense_vector.csv

@@ -0,0 +1,3 @@
+id:l, vector:dense_vector
+0,    [1.0, 2.0, 3.0]
+1,    [4.0, 5.0, 6.0]


this might be very nitpicky - but can we add another dense_vector value that does not have ordered values?
this might be why we did not caught #126456 (comment) during tests

mvOrdered does not impact retrieval as it's used as an optimization in some cases. But adding unordered data uncovered a small fix that needed to be done to support multivalued style fields: 4b2126e

ioanatia · 2025-04-30T08:25:31Z

...n/esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/DenseVectorFieldTypeIT.java

+            .field("type", "dense_vector")
+            .field("index", index);
+        if (index) {
+            mapping.field("similarity", "l2_norm");


nit: is it possible to randomize the similarity option? otherwise we could leave this completely out, since it optional?

The problem with that is some similarities normalize the vector values, so what we retrieve back is not what was stored, thus making comparisons difficult. I think we're ok with keeping this simple.

ioanatia · 2025-04-30T08:28:03Z

...n/esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/DenseVectorFieldTypeIT.java

+
+        try (var resp = run(query)) {
+            assertColumnNames(resp.columns(), List.of("id", "vector"));
+            assertColumnTypes(resp.columns(), List.of("integer", "dense_vector"));


are we missing some assertions here for the values? to at least check that they are not nulls?

That's done in other tests - I started with this test to just check that the field types are retrieved correctly.

ioanatia · 2025-04-30T08:32:41Z

server/src/main/java/org/elasticsearch/index/mapper/BlockSourceReader.java

+
+        @Override
+        public String toString() {
+            return "BlockSourceReader.Floats";


Should this be "BlockSourceReader.DenseVectors"?

++, afe79f7

ioanatia · 2025-04-30T08:39:07Z

server/src/main/java/org/elasticsearch/index/mapper/BlockSourceReader.java

+
+        @Override
+        protected String name() {
+            return "Floats";


Again - just checking if this the value we want to return given that the class is called DenseVectorBlockLoader?

Good catch, I renamed this multiple times but didn't follow on this method. Thanks! afe79f7

…t case

…ctor_support # Conflicts: # server/src/main/java/org/elasticsearch/index/mapper/vectors/DenseVectorFieldMapper.java # x-pack/plugin/esql/qa/testFixtures/src/main/java/org/elasticsearch/xpack/esql/CsvTestsDataLoader.java # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java

carlosdelest · 2025-05-06T08:44:28Z

Closing this PR, as the final approach will imply multiple field types for the different dense_vector element types.

…ctor_support # Conflicts: # x-pack/plugin/build.gradle # x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/data/BlockUtils.java # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/planner/PlannerUtils.java

…support' into feature/esql_dense_vector_support

carlosdelest · 2025-05-23T18:00:40Z

Reopening this, as we've decided to go for a single dense_vector field type. We can reevaluate the need for multiple dense_vector element types later.

carlosdelest · 2025-05-23T18:03:57Z

@ioanatia @ChrisHegarty @fang-xing-esql I intend to merge this PR on Monday to provide initial support for dense_vector float element types. Please comment if you have any issue with that.

Next steps will be to work on KNN query and similarity functions, and eventually provide support for byte and bit element types.

carlosdelest added 14 commits April 7, 2025 11:02

Register dense vector field type

6439422

Add first version of BlockDocValuesReader for dense_vector

c7e48a0

Fixed BlockDocValuesReader to work with all dense_vector types

9e40cd4

Small test fixes

d84c8dd

Fix casting

9ff5ed8

Improve testing to add random indexing, and fix similarity

2754697

Add index = false support

d196be0

Spotless

975b4db

Synthetic source testing

ae9fa5f

Add CSV tests and necessary infra for dense_vector field type

9896adf

Make CSV test loader to use numbers when there are multivalued numeri…

dfa420e

…c fields

Fixed test mapping to avoid parsing errors when numbers are used for …

d983495

…multivalued fields

Avoid non-float dense vector field types for now

5707e25

elasticsearchmachine added the v9.1.0 label Apr 8, 2025

carlosdelest added Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) :Analytics/ES|QL AKA ESQL Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch :Search Relevance/Search Catch all for Search Relevance labels Apr 8, 2025

[CI] Auto commit changes from spotless

a8c8a6a

carlosdelest commented Apr 8, 2025

View reviewed changes

carlosdelest added 9 commits April 8, 2025 14:12

Fix test error when checking block loaders

d827c6d

Fix value generation for dense vectors

d8e139d

Merge remote-tracking branch 'carlosdelest/feature/esql_dense_vector_…

0539aff

…support' into feature/esql_dense_vector_support

Fix testing for dense vectors and block loaders

7bbe7ee

Remove dense_vector from CaseTests

7d1f8b7

Provide support in tests for creating constant blocks from collection…

8870fef

… values

Fix test error

ad84f13

Fix unsupported types test

2619fe6

Support synthetic source with non indexed vectors

b23d5a9

carlosdelest added the >non-issue label Apr 14, 2025

Use Float as block type for dense_vector

cd462b8

carlosdelest and others added 4 commits April 15, 2025 10:27

Fix tests

0675a42

[CI] Auto commit changes from spotless

7f95d6a

Fix test

2cff4e7

Merge remote-tracking branch 'carlosdelest/feature/esql_dense_vector_…

efc3fb6

…support' into feature/esql_dense_vector_support

ChrisHegarty reviewed Apr 16, 2025

View reviewed changes

.../esql/compute/src/main/java/org/elasticsearch/compute/lucene/ValuesSourceReaderOperator.java Outdated Show resolved Hide resolved

.../esql/compute/src/main/java/org/elasticsearch/compute/lucene/ValuesSourceReaderOperator.java Outdated Show resolved Hide resolved

ioanatia reviewed Apr 16, 2025

View reviewed changes

carlosdelest added 4 commits April 21, 2025 10:34

Floats are not ordered for dense_vector values

f8741ca

Use dense vector specific methods for creating builders, and check di…

c951ee7

…mensions

Avoid doing checks on dense vectors as we can't cast to BlockBuilder

ba0a6b9

ioanatia reviewed Apr 30, 2025

View reviewed changes

ChrisHegarty approved these changes Apr 30, 2025

View reviewed changes

carlosdelest and others added 4 commits April 30, 2025 19:55

Add CSV tests with unordered data, and fix CSV data retrieval for tha…

4b2126e

…t case

Change source readers name() method

afe79f7

[CI] Auto commit changes from spotless

217ce84

ioanatia approved these changes May 2, 2025

View reviewed changes

carlosdelest closed this May 6, 2025

carlosdelest added 3 commits May 23, 2025 19:55

Spotless

af965a1

Merge remote-tracking branch 'carlosdelest/feature/esql_dense_vector_…

53ec29c

…support' into feature/esql_dense_vector_support

carlosdelest reopened this May 23, 2025

Fix tests

60da0fe

ES|QL dense vector field type support #126456

Are you sure you want to change the base?

ES|QL dense vector field type support #126456

Conversation

carlosdelest commented Apr 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

carlosdelest commented Apr 15, 2025

Uh oh!

ChrisHegarty left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ioanatia left a comment

Choose a reason for hiding this comment

Uh oh!

carlosdelest commented Apr 21, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

carlosdelest commented May 6, 2025

Uh oh!

carlosdelest commented May 23, 2025

Uh oh!

carlosdelest commented May 23, 2025

Uh oh!

Uh oh!

carlosdelest commented Apr 8, 2025 •

edited

Loading