Adding support to exclude semantic_text subfields #127664

Samiul-TheSoccerFan · 2025-05-02T20:33:13Z

Update the fieldCaps API to exclude semantic_text subfields in both legacy and new formats.

Legacy format:

setup:


PUT test-field-caps-with-legacy
{
    "settings": {
        "index.mapping.semantic_text.use_legacy_format": true
    },
    "mappings": {
        "properties": {
            "test_field_legacy": {
                "type": "semantic_text",
                "inference_id": ".elser-2-elasticsearch"
            },
            "non_infer_field_legacy": {
                "type": "text"
            },
            "sparse_vector_legacy": {
                "type": "sparse_vector"
            },
            "dense_vector_legacy": {
                "type": "dense_vector",
                "dims": 3,
                "similarity": "l2_norm"
            }
        }
    }
}

PUT test-field-caps-with-legacy/_doc/doc1
{
    "test_field_legacy": "these are not the droids you're looking for. He's free to go around",
    "sparse_vector_legacy": {
        "these": 1,
        "are": 2,
        "not": 3
    },
    "dense_vector_legacy": [1, 2, 3]
}

Query:

GET /_field_caps?allow_no_indices=true&fields=*&index=test*&ignore_unavailable=true&expand_wildcards=open

Response before update (Skimmed):

{
  "indices": [
    "test-field-caps-with-legacy"
  ],
  "fields": {
    "non_infer_field_legacy": {
      "text": {
        "type": "text",
        "metadata_field": false,
        "searchable": true,
        "aggregatable": false
      }
    },
    "test_field_legacy.inference.chunks.text": {
      "keyword": {
        "type": "keyword",
        "metadata_field": false,
        "searchable": false,
        "aggregatable": false
      }
    },
    "test_field_legacy.inference": {
      "object": {
        "type": "object",
        "metadata_field": false,
        "searchable": false,
        "aggregatable": false
      }
    },
    "sparse_vector_legacy": {
      "sparse_vector": {
        "type": "sparse_vector",
        "metadata_field": false,
        "searchable": true,
        "aggregatable": false
      }
    },
    "test_field_legacy": {
      "text": {
        "type": "text",
        "metadata_field": false,
        "searchable": true,
        "aggregatable": false
      }
    },
    "test_field_legacy.inference.chunks.embeddings": {
      "sparse_vector": {
        "type": "sparse_vector",
        "metadata_field": false,
        "searchable": true,
        "aggregatable": false
      }
    },
    "dense_vector_legacy": {
      "dense_vector": {
        "type": "dense_vector",
        "metadata_field": false,
        "searchable": true,
        "aggregatable": false
      }
    },
    "test_field_legacy.inference.chunks": {
      "nested": {
        "type": "nested",
        "metadata_field": false,
        "searchable": false,
        "aggregatable": false
      }
    }
  }
}

Response after update (Skimmed):

{
  "indices": [
    "test-field-caps-with-legacy"
  ],
  "fields": {
    "non_infer_field_legacy": {
      "text": {
        "type": "text",
        "metadata_field": false,
        "searchable": true,
        "aggregatable": false
      }
    },
    "sparse_vector_legacy": {
      "sparse_vector": {
        "type": "sparse_vector",
        "metadata_field": false,
        "searchable": true,
        "aggregatable": false
      }
    },
    "test_field_legacy": {
      "text": {
        "type": "text",
        "metadata_field": false,
        "searchable": true,
        "aggregatable": false
      }
    },
    "dense_vector_legacy": {
      "dense_vector": {
        "type": "dense_vector",
        "metadata_field": false,
        "searchable": true,
        "aggregatable": false
      }
    }
  }
}

new format:

setup:

PUT test-field-caps
{
    "mappings": {
        "properties": {
            "test_field": {
                "type": "semantic_text",
                "inference_id": ".elser-2-elasticsearch"
            },
            "non_infer_field": {
                "type": "text"
            },
            "sparse_vector": {
                "type": "sparse_vector"
            },
            "dense_vector": {
                "type": "dense_vector",
                "dims": 3,
                "similarity": "l2_norm"
            }
        }
    }
}

PUT test-field-caps/_doc/doc1
{
    "test_field": "these are not the droids you're looking for. He's free to go around",
    "sparse_vector": {
        "these": 1,
        "are": 2,
        "not": 3
    },
    "dense_vector": [1, 2, 3]
}

Query:

GET /_field_caps?allow_no_indices=true&fields=*&index=test*&ignore_unavailable=true&expand_wildcards=open

Response before update (Skimmed):

{
  "indices": [
    "test-field-caps"
  ],
  "fields": {
    "_ignored_source": {
      "_ignored_source": {
        "type": "_ignored_source",
        "metadata_field": true,
        "searchable": false,
        "aggregatable": false
      }
    },
    "non_infer_field": {
      "text": {
        "type": "text",
        "metadata_field": false,
        "searchable": true,
        "aggregatable": false
      }
    },
    "_index": {
      "_index": {
        "type": "_index",
        "metadata_field": true,
        "searchable": true,
        "aggregatable": true
      }
    },
    "_feature": {
      "_feature": {
        "type": "_feature",
        "metadata_field": true,
        "searchable": false,
        "aggregatable": false
      }
    },
    "sparse_vector": {
      "sparse_vector": {
        "type": "sparse_vector",
        "metadata_field": false,
        "searchable": true,
        "aggregatable": false
      }
    },
    "test_field.inference.chunks.embeddings": {
      "sparse_vector": {
        "type": "sparse_vector",
        "metadata_field": false,
        "searchable": true,
        "aggregatable": false
      }
    },
    "test_field.inference.chunks.offset": {
      "offset_source": {
        "type": "offset_source",
        "metadata_field": false,
        "searchable": false,
        "aggregatable": false
      }
    },
    "test_field": {
      "text": {
        "type": "text",
        "metadata_field": false,
        "searchable": true,
        "aggregatable": false
      }
    },
    "_inference_fields": {
      "_inference_fields": {
        "type": "_inference_fields",
        "metadata_field": true,
        "searchable": false,
        "aggregatable": false
      }
    },
    "test_field.inference": {
      "object": {
        "type": "object",
        "metadata_field": false,
        "searchable": false,
        "aggregatable": false
      }
    },
    "dense_vector": {
      "dense_vector": {
        "type": "dense_vector",
        "metadata_field": false,
        "searchable": true,
        "aggregatable": false
      }
    },
    "test_field.inference.chunks": {
      "nested": {
        "type": "nested",
        "metadata_field": false,
        "searchable": false,
        "aggregatable": false
      }
    }
  }
}

Response after update (Skimmed):

{
  "indices": [
    "test-field-caps"
  ],
  "fields": {
    "test_field": {
      "text": {
        "type": "text",
        "metadata_field": false,
        "searchable": true,
        "aggregatable": false
      }
    },
    "_inference_fields": {
      "_inference_fields": {
        "type": "_inference_fields",
        "metadata_field": true,
        "searchable": false,
        "aggregatable": false
      }
    },
    "non_infer_field": {
      "text": {
        "type": "text",
        "metadata_field": false,
        "searchable": true,
        "aggregatable": false
      }
    },
    "sparse_vector": {
      "sparse_vector": {
        "type": "sparse_vector",
        "metadata_field": false,
        "searchable": true,
        "aggregatable": false
      }
    },
    "dense_vector": {
      "dense_vector": {
        "type": "dense_vector",
        "metadata_field": false,
        "searchable": true,
        "aggregatable": false
      }
    },    
  }
}

elasticsearchmachine · 2025-05-02T20:35:10Z

Hi @Samiul-TheSoccerFan, I've created a changelog YAML for you.

...e/src/yamlRestTest/resources/rest-api-spec/test/inference/10_semantic_text_field_mapping.yml

Samiul-TheSoccerFan · 2025-05-02T20:45:55Z

...e/src/yamlRestTest/resources/rest-api-spec/test/inference/10_semantic_text_field_mapping.yml

+  - requires:
+      cluster_features: "gte_v8.16.0"
+      reason: field_caps support for semantic_text added in 8.16.0


Do we need to define a new cluster feature? As per my understanding, these fields are not expected from field_caps API so excluding these should not have an impact on the API level or discover. We have also covered backward compatibility through other yaml file.

I think it would be good to create a test feature for these tests.

kderusso

Agreed with @Mikep86 's comments in Slack, but good start!

kderusso · 2025-05-05T12:43:12Z

...e/src/yamlRestTest/resources/rest-api-spec/test/inference/10_semantic_text_field_mapping.yml

+  - requires:
+      cluster_features: "gte_v8.16.0"
+      reason: field_caps support for semantic_text added in 8.16.0


I think it would be good to create a test feature for these tests.

server/src/main/java/org/elasticsearch/action/fieldcaps/FieldCapabilitiesFetcher.java

Mikep86

Much better! We need to tweak the approach a bit but this is getting closer.

server/src/main/java/org/elasticsearch/index/mapper/MappedFieldType.java

server/src/main/java/org/elasticsearch/index/mapper/vectors/SparseVectorFieldMapper.java

server/src/main/java/org/elasticsearch/index/mapper/vectors/DenseVectorFieldMapper.java

...nference/src/main/java/org/elasticsearch/xpack/inference/mapper/OffsetSourceFieldMapper.java

…xt mapper

… functionality

elasticsearchmachine · 2025-05-08T14:17:30Z

Pinging @elastic/es-search-relevance (Team:Search Relevance)

elasticsearchmachine · 2025-05-08T14:17:30Z

Pinging @elastic/search-eng (Team:SearchOrg)

elasticsearchmachine · 2025-05-08T14:17:31Z

Pinging @elastic/search-relevance (Team:Search - Relevance)

elasticsearchmachine · 2025-05-08T14:17:31Z

Pinging @elastic/es-search-foundations (Team:Search Foundations)

Mikep86

Looks good overall, just some nits and tests to clean up

Mikep86 · 2025-05-08T13:19:16Z

server/src/main/java/org/elasticsearch/index/mapper/KeywordFieldMapper.java

@@ -448,10 +455,12 @@ public KeywordFieldMapper build(MapperBuilderContext context) {
                indexCreatedVersion,
                IndexVersions.SYNTHETIC_SOURCE_STORE_ARRAYS_NATIVELY_KEYWORD
            );
+
+            KeywordFieldType keywordFieldType = buildFieldType(context, fieldtype);


Nit: we can call buildFieldType(context, fieldtype) in-line when creating KeywordFieldMapper, the command stays short and readable

Mikep86 · 2025-05-08T13:24:22Z

server/src/main/java/org/elasticsearch/index/mapper/MappedFieldType.java

+    /**
+     * @return true if fieldType is subfields of semantic_text type
+     */


This documentation should be more generic. Semantic text is only one case where we want to exclude a field from field caps, there could be others in the future.

Suggested change

/**

* @return true if fieldType is subfields of semantic_text type

*/

/**

* @return true if the field should be excluded from field caps

*/

Mikep86 · 2025-05-08T13:28:37Z

server/src/main/java/org/elasticsearch/index/mapper/vectors/SparseVectorFieldMapper.java

@@ -83,18 +84,25 @@ public Builder setStored(boolean value) {
            return this;
        }

+        public Builder setExcludeFromFieldCaps(boolean value) {


Nit: Can we change this to excludeFromFieldCaps to match the other builders?

Mikep86 · 2025-05-08T14:16:50Z

...nference/src/main/java/org/elasticsearch/xpack/inference/mapper/SemanticTextFieldMapper.java

@@ -134,6 +134,9 @@ public class SemanticTextFieldMapper extends FieldMapper implements InferenceFie
    public static final NodeFeature SEMANTIC_TEXT_SKIP_INFERENCE_FIELDS = new NodeFeature("semantic_text.skip_inference_fields");
    public static final NodeFeature SEMANTIC_TEXT_BIT_VECTOR_SUPPORT = new NodeFeature("semantic_text.bit_vector_support");
    public static final NodeFeature SEMANTIC_TEXT_SUPPORT_CHUNKING_CONFIG = new NodeFeature("semantic_text.support_chunking_config");
+    public static final NodeFeature SEMANTIC_TEXT_SUB_FIELDS_EXCLUDE_FROM_FIELD_CAPS = new NodeFeature(


Nit: SEMANTIC_TEXT_EXCLUDE_SUB_FIELDS_FROM_FIELD_CAPS matches the node feature name better

Mikep86 · 2025-05-08T14:29:31Z

...e/src/yamlRestTest/resources/rest-api-spec/test/inference/10_semantic_text_field_mapping.yml

+  - not_exists: fields.sparse_field.chunks.embeddings
+  - not_exists: fields.sparse_field.chunks.offset
+  - not_exists: fields.dense_field.chunks.embeddings
+  - not_exists: fields.dense_field.chunks.offset


Shouldn't this be *.inference.chunks.*?

Also, can we check that field caps excludes inference and inference.chunks here too?

Mikep86 · 2025-05-08T14:29:57Z

...e/src/yamlRestTest/resources/rest-api-spec/test/inference/10_semantic_text_field_mapping.yml

+"Field caps exclude chunks and embedding fields":
+  - requires:
+      cluster_features: "semantic_text.exclude_sub_fields_from_field_caps"
+      reason: field caps api exclude semantic_text subfields from 9.1.0


8.19.0 & 9.1.0

Mikep86 · 2025-05-08T14:30:05Z

...c/yamlRestTest/resources/rest-api-spec/test/inference/10_semantic_text_field_mapping_bwc.yml

+"Field caps exclude chunks embedding and text fields":
+  - requires:
+      cluster_features: "semantic_text.exclude_sub_fields_from_field_caps"
+      reason: field caps api exclude semantic_text subfields from 9.1.0


8.19.0 & 9.1.0

Mikep86 · 2025-05-08T14:30:36Z

...c/yamlRestTest/resources/rest-api-spec/test/inference/10_semantic_text_field_mapping_bwc.yml

+  - not_exists: fields.sparse_field.inference.chunks.embeddings
+  - not_exists: fields.sparse_field.inference.chunks.text
+  - not_exists: fields.dense_field.inference.chunks.embeddings
+  - not_exists: fields.dense_field.inference.chunks.text


can we check that field caps excludes inference and inference.chunks here too?

Mikep86 · 2025-05-08T14:37:49Z

...nference/src/main/java/org/elasticsearch/xpack/inference/mapper/SemanticTextFieldMapper.java

+            var chunkTextField = new KeywordFieldMapper.Builder(TEXT_FIELD, indexVersionCreated).indexed(false)
+                .docValues(false)
+                .excludeFromFieldCaps(true);


@jimczi Do we need index version checks around setting this new flag on the mapping?

kderusso

Nice improvements! No additional feedback other than what Mike already flagged.

Samiul-TheSoccerFan · 2025-05-09T03:30:32Z

@elasticmachine update branch

javanna · 2025-05-09T09:44:28Z

Thanks for working on this. Could we be explicit in the description of this PR around the motivations behind this change? There is no opt-in or opt-out, hence we are just declaring that these fields are not useful in the field caps output at all times?

I am guessing that the issue is around Kibana showing these in Discover? Have we considered alternatives?

Mikep86 · 2025-05-09T11:53:38Z

@javanna

There is no opt-in or opt-out, hence we are just declaring that these fields are not useful in the field caps output at all times?

I am guessing that the issue is around Kibana showing these in Discover? Have we considered alternatives?

The subfields we are excluding are semantic_text implementation details that the user should not rely on. Yes, the primary motivation here is Kibana showing them in Discover, we get a fair amount of questions about why these fields show up and we have to reiterate the answer that these fields are implementation details and should not be used externally.

Hiding these fields from field caps codifies that these are implementation details at the API level, creating a consistent experience in this respect for Kibana and API users.

Are there alternatives you're aware of that would achieve the same result?

Edit: Another thing to point out here is that these subfields are not visible in the mappings returned to the user (on purpose), further anchoring them as implementation details

Mikep86 · 2025-05-09T13:16:15Z

I talked with @jimczi and we have an alternative implementation that does not involve adding a flag to MappedFieldType. The end effect on the fields caps API output is the same though, so the tests should remain valid.

Adding support to exclude semantic_text subfields

1a3bb97

Samiul-TheSoccerFan added >enhancement v9.1.0 :Search Foundations/Mapping Index mappings, including merging and defining field types :Search Relevance/Vectors Vector search :SearchOrg/Relevance Label for the Search (solution/org) Relevance team labels May 2, 2025

Update docs/changelog/127664.yaml

522730e

github-actions bot deployed to docs-preview May 2, 2025 20:35 View deployment

Updating changelog file

db64ad2

github-actions bot deployed to docs-preview May 2, 2025 20:39 View deployment

Samiul-TheSoccerFan commented May 2, 2025

View reviewed changes

...e/src/yamlRestTest/resources/rest-api-spec/test/inference/10_semantic_text_field_mapping.yml Outdated Show resolved Hide resolved

remove duplicate test from yaml file

e333e78

Samiul-TheSoccerFan commented May 2, 2025

View reviewed changes

github-actions bot deployed to docs-preview May 2, 2025 20:46 View deployment

kderusso reviewed May 5, 2025

View reviewed changes

Mikep86 reviewed May 5, 2025

View reviewed changes

server/src/main/java/org/elasticsearch/action/fieldcaps/FieldCapabilitiesFetcher.java Outdated Show resolved Hide resolved

Adding support to exclude semantic_text subfields from mapper builders

06caf66

github-actions bot deployed to docs-preview May 6, 2025 22:06 View deployment

Mikep86 reviewed May 7, 2025

View reviewed changes

Samiul-TheSoccerFan added 6 commits May 8, 2025 01:09

Adding support for generic field types

5572275

refactoring to use builder and setting exclude value from semantic_te…

785e8d6

…xt mapper

update in semantic_text mapper and fetcher to incorporate the support…

9e65cd0

… functionality

Fix code style issue

56fb75c

adding node feature for yaml tests

232879b

Merge branch 'main' into exclude-subfields-for-field-caps-api

f2e5dae

github-actions bot deployed to docs-preview May 8, 2025 05:30 View deployment

Samiul-TheSoccerFan marked this pull request as ready for review May 8, 2025 14:17

elasticsearchmachine added Team:SearchOrg Meta label for the Search Org (Enterprise Search) Team:Search Foundations Meta label for the Search Foundations team in Elasticsearch labels May 8, 2025

elasticsearchmachine added Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch Team:Search - Relevance The Search organization Search Relevance team labels May 8, 2025

Mikep86 reviewed May 8, 2025

View reviewed changes

kderusso reviewed May 8, 2025

View reviewed changes

Samiul-TheSoccerFan added 2 commits May 8, 2025 17:13

Adding more restrictive checks on yaml tests and few refactoring

f75d236

Merge branch 'main' into exclude-subfields-for-field-caps-api

eb9b56b

github-actions bot deployed to docs-preview May 8, 2025 21:15 View deployment

Samiul-TheSoccerFan added the v8.19.0 label May 8, 2025

Samiul-TheSoccerFan requested a review from Mikep86 May 8, 2025 21:21

Merge branch 'main' into exclude-subfields-for-field-caps-api

1e53918

github-actions bot deployed to docs-preview May 9, 2025 03:31 View deployment

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding support to exclude semantic_text subfields #127664

Adding support to exclude semantic_text subfields #127664

Samiul-TheSoccerFan commented May 2, 2025

elasticsearchmachine commented May 2, 2025

Samiul-TheSoccerFan May 2, 2025

kderusso May 5, 2025

kderusso left a comment

kderusso May 5, 2025

Mikep86 left a comment

elasticsearchmachine commented May 8, 2025

elasticsearchmachine commented May 8, 2025

elasticsearchmachine commented May 8, 2025

elasticsearchmachine commented May 8, 2025

Mikep86 left a comment

Mikep86 May 8, 2025

Mikep86 May 8, 2025

Mikep86 May 8, 2025

Mikep86 May 8, 2025

Mikep86 May 8, 2025

Mikep86 May 8, 2025

Mikep86 May 8, 2025

Mikep86 May 8, 2025

Mikep86 May 8, 2025

kderusso left a comment

Samiul-TheSoccerFan commented May 9, 2025

javanna commented May 9, 2025

Mikep86 commented May 9, 2025 •

edited

Loading

Mikep86 commented May 9, 2025

Adding support to exclude semantic_text subfields #127664

Are you sure you want to change the base?

Adding support to exclude semantic_text subfields #127664

Conversation

Samiul-TheSoccerFan commented May 2, 2025

Legacy format:

setup:

Query:

Response before update (Skimmed):

Response after update (Skimmed):

new format:

setup:

Query:

Response before update (Skimmed):

Response after update (Skimmed):

elasticsearchmachine commented May 2, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kderusso left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Mikep86 left a comment

Choose a reason for hiding this comment

elasticsearchmachine commented May 8, 2025

elasticsearchmachine commented May 8, 2025

elasticsearchmachine commented May 8, 2025

elasticsearchmachine commented May 8, 2025

Mikep86 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kderusso left a comment

Choose a reason for hiding this comment

Samiul-TheSoccerFan commented May 9, 2025

javanna commented May 9, 2025

Mikep86 commented May 9, 2025 • edited Loading

Mikep86 commented May 9, 2025

Mikep86 commented May 9, 2025 •

edited

Loading