feat(es-compat): add index_filter support for field capabilities API (#6102)

* feat(es-compat): add index_filter support for field capabilities API

Implements index_filter parameter support for the ES-compatible
_field_caps endpoint, allowing users to filter field capabilities
based on document queries.

Changes:
- Add query_ast field to ListFieldsRequest and LeafListFieldsRequest protos
- Parse index_filter from ES Query DSL and convert to QueryAst
- Pass query_ast through to leaf nodes for future filtering support
- Add unit tests for index_filter parsing
- Add REST API integration tests

Note: This implementation accepts and parses the index_filter parameter
for API compatibility. Full split-level document filtering will be
added as a follow-up enhancement.

Closes #5693

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: ruo <ruoliu.dev@gmail.com>

* feat(es-compat): implement split-level filtering for field_caps index_filter

Address PR review comments for index_filter support in _field_caps API:

- Extract `parse_index_filter_to_query_ast()` function with clean prototype
- Implement split-level filtering via `split_matches_query()` using
  lightweight `query.count()` execution (no document materialization)
- Add proper async handling with ByteRangeCache, warmup(), and
  run_cpu_intensive() for Quickwit's async-only storage
- Add metastore-level pruning:
  - Tag extraction via `extract_tags_from_query()`
  - Time range extraction via `refine_start_end_timestamp_from_ast()`
- Build DocMapper only when query_ast is provided (no overhead for
  common path without index_filter)
- Fix REST API tests: use `json:` key (not `json_body:`), use lowercase
  term values to match tokenizer behavior
- Update tests to run against both quickwit and elasticsearch engines

Two-level filtering now implemented:
1. Metastore level: tags + time range from query AST
2. Split level: lightweight query execution for accurate filtering

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: ruo <ruoliu.dev@gmail.com>

* refactor(es-compat): use best-effort metadata filtering for index_filter

Remove heavy split-level query execution for field_caps index_filter.
The implementation now aligns with ES's "best-effort" approach that uses
metadata-level filtering only (time range, tags) instead of opening
splits and executing queries.

Changes:
- Remove split_matches_query function (no longer opens splits)
- Remove query_ast and doc_mapper from LeafListFieldsRequest proto
- Keep metadata-level filtering in root_list_fields:
  - Time range extraction from query AST
  - Tag-based split pruning
- Simplify leaf_list_fields to just return fields from all splits

This matches ES semantics: "filtering is done on a best-effort basis...
this API may return an index even if the provided filter matches no
document."

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: ruo <ruoliu.dev@gmail.com>

* fix(es-compat): reject empty index_filter to match ES behavior

- Remove empty object {} handling in parse_index_filter_to_query_ast
- ES rejects empty index_filter with 400, now QW does too
- Add tag_fields config and tag-based index_filter test
- Update unit and integration tests accordingly

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: ruo <ruoliu.dev@gmail.com>

* Added ref doc for the new functionality.

* cargo fmt fix

---------

Signed-off-by: ruo <ruoliu.dev@gmail.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: fulmicoton <paul.masurel@datadoghq.com>

Author: 3 people

docs/reference/es_compatible_api.md

@@ -365,6 +365,79 @@ Example response:
 
 [HTTP accept header]: https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html
 
+
+### `_field_caps`   Field capabilities API
+
+```
+GET api/v1/_elastic/<index>/_field_caps
+```
+```
+POST api/v1/_elastic/<index>/_field_caps
+```
+```
+GET api/v1/_elastic/_field_caps
+```
+```
+POST api/v1/_elastic/_field_caps
+```
+
+The [field capabilities API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-field-caps.html) returns information about the capabilities of fields among multiple indices.
+
+#### Supported Query string parameters
+
+| Variable              | Type       | Description                                                                    | Default value |
+| --------------------- | ---------- | ------------------------------------------------------------------------------ | ------------- |
+| `fields`              | `String`   | Comma-separated list of fields to retrieve capabilities for. Supports wildcards (`*`). | (Optional) |
+| `allow_no_indices`    | `Boolean`  | If `true`, missing or closed indices are not an error.                          | (Optional)    |
+| `expand_wildcards`    | `String`   | Controls what kind of indices that wildcard patterns can match.                 | (Optional)    |
+| `ignore_unavailable`  | `Boolean`  | If `true`, unavailable indices are ignored.                                    | (Optional)    |
+| `start_timestamp`     | `Integer`  | *(Quickwit-specific)* If set, restricts splits to documents with a timestamp range start >= `start_timestamp` (seconds since epoch). | (Optional) |
+| `end_timestamp`       | `Integer`  | *(Quickwit-specific)* If set, restricts splits to documents with a timestamp range end < `end_timestamp` (seconds since epoch). | (Optional) |
+
+#### Supported Request Body parameters
+
+| Variable           | Type          | Description                                                                 | Default value |
+| ------------------ | ------------- | --------------------------------------------------------------------------- | ------------- |
+| `index_filter`     | `Json object` | A query to filter indices. If provided, only fields from indices that can potentially match the filter are returned. See [index_filter](#index_filter). | (Optional) |
+| `runtime_mappings`  | `Json object` | Accepted but not supported.                                                 | (Optional)    |
+
+#### `index_filter`
+
+The `index_filter` parameter allows you to filter which indices contribute to the field capabilities response. When provided, Quickwit uses the filter query to prune indices (splits) that cannot match the filter, and only returns field capabilities for the remaining ones.
+
+Like Elasticsearch, this is a **best-effort** approach: Quickwit may return field capabilities from indices that do not actually contain any matching documents. In Quickwit, the filtering is limited to the existing split-pruning based on metadata:
+
+- **Time pruning**: Range queries on the timestamp field can eliminate splits whose time range does not overlap with the filter.
+- **Tag pruning**: Term queries on [tag fields](../configuration/index-config.md#tag-fields) can eliminate splits that do not contain the requested tag value.
+
+Other filter types (e.g. full-text queries or term queries on non-tag fields) are accepted but will not prune any splits — all indices will be returned as if no filter was specified. In particular, Quickwit does not check whether terms are present in the term dictionary.
+
+#### Request Body example
+
+```json
+{
+  "index_filter": {
+    "range": {
+      "timestamp": {
+        "gte": "2024-01-01T00:00:00Z",
+        "lt": "2024-02-01T00:00:00Z"
+      }
+    }
+  }
+}
+```
+
+```json
+{
+  "index_filter": {
+    "term": {
+      "status": "active"
+    }
+  }
+}
+```
+
+
 ## Query DSL
 
 [Elasticsearch Query DSL reference](https://www.elastic.co/guide/en/elasticsearch/reference/8.8/query-dsl.html).

quickwit/quickwit-proto/protos/quickwit/search.proto

@@ -125,6 +125,10 @@ message ListFieldsRequest {
   optional int64 start_timestamp = 3;
   optional int64 end_timestamp = 4;
 
+  // JSON-serialized QueryAst for index_filter support.
+  // When provided, only fields from documents matching this query are returned.
+  optional string query_ast = 5;
+
   // Control if the request will fail if split_ids contains a split that does not exist.
   // optional bool fail_on_missing_index = 6;
 }
@@ -141,7 +145,6 @@ message LeafListFieldsRequest {
   // Optional limit query to a list of fields
   // Wildcard expressions are supported.
   repeated string fields = 4;
-
 }
 
 message ListFieldsResponse {

quickwit/quickwit-proto/src/codegen/quickwit/quickwit.search.rs

@@ -24,13 +24,16 @@ use itertools::Itertools;
 use quickwit_common::rate_limited_warn;
 use quickwit_common::shared_consts::{FIELD_PRESENCE_FIELD_NAME, SPLIT_FIELDS_FILE_NAME};
 use quickwit_common::uri::Uri;
+use quickwit_config::build_doc_mapper;
+use quickwit_doc_mapper::tag_pruning::extract_tags_from_query;
 use quickwit_metastore::SplitMetadata;
 use quickwit_proto::metastore::MetastoreServiceClient;
 use quickwit_proto::search::{
     LeafListFieldsRequest, ListFields, ListFieldsEntryResponse, ListFieldsRequest,
     ListFieldsResponse, SplitIdAndFooterOffsets, deserialize_split_fields,
 };
 use quickwit_proto::types::{IndexId, IndexUid};
+use quickwit_query::query_ast::QueryAst;
 use quickwit_storage::Storage;
 
 use crate::leaf::open_split_bundle;
@@ -310,6 +313,8 @@ impl FieldPattern {
 }
 
 /// `leaf` step of list fields.
+///
+/// Returns field metadata from the assigned splits.
 pub async fn leaf_list_fields(
     index_id: IndexId,
     index_storage: Arc<dyn Storage>,
@@ -322,6 +327,12 @@ pub async fn leaf_list_fields(
         .map(|pattern_str| FieldPattern::from_str(pattern_str))
         .collect::<crate::Result<_>>()?;
 
+    // If no splits, return empty response
+    if split_ids.is_empty() {
+        return Ok(ListFieldsResponse { fields: Vec::new() });
+    }
+
+    // Get fields from all splits
     let single_split_list_fields_futures: Vec<_> = split_ids
         .iter()
         .map(|split_id| {
@@ -375,7 +386,7 @@ pub async fn leaf_list_fields(
     Ok(ListFieldsResponse { fields })
 }
 
-/// Index metas needed for executing a leaf search request.
+/// Index metas needed for executing a leaf list fields request.
 #[derive(Clone, Debug)]
 pub struct IndexMetasForLeafSearch {
     /// Index id.
@@ -399,29 +410,63 @@ pub async fn root_list_fields(
     if indexes_metadata.is_empty() {
         return Ok(ListFieldsResponse { fields: Vec::new() });
     }
-    let index_uid_to_index_meta: HashMap<IndexUid, IndexMetasForLeafSearch> = indexes_metadata
-        .iter()
-        .map(|index_metadata| {
-            let index_metadata_for_leaf_search = IndexMetasForLeafSearch {
-                index_uri: index_metadata.index_uri().clone(),
-                index_id: index_metadata.index_config.index_id.to_string(),
-            };
-
-            (
-                index_metadata.index_uid.clone(),
-                index_metadata_for_leaf_search,
+
+    // Build index metadata map and extract timestamp field for time range refinement
+    let mut index_uid_to_index_meta: HashMap<IndexUid, IndexMetasForLeafSearch> = HashMap::new();
+    let mut index_uids: Vec<IndexUid> = Vec::new();
+    let mut timestamp_field_opt: Option<String> = None;
+
+    for index_metadata in indexes_metadata {
+        // Extract timestamp field for time range refinement (use first index's field)
+        if timestamp_field_opt.is_none()
+            && list_fields_req.query_ast.is_some()
+            && let Ok(doc_mapper) = build_doc_mapper(
+                &index_metadata.index_config.doc_mapping,
+                &index_metadata.index_config.search_settings,
             )
-        })
-        .collect();
-    let index_uids: Vec<IndexUid> = indexes_metadata
-        .into_iter()
-        .map(|index_metadata| index_metadata.index_uid)
-        .collect();
+        {
+            timestamp_field_opt = doc_mapper.timestamp_field_name().map(|s| s.to_string());
+        }
+
+        let index_metadata_for_leaf_search = IndexMetasForLeafSearch {
+            index_uri: index_metadata.index_uri().clone(),
+            index_id: index_metadata.index_config.index_id.to_string(),
+        };
+
+        index_uids.push(index_metadata.index_uid.clone());
+        index_uid_to_index_meta.insert(
+            index_metadata.index_uid.clone(),
+            index_metadata_for_leaf_search,
+        );
+    }
+
+    // Extract tags and refine time range from query_ast for split pruning
+    let mut start_timestamp = list_fields_req.start_timestamp;
+    let mut end_timestamp = list_fields_req.end_timestamp;
+    let tags_filter_opt = if let Some(ref query_ast_json) = list_fields_req.query_ast {
+        let query_ast: QueryAst = serde_json::from_str(query_ast_json)
+            .map_err(|err| SearchError::InvalidQuery(err.to_string()))?;
+
+        // Refine time range from query AST if timestamp field is available
+        if let Some(ref timestamp_field) = timestamp_field_opt {
+            crate::root::refine_start_end_timestamp_from_ast(
+                &query_ast,
+                timestamp_field,
+                &mut start_timestamp,
+                &mut end_timestamp,
+            );
+        }
+
+        extract_tags_from_query(query_ast)
+    } else {
+        None
+    };
+
     let split_metadatas: Vec<SplitMetadata> = list_relevant_splits(
         index_uids,
-        list_fields_req.start_timestamp,
-        list_fields_req.end_timestamp,
-        None,
+        start_timestamp,
+        end_timestamp,
+        tags_filter_opt,
         &mut metastore,
     )
     .await?;