MB-27666: Fix nested mode handling for field-implicit queries (#2272)

- Fix nested-mode search for field-implicit queries (`match_all`,
`docID`): These queries previously filtered nested documents at the
searcher level, but since `ExtractFields` returned no fields, the nested
collector was never activated. In compound queries (e.g., Boolean
filters), this caused a level mismatch—searcher filtering vs
nested-level execution—resulting in zero matches.
- Unify filtering at collector level: Introduced `HasID()` and
`HasAll()` flags in `FieldSet` to track field-implicit queries. Removed
searcher-level filtering from `MatchAllSearcher`.
- Nested collector now correctly activates, ensuring consistent behavior
regardless of query composition.
- Fix an edge case where usage of `match_all query` as a sub-clause in a
`NestedConjunctionSearcher` could cause a panic.
- Fix a case where root document fields would get excluded from the
`_all` field even when included.
- Fix outdated information in `hierarchy.md`
- Add unit tests for Hierarchical Nested Vector Search.

Author: CascadingRadium

docs/hierarchy.md

@@ -1,6 +1,6 @@
 # Hierarchical nested search
 
-* *v2.6.0* (and after) will come with support for **Array indexing and hierarchy search**.
+* *v2.6.0* (and after) will come with support for **Array indexing and hierarchical nested search**.
 * We've achieved this by embedding nested documents within our bleve (scorch) indexes.
 * Usage of zap file format: [v17](https://github.com/blevesearch/zapx/blob/master/zap.md). Here we preserve hierarchical document relationships within segments, continuing to conform to the segmented architecture of *scorch*.
 
@@ -146,7 +146,7 @@
 }
 ```
 
-* Any Bleve query (e.g., match, phrase, term, fuzzy, numeric/date range etc.) can be executed against fields within nested documents, with no special handling required. The query processor will automatically traverse the nested structures to find matches. Additional search constructs
+* Any Bleve query (e.g., `match`, `phrase`, `term`, `fuzzy`, `numeric/date range` etc.) can be executed against fields within nested documents, with no special handling required. The query processor will automatically traverse the nested structures to find matches. Additional search constructs
 like vector search, synonym search, hybrid and pre-filtered vector search integrate seamlessly with hierarchy search.
 
 * Conjunction Queries (AND queries) and other queries that depend on term co-occurrence within the same hierarchical context will respect the boundaries of nested documents. This means that terms must appear within the same nested object to be considered a match. For example, a conjunction query searching for an employee named "Alice" with the role "Engineer" within the "Engineering" department will only return results where both name and role terms are found within the same employee object, which is itself within a "Engineering" department object.
@@ -156,11 +156,11 @@ like vector search, synonym search, hybrid and pre-filtered vector search integr
 
   * Nested Faceting / Aggregations: Facets are computed within matched nested objects, producing context-aware buckets. E.g., a facet on `departments.projects.status` returns ongoing or completed only for projects in matched departments.
 
-  * Sorting by Nested Fields: Sorting can use fields from the relevant nested object, e.g., ordering companies by `departments.budget sorts` based on the budget of the specific matched department, not unrelated departments.
+  * Sorting by Nested Fields: Sorting can use fields from the relevant nested object, e.g., ordering companies by `departments.budget` sorts based on the budget of the specific matched department, not unrelated departments.
 
-* Vector Search (KNN / Multi-KNN): When an array of objects is marked as nested and contains vector fields, each vector is treated as belonging to its own nested document. Vector similarity is computed only within the same nested object, not across siblings. For example, if `departments.employees` is a nested array where each employee has a `skills_vector`, a KNN search using the embedding of `machine learning engineer` will match only employees whose own `skills_vector` is similar; other employees vectors within the same department or document do not contribute to the score or match. This also means that a vector search query for `K = 3` will return the top 3 most similar employees across all departments and all companies, and may return multiple employees from the same department or company if they rank among the top 3 most similar overall.
+* Vector Search (KNN / Multi-KNN): When a document contains an array of objects with vector/multi-vector fields, the final document score and ranking are identical whether or not the array is marked as `nested`. In both cases, the highest-scoring vector is selected; either directly from the array (non-nested) or from the best-matching nested object with its score bubbled up to the parent document.
 
-* Pre-Filtered Vector Search: When vector search is combined with filters on fields inside a nested array, the filters are applied first to pick which nested items are eligible. The vector search then runs only on those filtered items. For example, if `departments.employees` is a `nested` array, a pre-filtered KNN query for employees with the role `Manager` in the `Sales` department will first narrow the candidate set to only employees who meet those field conditions, and then compute vector similarity on the `skills_vector` of that filtered subset. This ensures that vector search results come only from the employees that satisfy the filter, while still treating each employee as an independent vector candidate.
+* Pre-Filtered Vector Search: When vector search is combined with filters on fields inside a nested array, the filters are applied first to pick which nested items are eligible. Vector similarity is then computed only on the vector fields of those filtered nested objects. For example, if `departments.employees` is a `nested` array, a pre-filtered KNN query for employees with a `skills_vector` matching `machine learning engineer`, a role of `Manager`, and belonging to the `Sales` department will first narrow the candidate set to only employees who meet the requirement, and then compute vector similarity on the `skills_vector` of that filtered subset. This ensures that vector search results come only from the employees that satisfy the filter, and not from unrelated employees in other departments.
 
 ## Indexing
 

document/document.go

@@ -18,7 +18,6 @@ import (
 	"fmt"
 	"reflect"
 
-	"github.com/blevesearch/bleve/v2/search"
 	"github.com/blevesearch/bleve/v2/size"
 	index "github.com/blevesearch/bleve_index_api"
 )
@@ -164,27 +163,6 @@ func (d *Document) AddNestedDocument(doc *Document) {
 	d.NestedDocuments = append(d.NestedDocuments, doc)
 }
 
-func (d *Document) NestedFields() search.FieldSet {
-	if len(d.NestedDocuments) == 0 {
-		return nil
-	}
-	fieldSet := search.NewFieldSet()
-	var collectFields func(index.Document)
-	collectFields = func(doc index.Document) {
-		// Add all field names from this nested document
-		doc.VisitFields(func(field index.Field) {
-			fieldSet.AddField(field.Name())
-		})
-		// Recursively collect from this document's nested documents
-		if nd, ok := doc.(index.NestedDocument); ok {
-			nd.VisitNestedDocuments(collectFields)
-		}
-	}
-	// Start collection from nested documents only (not root document)
-	d.VisitNestedDocuments(collectFields)
-	return fieldSet
-}
-
 func (d *Document) VisitNestedDocuments(visitor func(doc index.Document)) {
 	for _, doc := range d.NestedDocuments {
 		visitor(doc)

index_impl.go

@@ -1569,7 +1569,7 @@ func (i *indexImpl) buildTopNCollector(ctx context.Context, req *SearchRequest,
 				if err != nil {
 					return nil, err
 				}
-				if nm.IntersectsPrefix(fs) {
+				if fs.HasID() || nm.IntersectsPrefix(fs) {
 					return newNestedCollector(nr), nil
 				}
 			}

mapping/index.go

@@ -376,13 +376,7 @@ func (im *IndexMappingImpl) MapDocument(doc *document.Document, data interface{}
 		// see if the _all field was disabled
 		allMapping, _ := docMapping.documentMappingForPath("_all")
 		if allMapping == nil || allMapping.Enabled {
-			excludedFromAll := walkContext.excludedFromAll
-			nf := doc.NestedFields()
-			if nf != nil {
-				// if the document has any nested fields, exclude them from _all
-				excludedFromAll = append(excludedFromAll, nf.Slice()...)
-			}
-			field := document.NewCompositeFieldWithIndexingOptions("_all", true, []string{}, excludedFromAll, index.IndexField|index.IncludeTermVectors)
+			field := document.NewCompositeFieldWithIndexingOptions("_all", true, []string{}, walkContext.excludedFromAll, index.IndexField|index.IncludeTermVectors)
 			doc.AddField(field)
 		}
 		doc.SetIndexed()

search/query/conjunction.go

@@ -106,6 +106,12 @@ func (q *ConjunctionQuery) Searcher(ctx context.Context, i index.IndexReader, m
 		// if we have common depth == max depth then we can just use
 		// the normal conjunction searcher, as all fields share the same
 		// nested context, otherwise we need to use the nested conjunction searcher
+		// also, if we are querying the _all or _id fields, we need to use
+		// the nested conjunction searcher as well, with common depth 0
+		// indicating matches happen only at the root level
+		if qfs.HasAll() || qfs.HasID() {
+			commonDepth = 0
+		}
 		if commonDepth < maxDepth {
 			return searcher.NewNestedConjunctionSearcher(ctx, i, ss, commonDepth, options)
 		}

search/query/query.go

@@ -502,6 +502,11 @@ func ExtractFields(q Query, m mapping.IndexMapping, fs search.FieldSet) (search.
 				break
 			}
 		}
+	case *DocIDQuery, *MatchAllQuery:
+		if fs == nil {
+			fs = search.NewFieldSet()
+		}
+		fs.AddField("_id")
 	}
 	return fs, err
 }

search/query/query_test.go

@@ -1017,6 +1017,18 @@ func TestExtractFields(t *testing.T) {
 					}`,
 			expFields: []string{"text"},
 		},
+		{
+			query: `{
+						"match_all": {}
+					}`,
+			expFields: []string{"_id"},
+		},
+		{
+			query: `{
+						"ids": ["a", "b", "c"]
+					}`,
+			expFields: []string{"_id"},
+		},
 	}
 
 	m := mapping.NewIndexMapping()

search/searcher/search_match_all.go

@@ -36,8 +36,6 @@ type MatchAllSearcher struct {
 	reader      index.DocIDReader
 	scorer      *scorer.ConstantScorer
 	count       uint64
-	nested      bool
-	ancestors   []index.AncestorID
 }
 
 func NewMatchAllSearcher(ctx context.Context, indexReader index.IndexReader, boost float64, options search.SearcherOptions) (*MatchAllSearcher, error) {
@@ -52,15 +50,11 @@ func NewMatchAllSearcher(ctx context.Context, indexReader index.IndexReader, boo
 	}
 	scorer := scorer.NewConstantScorer(1.0, boost, options)
 
-	// check if we are in nested mode
-	nested, _ := ctx.Value(search.NestedSearchKey).(bool)
-
 	return &MatchAllSearcher{
 		indexReader: indexReader,
 		reader:      reader,
 		scorer:      scorer,
 		count:       count,
-		nested:      nested,
 	}, nil
 }
 
@@ -82,23 +76,6 @@ func (s *MatchAllSearcher) SetQueryNorm(qnorm float64) {
 	s.scorer.SetQueryNorm(qnorm)
 }
 
-func (s *MatchAllSearcher) isNested(id index.IndexInternalID) bool {
-	// if not running in nested mode, always return false
-	if !s.nested {
-		return false
-	}
-	var err error
-	// check if this doc has ancestors, if so it is nested
-	if nr, ok := s.reader.(index.NestedReader); ok {
-		s.ancestors, err = nr.Ancestors(id, s.ancestors[:0])
-		if err != nil {
-			return false
-		}
-		return len(s.ancestors) > 1
-	}
-	return false
-}
-
 func (s *MatchAllSearcher) Next(ctx *search.SearchContext) (*search.DocumentMatch, error) {
 	id, err := s.reader.Next()
 	if err != nil {
@@ -109,11 +86,6 @@ func (s *MatchAllSearcher) Next(ctx *search.SearchContext) (*search.DocumentMatc
 		return nil, nil
 	}
 
-	if s.isNested(id) {
-		// if nested then skip and get next
-		return s.Next(ctx)
-	}
-
 	// score match
 	docMatch := s.scorer.Score(ctx, id)
 	// return doc match
@@ -131,11 +103,6 @@ func (s *MatchAllSearcher) Advance(ctx *search.SearchContext, ID index.IndexInte
 		return nil, nil
 	}
 
-	if s.isNested(id) {
-		// if nested then return next
-		return s.Next(ctx)
-	}
-
 	// score match
 	docMatch := s.scorer.Score(ctx, id)
 

search/util.go

@@ -291,11 +291,14 @@ func (fs FieldSet) AddField(field string) {
 	fs[field] = struct{}{}
 }
 
-// Slice returns the fields in this set as a slice of strings.
-func (fs FieldSet) Slice() []string {
-	rv := make([]string, 0, len(fs))
-	for field := range fs {
-		rv = append(rv, field)
-	}
-	return rv
+// HasID returns true if the field set contains the "_id" field.
+func (fs FieldSet) HasID() bool {
+	_, ok := fs["_id"]
+	return ok
+}
+
+// HasAll returns true if the field set contains the "_all" field.
+func (fs FieldSet) HasAll() bool {
+	_, ok := fs["_all"]
+	return ok
 }