[feature] Index-first Lucene search: ft:query-scope (live nodes) and ft:search-scope (ES-shaped map)

[joewiz]

[This PR was co-authored with Claude Code. -Joe]

Names are provisional. ft:query-scope / ft:search-scope are working names — happy to rename before merge.

Summary

Two new Lucene functions for index-first, name-independent full-text search over a collection scope:

• ft:query-scope($scope, $query, $options?) → live nodes. The index-first sibling of ft:query: it searches the Lucene index directly over $scope (collection/document URIs, recursive) and returns every matching indexed element — of any element type — as a live node carrying its score and matches. ft:score/ft:field/ft:facets/ft:highlight-field-matches compose on the result as usual.

• ft:search-scope($scope, $query, $options?) → one Elasticsearch _search-shaped map(*). The detached, map-returning companion for API builders: { total, max-score, hits[], facets }, each hit { uri, node-id, score, source, highlight }. Plain result data, no node-set to walk and re-serialize.

Motivation

Two recurring needs:

1. Index-first, name-independent search. To search "all indexed content under a collection regardless of element name" today, you write collection(...)//*[ft:query(., 'field:(…)')]. The descendant-wildcard form loses ft:score for field queries (a known artifact), and it forces callers to enumerate the contributing element names. ft:query-scope queries the index directly over the scope and returns every matching indexed node with correct relevance — no wildcard, no element-name union to maintain.

2. An Elasticsearch _search-style result for API builders. For something like existdb-openapi's /api/search, what you want is plain result data — total, hits with fields, facets — not a node-set to walk and re-serialize. ft:search-scope returns exactly that, assembled natively.

Why two functions (and the naming)

It helps to place these against the existing functions on two axes — detached vs live result, and XML vs map output:

	XML output	map output
detached (a snapshot, no live nodes)	ft:search (legacy report)	ft:search-scope (new)
live nodes	ft:query (context-scoped), ft:query-scope (scope-scoped, new)	(n/a — flattening live nodes to a map defeats their purpose)

• The two new functions are not variants of one another and shouldn't be folded into one: they have divergent return contracts (live node-set vs detached map) and sit on different LuceneIndexWorker methods. The query / search pair mirrors eXist's existing split (ft:query returns live nodes; the legacy ft:search returns a detached report).
• The -scope suffix signals "scoped by a collection, not by an XPath context node-set." I avoided ft:search-index: from an Elasticsearch mindset "index" is the corpus you search (a collection here), not part of a verb, so it reads ambiguously.

Hit granularity (the one decision worth calling out)

"Document" means three things in this stack, and conflating them produces real count bugs:

• an eXist collection — the storage container $scope names;
• an eXist document — an XML resource (intro.xml), an XmldbURI;
• an indexed element — eXist creates one Lucene document per indexed element occurrence (per <text qname="…">), so one document with two <para> and one <caption> yields three Lucene documents.

So $scope filters at eXist-document granularity, but hits/counts/facets are at indexed-element granularity. Elasticsearch guarantees one Lucene document per ES document (1:1); eXist does not. ft:search-scope therefore defaults to indexed-element granularity (honest to the index, sub-document precision), with a collapse option for the ES-faithful one-hit-per-document view (group by document, best-scoring element, total = distinct documents) — the analog of ES field-collapse / top_hits.

ft:search-scope options

$options carries a filter that restricts the query, plus keys that shape the result:

key	type	meaning
"filter"	map(*)	facet drill-down { dimension: value(s) } restricting the search (ES post-filter analog; keeps total/paging consistent)
"fields"	xs:string*	stored fields to include in each hit's source (_source)
"highlight"	xs:string*	fields to highlight; adds a per-hit highlight map of exist:field/exist:match nodes
"facets"	xs:string*	facet dimensions to aggregate over the full match set
"collapse"	xs:boolean	group hits to one-per-document (best-scoring element; total = distinct documents)
"offset" / "limit"	xs:integer	page the ranked hits

Note: ft:search-scope returns a single envelope map, so read the hit count from ?total (or count(?hits?*)) — count(ft:search-scope(...)) is always 1.

What changed (extensions/indexes/lucene)

• QueryScope.java (ft:query-scope) and SearchScope.java (ft:search-scope) — the two functions; SearchScope assembles the ES map natively from the same index-first query, reading scores/fields/facets the way ft:score/ft:field/ft:facets do, and reusing the ft:highlight-field-matches engine for highlight.
• LuceneScope.java — shared scope-resolution + index-first query execution, used by both functions (no duplication).
• LuceneModule.java — registers the four signatures.
• Field.java — highlightMatches made package-private static so ft:search-scope can reuse it on the live node it materializes; no behavior change to ft:highlight-field-matches.
• Tests: ft-query-scope.xqm, ft-search-scope.xqm, scope-dls.xqm, and the LuceneQueryScopeTests runner.

Companion fix (faceted highlighting)

ft:search-scope's highlight works on its own for unfaceted queries. Combining filter (facet drill-down) with highlight additionally requires a small fix to a pre-existing defect — a facet DrillDownQuery silently disabled ft:highlight-field-matches term extraction — submitted as a separate bugfix PR, #6454 (it affects ft:query faceted highlighting independent of these functions, so it's based on develop, not this branch). This PR doesn't depend on it except for that combined path.

Test plan

• 42 XQSuite tests across the suite: ft:query-scope (name-independence, nested-element scoring, composition with ft:score/ft:field/ft:facets/ft:highlight-field-matches, live nodes, sortable-by-score); ft:search-scope (envelope shape, element vs collapse granularity, source/facets/highlight/filter/offset/limit).
• Document-level security (scope-dls.xqm): a guest never gets nodes or hits from documents it cannot read (verified via system:as-user against mixed read permissions) — the restricted document is absent from both hits and total, so the count does not leak its existence.
• Validated against a real corpus (eXist's own function docs) behind an /api/search-style endpoint: identical hit set to the collection(...)//el[ft:query(...)] approach with ft:score preserved; faceted highlighting end-to-end (with [bugfix] Preserve ft:highlight-field-matches under facet drill-down #6454 in place).
• Codacy/PMD clean on the new files.