Scalable text search over Qdrant: lexical-gated dense + server-side sparse scoring

[Udjin79]

Scalable text search over Qdrant

The lexical arm of hybrid recall no longer has to live in client memory. The in-process BM25 is honest about its ceiling (<100K documents, corpus in RAM) — this PR moves lexical matching into Qdrant itself, next to the vectors, at any collection size, selectable by one config switch:

recall:
  text_search: lexical   # auto | off | bm25 | qdrant_bm25 | lexical | sparse

The two new arms

• lexical — lexical-gated dense (QdrantTextRetriever): the query's salient tokens (exact technical tokens first — IDs, IPs, versions — then content words) become a server-side full-text MatchText gate, dense similarity ranks within the lexically-matching subset. Works on a pre-existing collection without reindexing; the one server prerequisite is a full-text payload index (mnemostack text-index, idempotent/non-destructive). The gate filters, it does not score.
• sparse — server-side lexical scoring (QdrantSparseRetriever): writes maintain a dependency-free sparse tf encoding of each chunk (SparseTextEncoder, CRC-32 token hashing) in a named sparse space with Qdrant's IDF modifier — queries rank by a tf·idf-like score computed on the server, no client-side corpus. New collections indexed under this mode get the space automatically; populated collections migrate via mnemostack sparse-backfill (scroll + update_vectors: nothing re-embedded, idempotent) — enabling sparse on a populated collection without the encodings refuses loudly rather than silently serving partial recall.

Selector semantics

• auto (default) keeps historical behavior byte-for-byte: file-corpus BM25 when bm25_paths is set, else no lexical arm. qdrant_bm25 wires the existing payload-scrolled BM25 corpus as a first-class option for sub-100K collections.
• The --source bm25 selector is an umbrella for the configured lexical arm (whatever mode is active answers to it) — on the CLI, in synthesize(), and in result filtering.
• Wired on every surface (HTTP server, MCP incl. the standalone entry point, CLI incl. synthesize); doctor live-checks the configured arm's readiness (the sparse space / the full-text index on the actual collection), not just the config string.

Boundaries stated plainly

• Both arms respect the tenant boundary and validity view exactly like dense search, and follow the configurable payload schema (text_key/timestamp_key/timestamp_format), including schema declaration for the Recaller's per-field derivation.
• Sparse writes are maintained by the sync VectorStore only (AsyncVectorStore does not write sparse encodings — documented); set_payload text edits leave the sparse encoding stale until re-upsert/backfill (documented).
• VectorStore.search gains text_any= (nested should-gate composed with tenant/validity filters; an explicitly empty gate raises rather than degrade to match-all) and sparse_search().
• Minimum qdrant-client is now >=1.10 (sparse models + IDF modifier).

Tests / docs

• tests/test_text_search.py (23 tests): encoder shapes, sparse roundtrip/ranking/tenant boundary, lexical gate composition, both retrievers incl. foreign schema + gate-token quality, mode resolution/env/doctor, CLI wiring for all three Qdrant modes, indexing-store sparse flag, dense→sparse migration (refusal + backfill coverage), --source bm25 umbrella through synthesize().
• CHANGELOG; docs/deployment.md gains a lexical-arm scaling table + migration notes; docs/api-stability.md documents the mode contract.

Full suite: 1415 passed / 9 skipped; ruff + mypy clean.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: dc2e8ac071

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a97efdb343

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".