feat(keyword): add local FTS5 keyword retrieval

[jamespud]

Description

Adds an opt-in per-account SQLite FTS5 keyword sidecar that provides backend-owned, search-time BM25 recall for deployments without a remote VikingDB full-text index.

The local keyword sidecar is a recall accelerator only: grep still performs its final regex match against on-disk content, and find/search fuse keyword candidates with dense results. When the sidecar is missing or incomplete, both paths degrade to their existing behavior.

This follows the direction maintainers indicated in #1857: search-time BM25 over a local SQLite FTS5 sidecar instead of precomputed sparse vectors.

Human Involvement

• A human participated in the implementation or review loop
• This PR was generated entirely by AI agents without human participation in the loop

Related Issue

• Related to feat: add local_bm25 sparse provider for hybrid retrieval #1857 (previous local_bm25 sparse-provider attempt, closed without merge)
• Related to [Bug] VikingDB BM25 grep returns empty results on large corpus — fallback to fs grep not triggered reliably #2850 and fix(grep): fall back to fs on empty VikingDB recall and query timeout (#2850) #2900 (remote BM25 grep fallback semantics)

Type of Change

• New feature (non-breaking change that adds functionality)
• Documentation update
• Test update

Changes Made

• Add openviking/storage/keywordfs/: per-account KeywordFS SQLite FTS5 store with search-time bm25() recall, CJK-aware tokenizer (char/bigram/optional jieba), KeywordMsg, KeywordQueue, and KeywordProcessor
• Extend grep engine resolution: auto prefers remote VikingDB BM25 when available, falls back to the local sidecar, then to a filesystem scan; add explicit local and vikingdb engine modes
• Add _grep_local_then_fs: local keyword recall + _grep_in_files exact matching
• Add KeywordQueue setup to QueueManager and co-enqueue keyword upserts alongside embedding messages so the sidecar stays in sync with vector coverage
• Wire keyword delete/move messages into rm/mv and prefix cleanup into snapshot restore
• Add opt-in hybrid retrieval for find/search via HybridKeywordRecaller with RRF and weighted fusion, exposed through retrieval.hybrid.enabled and a request-level hybrid boolean
• Add /api/v1/observer/keyword for sidecar health and document keyword/retrieval.hybrid configuration
• Add design document docs/design/local-keyword-fts5-sidecar-design.md

Testing

• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• I have tested this on the following platforms:
  • Linux
  • macOS
  • Windows

Added tests cover KeywordFS upsert/delete/move/prefix-delete/rebuild, KeywordProcessor, KeywordMsg.from_embedding, local grep engine resolution and recall, hybrid RRF/weighted fusion, and the keyword observer component. 71 passed across the new and adjacent test suites.

Checklist

• My code follows the project's coding style
• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• Any dependent changes have been merged and published

Additional Notes

The keyword sidecar is disabled by default (keyword.enabled=false), and it is automatically disabled on encrypted deployments when respect_encryption=true because it stores plaintext. Full ov reindex --mode keyword rebuild wiring is left as a follow-up.

[jamespud]

Hi @ByteDanceLiuYang @zhoujh01 @qin-ctx, would appreciate a review of this PR.
It continues the direction discussed in #1857.

This PR implements "search-time BM25 over a local SQLite FTS5 sidecar" as
suggested in #1857:

• Adds openviking/storage/keywordfs/: a per-account SQLite FTS5 sidecar that
  does search-time bm25() keyword recall, with a CJK-aware tokenizer
  (char/bigram/optional jieba)
• Positioned as a recall accelerator only: grep still runs its final regex
  match against on-disk content; find/search fuse keyword candidates with
  dense results via HybridKeywordRecaller (RRF / weighted)
• Engine resolution: auto prefers remote VikingDB BM25 → local FTS5 sidecar →
  filesystem scan
• Stays in sync with the Embedding queue via co-enqueue; rm/mv/restore are
  all hooked up
• Off by default (keyword.enabled=false); automatically disabled on encrypted
  deployments (plaintext FTS); degrades gracefully when the sidecar is missing
  or incomplete
• Rebased onto the latest main (conflict resolved); all 71 added tests pass

Design doc: docs/design/local-keyword-fts5-sidecar-design.md

Three things I'd especially like feedback on:

1. Should the sidecar land as a first-class backend in the retrieval-layer
   framework, or as a standalone path first? (feat: add local_bm25 sparse provider for hybrid retrieval #1857 mentioned the framework may
   land in the near term — want to align on timing)
2. Does the degrade semantics of FTS5 recall + on-disk exact match look right?
3. Is the co-enqueue wiring with the Embedding queue the best way to keep the
   sidecar in sync?

Thanks in advance!

[jamespud]

I ran local performance benchmarks to get a baseline.
The benchmark script is on the perf/keyword-fs-benchmark branch:
jamespud/OpenViking@perf/keyword-fs-benchmark/tests/benchmark/bench_keyword_fs.py

Environment: Python 3.14 / SQLite 3.53.2 with FTS5 / single-threaded /
synthetic corpus (~200 words/doc, latin-only). Results measured directly
against KeywordFS (no queue/async layer overhead).

Indexing throughput

Corpus size	Docs/sec	MB/sec	DB / raw ratio
1k docs	3,167	4.52	3.28x
5k docs	2,788	3.98	3.17x
20k docs	2,462	3.52	3.11x

The ~3x space overhead is expected for an inverted index with tokenized
unicode61 content. For a typical small-to-medium workspace (say 5k indexed
documents at ~1 KB each), that's ~15 MB of sidecar storage.

BM25 lookup latency

Corpus size	p50	p95	p99
5k docs	0.07 ms	0.15 ms	0.16 ms
20k docs	0.10 ms	0.24 ms	0.27 ms

Sub-millisecond across the board — lookup is dominated by SQLite
overhead, not corpus size, consistent with FTS5's design.

Grep: FTS5 recall vs brute-force scan

This is the main value proposition — how much does the sidecar narrow
the candidate set before the on-disk regex match?

Corpus size	FTS5 recall	Brute-force scan	Speedup
1k docs	0.04 ms	0.5 ms	14x
5k docs	0.04 ms	2.6 ms	62x
20k docs	0.04 ms	11.2 ms	253x

The FTS5 path stays flat (~0.04ms) while brute-force scales linearly.
For real greps where the on-disk regex is the expensive part, the
sidecar reduces the candidate set from "everything" to "matches the lex
term" — that's where the real savings come from.

CJK tokenizer overhead

• char mode: ~5,000 docs/sec (500-word mixed CJK docs)
• bigram mode: ~4,300 docs/sec

Tokenization is a meaningful but not dominant cost during indexing.
jieba would be slower but is opt-in and lazy-imported.

Notes / caveats

• These are synthetic micro-benchmarks; real-world perf depends on
  document size distribution, token diversity, and disk speed.
• The sidecar is off by default (keyword.enabled=false), so zero
  overhead for users who don't opt in.
• The recall-accelerator model means we never return incorrect results —
  FTS5 narrows candidates, the final match still happens on disk (grep)
  or via dense fusion (find/search).

Happy to take suggestions on what additional benchmarks would be useful.