research(nightly): hybrid sparse-dense search — BM25 + ANN with RRF and RSF (ADR-256)

[ruvnet]

Summary

Nightly research spike (2026-06-17) implementing hybrid sparse-dense retrieval in a new
standalone crate ruvector-hybrid, with three fusion strategies benchmarked on a 10K-document
128-D corpus.

• ADR-256: proposes adding RRF and RSF to ruvector-core::advanced_features::hybrid_search
• crates/ruvector-hybrid: working Rust PoC — zero unsafe, WASM-compilable, 19 unit tests
• Benchmark (10K docs × 128-D, 500 queries, k=10, Intel Xeon 2.80 GHz, rustc 1.94.1 --release):

Variant	Recall@10	QPS	Memory
Dense flat (exact cosine)	7.5%	371	5,000 KB
BM25 (sparse only)	77.3%	57,174	637 KB
ScoreFusion α=0.7	68.8%	357	5,637 KB
RRF k=60	50.5%	360	5,637 KB
RSF α=0.5	76.6%	360	5,637 KB

Key Findings

• RSF (Weaviate-style per-list min-max normalisation) nearly matches pure BM25 (76.6% vs 77.3%) on keyword-dominated tasks while adding semantic coverage.
• RRF (rank-only, score-agnostic) is the safer default when signal balance is unknown — appropriate for agentic RAG pipelines.
• ScoreFusion with hard-coded α=0.7 (the current ruvector-core default) is worst among hybrids (68.8%) on keyword-heavy workloads.
• The existing ruvector-core BM25 re-tokenises doc texts at query time (O(N×|d|)); this crate pre-computes TF at index time (O(|q|×P)), eliminating the regression.

Files Changed

crates/ruvector-hybrid/
  Cargo.toml          — crate manifest (rand dep only)
  src/lib.rs          — SparseSearch, DenseSearch, HybridSearch traits + recall_at_k
  src/bm25.rs         — Robertson BM25, k1=1.2 b=0.75, pre-computed TF postings
  src/dense.rs        — FlatDenseIndex, cosine flat-scan
  src/fusion.rs       — ScoreFusionIndex, RrfHybridIndex, RsfHybridIndex
  src/main.rs         — benchmark binary + 7 acceptance tests

docs/adr/ADR-256-hybrid-sparse-dense-search.md
docs/research/nightly/2026-06-17-hybrid-sparse-dense/README.md
docs/research/nightly/2026-06-17-hybrid-sparse-dense/gist.md
Cargo.toml            — workspace member added

Test Plan

• cargo test -p ruvector-hybrid — 19 unit tests pass
• cargo run --release -p ruvector-hybrid — 7 acceptance tests pass, real benchmark numbers
• cargo build --release -p ruvector-hybrid — clean build, zero warnings

Phase 2 Work (not in this PR)

• Add FusionStrategy enum to ruvector-core::advanced_features::hybrid_search
• Add search_rrf() and search_rsf() to HybridSearch struct
• Fix BM25 re-tokenisation bug in ruvector-core
• Add incremental IDF update for streaming inserts

References

• ADR-256: docs/adr/ADR-256-hybrid-sparse-dense-search.md
• Research: docs/research/nightly/2026-06-17-hybrid-sparse-dense/README.md
• Gist: docs/research/nightly/2026-06-17-hybrid-sparse-dense/gist.md
• Cormack et al. "Reciprocal rank fusion outperforms Condorcet and individual rank learning methods." CIKM 2009.
• Robertson & Zaragoza "The Probabilistic Relevance Framework: BM25 and Beyond." FnTIR 2009.

🤖 Generated with claude-flow

https://claude.ai/code/session_01NFp4fjSarGCp2xpqJtqP2Z

---

Generated by Claude Code