Skip to content

feat: optional BM25+vector hybrid entity seeding with RRF fusion#3377

Open
BigStartByXuyb wants to merge 1 commit into
HKUDS:mainfrom
BigStartByXuyb:feat/bm25-entity-seeding
Open

feat: optional BM25+vector hybrid entity seeding with RRF fusion#3377
BigStartByXuyb wants to merge 1 commit into
HKUDS:mainfrom
BigStartByXuyb:feat/bm25-entity-seeding

Conversation

@BigStartByXuyb

Copy link
Copy Markdown

Sorry, my English isn't very good. The following introduction was translated with the help of Claude. The commits were merged into one.

Summary

Adds an optional BM25 keyword index over entity names as a parallel seed path beside vector search in _get_node_data. Targets the exact failure mode in #3198: jargon/acronym queries where dense embeddings fail to seed the knowledge graph.

  • Feature is off by default — zero behavior change unless enable_bm25_seeding=True
  • Any BM25 path failure silently falls back to vector-only (a query never fails because of this)
  • bm25s is an optional dependency (lazy import, graceful degradation)

What's Changed

Component Description
BaseKeywordStorage New pluggable storage interface (5th role)
Bm25KeywordStorage Default impl: bm25s + TiktokenTokenizer (BPE, CJK-safe)
fuse_seed_rankings RRF fusion of vector + BM25 ranked lists (k=60)
Query integration QueryParam.enable_bm25_seeding flag, threaded through query chain
Lifecycle Index rebuilt on insert + entity delete; workspace-isolated
Server config LIGHTRAG_KEYWORD_STORAGE + ENABLE_BM25_SEEDING env vars

Design Decisions

  • Entity names only (not chunks): smallest index that solves the seeding problem, avoids IDF drift
  • Full rebuild (not incremental): milliseconds at entity-name scale, idempotent
  • Fusion before graph expansion: one traversal on merged seeds, not two separate retrievals
  • Local path only (MVP): global path is a follow-up

Test Plan

  • 25 unit tests covering interface, BM25 build/search/persist/degrade, RRF fusion, flag-off regression, exception fallback, workspace isolation, CJK
  • ruff check . clean
  • QueryParam().enable_bm25_seeding == False confirmed

Closes #3198

Add an optional BM25 keyword index over entity names as a parallel seed
path beside vector search in LightRAG's local retrieval, fused via
Reciprocal Rank Fusion. Targets jargon/acronym queries where dense
embeddings fail to seed the graph.

Key changes:
- New storage role KEYWORD_STORAGE with BaseKeywordStorage interface
- Bm25KeywordStorage impl backed by optional bm25s library + TiktokenTokenizer
- RRF fusion helper (fuse_seed_rankings) merges vector + BM25 ranked lists
- Query integration: QueryParam.enable_bm25_seeding flag (off by default)
- Rebuild hook on insert and entity delete
- Server config: LIGHTRAG_KEYWORD_STORAGE + ENABLE_BM25_SEEDING env vars
- 25 tests, zero behavior change when disabled

Refs: HKUDS#3198
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RFC: hybrid BM25 + vector retrieval with graph traversal seeding for jargon-heavy domains

1 participant