feat(ts-oss): add Qdrant server-side BM25 keywordSearch() + filter indexes

[bmsvinci1729]

Summary

Implements keywordSearch() for the TypeScript OSS Qdrant adapter, which was previously a null stub. So hybrid search on Qdrant silently ran as semantic + entity only, with no BM25 keyword signal.
The following PR brings TS Qdrant to parity with the other TS stores (PGVector, Azure AI Search) and with the Python Qdrant adapter.

Closes #5828

Direction was confirmed by a maintainer in #5828 (server-side BM25, Qdrant ≥ 1.15.2 floor, payload-index fix in the same PR).

What changed (mem0-ts/src/oss/src/vector_stores/qdrant.ts)

• Collection setup: add a bm25 named sparse-vector slot with modifier: "idf" so Qdrant computes IDF over the live corpus.
• insert() / update(): attach the memory text so Qdrant encodes the BM25 sparse vector server-side ({ text, model: "Qdrant/bm25" }).
• keywordSearch(): query the bm25 slot (using: "bm25") and return { id, payload, score }[]
• Capability guard (per maintainer request): detect whether a collection has the bm25 slot. Collections created before this change lack it -> keywordSearch() returns null (existing behavior) and a console.warn is logged, also no BM25 vectors are written, so semantic search is unchanged, thus ensuring no breakage on upgrade.
• Filter payload indexes: create keyword payload indexes for ["user_id", "agent_id", "run_id", "actor_id"] at collection setup (skipped for local/embedded mode, best-effort per field). Without these, filtered keyword/semantic search is rejected by remote Qdrant ("Index required but not found"). Mirrors Python's _create_filter_indexes().

Requirements

• Qdrant ≥ 1.15.2 for the built-in server-side Qdrant/bm25 model.

Tests

Added qdrant-keyword-search.test.ts (30 tests, following the existing qdrant-url-port.test.ts style) covering collection setup (bm25 slot, 409/401/403 paths, dimension guard), insert/update BM25 attachment, query result mapping, graceful null fallback on errors, and filter construction.

Also validated end-to-end against a real Qdrant Cloud cluster (v1.18.2): insert -> server-side BM25 encode -> keyword query -> correctly ranked results, including the metadata-filtered case. Happy to share that script if useful.

Type of Change

• New feature (non-breaking change that adds functionality)

Checklist

• My code follows the project's style guidelines
• I have performed a self-review of my code
• I have added tests that prove my fix/feature works
• New and existing tests pass locally

[CLAassistant]

All committers have signed the CLA.

[kartik-mem0]

Changes Requested

Great work on this — the capability-gated design is exactly right, the test coverage is thorough (30 tests, all passing), and the backward-compatibility path for pre-existing collections is solid. Two small asks before merge.

Strengths

• qdrant.ts — capability guard (_hasBm25Slot, lines +63–64 in the diff) mirrors Python's _has_bm25_slot exactly. Detecting at init time and gating every BM25 write path is the correct shape.
• Collection setup: vectors:{size, distance} (unnamed dense) + sparse_vectors:{bm25:{modifier:"idf"}} — correct hybrid pattern; the unnamed dense slot is untouched so existing search() calls are not affected.
• Filter index creation (createFilterIndexes) mirrors _create_filter_indexes() in Python, same four fields, same best-effort-per-field try/catch, same remote-only guard.
• client.query() return shape ({points: [...]}) is correct per the JS client type definitions — response.points.map() is safe.
• textLemmatized (camelCase) matches what memory/index.ts:954,1790 stores in the payload — the field name is right.
• 30/30 tests pass, covering collection setup (9 variants including 409/401/403/slot-present/slot-absent), insert/update BM25 attachment, result mapping (null/missing fields/numeric id), and filter construction.

Issues

Important (should fix before merge)

1. qdrant.ts — Python/TS BM25 encoding divergence is undocumented. Python uses client-side fastembed (SparseTextEmbedding(model_name="Qdrant/bm25")); this PR uses server-side Qdrant inference ({text, model: "Qdrant/bm25"}). Two consequences: (a) Python requires the fastembed extra, TS requires Qdrant ≥ 1.15.2 — these are different deployment constraints; (b) if fastembed and Qdrant's server encoder produce different IDF weights (possible across versions), Python and TS BM25 scores won't be numerically equivalent for the same corpus. Neither the code nor the docs mentions this. A brief comment on BM25_MODEL or buildPointVector is enough: something like "TS uses server-side Qdrant inference (no fastembed dep, requires Qdrant ≥ 1.15.2); Python uses client-side fastembed — IDF values may differ across implementations." The PR body confirms the 1.15.2 floor exists but it doesn't surface in code for the next maintainer to find.

2. qdrant-keyword-search.test.ts — no test exercises store.search() (the semantic/dense path) after _hasBm25Slot is set to true. The 440-line test file covers keywordSearch() exhaustively but has zero calls to store.search(). The current implementation is correct (plain number[] on client.search() targets the unnamed dense slot — verified against the JS client type definitions), but there's no regression guard for the semantic path. One test covers it cleanly:

   it("search() still uses a plain vector after BM25 init", async () => {
     const { store, client } = await freshStore(); // _hasBm25Slot = true
     await store.search([0.1, 0.2, 0.3], 5);
     expect(client.search).toHaveBeenCalledWith(
       "test_memories",
       expect.objectContaining({ vector: [0.1, 0.2, 0.3] }),
     );
   });

Minor (optional)

• qdrant.ts constructor — when config.client is passed, _isRemote stays at its default true with no explicit assignment. Works correctly (matches Python's is_local=False for pre-configured clients) but a short comment would help future readers: // pre-configured client → treat as remote (mirrors Python is_local=False).
• createFilterIndexes — Python logs logger.debug(...) on index errors; TS's catch (_) {} is silent. A console.debug would help operators diagnose filtered-search failures.
• The 5× as any casts are necessary given the JS client's type gaps around sparse vectors — worth a brief "SDK types predate sparse_vectors" note so they don't look like lazy shortcuts.

Evidence

• CI: Vercel preview fails (expected on fork PRs, non-blocking). CLA signed. No real test/lint/build failures.
• Targeted tests: pnpm run test -- --testPathPattern="qdrant-keyword-search" → 30 passed in 0.257s (exit 0) against HEAD 3a9d899.
• Red-green proof: N/A — this implements a feature stub, not a regressive bug. Semantic search regression investigated and confirmed NOT present: ensureCollection creates an unnamed dense vector (not a named-vector collection), so client.search({vector: plainArray}) continues to target the default slot. Verified via @qdrant/js-client-rest type definitions (NamedVectorStruct = number[] | NamedVector | NamedSparseVector).
• Linked issue: TS OSS Qdrant adapter: keywordSearch() is a null stub - hybrid search runs without BM25 on Qdrant #5828 — PR directly addresses the confirmed null-stub root cause.
• Cross-package check: TS design mirrors Python closely (_hasBm25Slot/_has_bm25_slot, createFilterIndexes/_create_filter_indexes, same four indexed fields, same _isRemote/is_local guard). Known divergence: server-side vs. client-side BM25 encoding — see Important item 1.
• Could NOT verify: live Qdrant server behavior for BM25 inference (code-tracing only). Numerical equivalence of Python fastembed vs. TS server-side BM25 scores not tested cross-SDK.

Assessment

Ready to merge: With fixes — please address the two Important items (a comment + one test); they're both small. Happy to re-review once pushed.

[bmsvinci1729]

Thanks, will fix accordingly, and acknowledge, thanks for the timely reviews .

[bmsvinci1729]

Addressed both required items:

1. Added a comment on BM25_MODEL documenting the server-side (TS) vs client-side fastembed (Python) divergence and the ≥1.15.2 floor. Cross-SDK score equivalence is now flagged in the comment as a known divergence.
2. Added the search() regression test confirming the dense path still targets the unnamed slot after BM25 init.

Also picked up the three optional nits: the constructor _isRemote comment, console.debug in createFilterIndexes, and a note on the as any casts.

Looking forward to further reviews and fixes (if any).

Thanks

[kartik-mem0]

Re-review @ 4d35115 — Changes Requested (Prettier only)

Both Important items and all three optional nits: ✓ addressed.

Verified fixed:

• BM25 encoding comment (qdrant.ts lines 7–10) — server-side approach, Qdrant ≥ 1.15.2 floor, and Python/TS score-equivalence caveat are all documented. Exactly right.
• search() regression test — "search() still targets the unnamed dense slot after BM25 init" (line 443 of the test file) asserts client.search receives vector: [0.1, 0.2, 0.3] after BM25 init. 31/31 tests pass.
• _isRemote constructor comment, console.debug in createFilterIndexes, and as any explanatory note: all present.

One remaining blocker — CI Prettier failure:

TypeScript SDK / build_ts_sdk fails on the Lint step (Node 20 and 22). The issue is in the new comment block you added:

qdrant.ts line 5-6:  double blank line (blank line after imports + blank line before comment = 2 consecutive blanks)
qdrant.ts line 9:    trailing space after "the " — "…, but IDF weights, and therefore the↵" has a trailing space

Run npx prettier --write src/oss/src/vector_stores/qdrant.ts from mem0-ts/, commit, and CI should go green. That's the only thing left.

Evidence:

• CI log: [warn] src/oss/src/vector_stores/qdrant.ts → Code style issues found. Run Prettier with --write to fix. (job 83405020696, Lint step)
• Local npx prettier --check src/oss/src/vector_stores/qdrant.ts: exit 1, same two locations confirmed
• Targeted tests at 4d35115: 31 passed (0.256s), exit 0

[bmsvinci1729]

Sure, working on it
Thanks

[bmsvinci1729]

Pushed the prettier fix - CI Gate is waiting on workflow approval whenever you get a chance. Thanks !!

[kartik-mem0]

Verified against 2ddd158 — Prettier is clean.

`mem0-ts/src/oss/src/vector_stores/qdrant.ts:5–9` — double blank line and trailing space both gone: `npx prettier --check src/oss/src/vector_stores/qdrant.ts` → All matched files use Prettier code style!

LGTM — ready for human review & squash-merge

What this PR does

Replaces the null stub `keywordSearch()` on the TypeScript Qdrant store with a real BM25 hybrid-search implementation backed by Qdrant's server-side inference (≥ 1.15.2), including capability-gated collection setup, payload index creation, and a dense-path regression test.

Strengths

• `qdrant.ts:284–352` — capability gate (`_hasBm25Slot`) is clean: pre-existing collections get the dense path untouched, new collections opt in via `initialize()`. No regressions to existing callers.
• `qdrant.ts:7–10` — BM25 comment documents the Python↔TS divergence (server-side vs fastembed client-side), the ≥ 1.15.2 floor, and the score-equivalence caveat — exactly what a future maintainer needs.
• `qdrant-keyword-search.test.ts:442–463` — `search()` regression test confirms the dense path still targets the unnamed slot after BM25 init.
• 31 targeted tests covering collection setup, insert/update, keyword query construction + filters, and result mapping.

Issues

Minor: none remaining (nits addressed in prior push).

Evidence

• CI: GitHub Actions check suite registered but 0 check runs at 2ddd158 — fork PR awaiting "approve to run"; Vercel advisory fail (non-blocking, expected on all forks). Previous required-check failure (Prettier lint) is now absent.
• Local Prettier check (2ddd158): `cd mem0-ts && npx prettier --check src/oss/src/vector_stores/qdrant.ts` → `All matched files use Prettier code style!` ✓
• Targeted tests: 31 passed at 4d35115 (prior head); new commit is formatting-only (+1/-2 in qdrant.ts), test file unchanged.
• Red-green proof: N/A — feature addition, not a bug fix.
• Linked issue: TS OSS Qdrant adapter: keywordSearch() is a null stub - hybrid search runs without BM25 on Qdrant #5828 (TS OSS Qdrant `keywordSearch` null stub) — PR replaces the stub with a real implementation.
• Cross-package check: Python adapter uses fastembed client-side BM25; documented divergence is acceptable and noted in code.
• Could NOT verify: live Qdrant ≥ 1.15.2 server behavior — code tracing only.

Assessment

Ready to merge: Yes (squash) · Est. human time: ~2 min
All substantive findings resolved across both pushes; the sole remaining blocker (Prettier formatting) is now fixed. Squash-merge when CI runs green after "approve to run".

[bmsvinci1729]

Hi @kartik-mem0, hope you're doing well! Just wanted to check if there are any updates on the PR. If there are any remaining changes or anything else I can do to help move it forward, please let me know. Thanks!

[bmsvinci1729]

Hi @kartik-mem0
Just a quick follow-up — our work is dependent on hybrid search support for Qdrant in mem0, so we're currently blocked until this is merged. Could you please let us know if there are any further changes needed on our end? Otherwise, it would be great if you could move forward with merging this at your earliest convenience.
Thank you so much for your help with this!

[bmsvinci1729]

Hi @kartik-mem0, the conflicts have been resolved with respect to the recent client refactoring on main. I've integrated the BM25 setup and tests into the new ensureClient() flow. Please do verify and help us get it merged.
Thank you !!

[kartik-mem0]

Changes Requested @ d1d9dae (reversing my earlier LGTM)

@bmsvinci1729 first: apologies for the delay, and thank you for staying on top of every round including the main merge.

My previous review said "ready to merge" with the caveat "Could NOT verify: live Qdrant >= 1.15.2 server behavior, code tracing only." I've now run this against real Qdrant servers in Docker, and that verdict was wrong. There is a silent data-loss path that code reading alone did not surface. Everything below is from actual runs, not inspection.

What works (verified on self-hosted Qdrant v1.18.2)

The good news up front: server-side Qdrant/bm25 inference works on plain self-hosted Qdrant, not just Cloud. That was the main open risk from my earlier review and it is disproven. All of these passed against a real server:

Check	Result
collection sparse_vectors config	{"bm25":{"modifier":"idf"}}
insert() with server-side BM25	OK, all points stored
stored vector shape	{"bm25":{"indices":[...],"values":[1.674,...]},"":[0.109,0.986,...]} (the {"": vector} unnamed-slot trick is accepted)
keywordSearch("espresso")	2 correct hits with real IDF-weighted scores
filtered keywordSearch	correctly narrowed to user_id=alice
search() dense path	unaffected: 1.0000 / 0.7568 / 0.2292
payload indexes	all four created as keyword
legacy collection (no bm25 slot)	warns once, keywordSearch -> null, insert + semantic unaffected
update() re-encode	old term drops out of results, new term becomes searchable
unit tests / prettier	31/31 pass (0.257s), prettier clean, CI green

The capability-gate design is right, and the backward-compat path for pre-existing collections genuinely holds up under a real server. No complaints there.

Blocker: silent data loss when Qdrant inference is unavailable

Same code, same script, against Qdrant v1.14.1:

=== TEST 1: fresh collection, server-side BM25 insert ===
sparse_vectors config: {"bm25":{"modifier":"idf"}}     <- createCollection SUCCEEDS
INSERT: FAILED -> 500 Internal Server Error
raw: {"status":{"error":"Service internal error: InferenceService is not initialized.
      Please check if it was properly configured and initialized during startup."}}
points in collection: {"count":0}

Root cause: ensureCollection sets _hasBm25Slot = true immediately after createCollection resolves:

await this.client.createCollection(name, createParams as any);
if (enableBm25) {
  this._hasBm25Slot = true;   // <- optimistic; server may not support inference
}

Older servers accept the sparse_vectors config just fine (sparse vectors have existed since 1.7), so the guard never trips. Then buildPointVector attaches {text, model: "Qdrant/bm25"} to every point, and every single upsert fails with a 500.

Why this is silent rather than loud: memory/index.ts:1034 has a two-level insert fallback whose inner catch only logs.

try {
  await this.vectorStore.insert(allVectors, allIds, allPayloads);
} catch {
  for (let i = 0; i < allIds.length; i++) {
    try { await this.vectorStore.insert([allVectors[i]], [allIds[i]], [allPayloads[i]]); }
    catch (e) { console.error(`Failed to insert memory ${allIds[i]}: ${e}`); }  // swallowed
  }
}

I ran that exact loop against 1.14.1:

Failed to insert memory aaaaaaaa-0000-0000-0000-000000000001: Internal Server Error
Failed to insert memory aaaaaaaa-0000-0000-0000-000000000002: Internal Server Error
=> add() would now RETURN SUCCESS with ids: [ '...0001', '...0002' ]
=> actual points stored: {"count":0}

So add() returns success with memory IDs that were never persisted. Same failure class as #5557.

Who is affected (new collections only; pre-existing collections without the slot stay on the safe path):

1. Self-hosted Qdrant < 1.15.2: every write lost.
2. Qdrant Cloud clusters created before 2025-07-07. Per Qdrant's Cloud Inference docs, "Inference is enabled by default for all new clusters, created after July, 7th 2025"; older clusters must activate it manually from the Cluster Detail page. Such a cluster runs a recent Qdrant version with inference off, and fails identically.

Point 2 is why a server-version check will not close this: the version can be 1.18 and inference still be unavailable.

Suggested fix

Make the capability detection reactive instead of optimistic: if an upsert fails while _hasBm25Slot is true, turn it off and retry once with the plain dense vector. That covers the old-version case and the inference-disabled-Cloud case with one mechanism, needs no version parsing, and degrades to exactly today's behavior (semantic-only, keywordSearch -> null). Roughly:

private async upsertPoints(points: any[]): Promise<void> {
  try {
    await this.client.upsert(this.collectionName, { points });
  } catch (error) {
    if (!this._hasBm25Slot) throw error;
    // Server accepted the bm25 slot but cannot run inference (Qdrant < 1.15.2,
    // or a Cloud cluster with inference disabled). Downgrade permanently and retry
    // dense-only so writes are never lost.
    this._hasBm25Slot = false;
    console.warn(
      "Qdrant rejected the server-side BM25 vector; falling back to dense-only inserts. " +
        "BM25 keyword scoring requires Qdrant >= 1.15.2 with inference enabled.",
      error,
    );
    await this.client.upsert(this.collectionName, {
      points: points.map((p) => ({ ...p, vector: p.vector?.[""] ?? p.vector })),
    });
  }
}

Then route both insert() and update() through it. Two tests would lock it in: upsert rejects once then succeeds on retry with a plain vector, and _hasBm25Slot is false afterwards so keywordSearch() returns null.

Worth noting Python already degrades correctly here for free: it encodes client-side with fastembed, so a missing encoder just omits the sparse vector and the insert still lands. Staying with server-side inference in TS is still the right call (no heavy dependency, and no JS BM25 encoder matches Qdrant's tokenization); it just needs the same "never lose the write" property.

Minor, non-blocking

• ensureCollection 409 branch calls await this.createFilterIndexes(name) without the name === this.collectionName guard the success branch has, so on re-init memory_migrations picks up four unused user_id/agent_id/run_id/actor_id indexes. One-line fix.
• The >= 1.15.2 floor only lives in a code comment. docs/components/vectordbs/dbs/qdrant.mdx mentions neither BM25/hybrid search nor the version requirement. Worth a short "Hybrid search requires Qdrant >= 1.15.2 with inference enabled" note.
• _isRemote = false in the path branch: @qdrant/js-client-rest has no embedded mode, so a path-only config actually falls through to localhost:6333 and then skips index creation. Pre-existing quirk this PR inherits, not caused by it, flagging only so it is on record.

Evidence

• Servers: qdrant/qdrant:v1.18.2 (pass) and qdrant/qdrant:v1.14.1 (insert 500, count: 0), both via Docker, @qdrant/js-client-rest@1.18.0, exercising the real Qdrant class from this branch (no mocks).
• Unit tests: npx jest src/oss/tests/qdrant-keyword-search.test.ts at d1d9dae -> 31 passed, exit 0.
• Prettier: npx prettier --check src/oss/src/vector_stores/qdrant.ts src/oss/tests/qdrant-keyword-search.test.ts -> "All matched files use Prettier code style!"
• CI: all required checks green (TS SDK build + integration on Node 20/22, CI Gate pass). Vercel fail is the usual non-blocking fork authorization.
• Cross-SDK: Python mem0/vector_stores/qdrant.py already implements the whole feature (_has_bm25_slot, _encode_bm25, keyword_search, _create_filter_indexes, same text_lemmatized or data preference), so this PR is TS reaching parity, not a gap in both.
• Could not verify: numerical equivalence of fastembed vs server-side IDF weights across SDKs (documented as a known divergence in your comment, which is the right call).

Assessment

Score: 7/10. Ready to merge: not yet, one fix.

The feature is real, correct on supported servers, and the test coverage is better than most PRs this size. The single blocker is the optimistic _hasBm25Slot = true, and the fallback above is a contained change. Push it and I will re-review promptly; I owe you a fast turnaround after this one sat.

[kartik-mem0]

Pushed the fix myself + retracting one of my findings

@bmsvinci1729 you've been through enough rounds on this one, so rather than send you back for another, I merged main and pushed the fallback fix directly to your branch (d1d9dae7..8a4b6e3d). Nothing of yours was rewritten, it's a fast-forward on top of your work. Please pull before you touch the branch again.

First, a correction: my memory_migrations finding was wrong

I claimed the 409 branch of ensureCollection calls createFilterIndexes(name) without the name === this.collectionName guard. That's not true, the call is already nested inside that guard. I misread the brace nesting and should have checked before posting. Confirmed empirically as well, on both servers memory_migrations comes back with payload_schema: {}. Your code was right, sorry for the noise.

The blocker itself stands, and everything below is from the fix as pushed.

What changed

insert() and update() now route through one upsertPoints() helper. If an upsert fails while _hasBm25Slot is true, it flips the flag off, warns once, and retries with the bm25 slot stripped:

private async upsertPoints(points) {
  try {
    await this.client.upsert(this.collectionName, { points });
  } catch (error) {
    if (!this._hasBm25Slot) throw error;
    this._hasBm25Slot = false;
    console.warn(/* ... requires Qdrant >= 1.15.2 with inference enabled ... */);
    const denseOnly = points.map((p) => ({
      ...p,
      vector: Array.isArray(p.vector) ? p.vector : p.vector[""],
    }));
    await this.client.upsert(this.collectionName, { points: denseOnly });
  }
}

I deliberately left ensureCollection's optimistic _hasBm25Slot = true alone. It has to stay optimistic, there is no pre-flight probe that distinguishes "accepts the config" from "can run inference", so the first failed write is the only honest signal.

Two notes on why it's shaped this way:

• Genuine errors still propagate. The if (!this._hasBm25Slot) throw error guard means a real network failure on a legacy collection rejects as before, it does not get silently retried. Locked in by a test.
• The downgrade is sticky. One warning per process, not one per write, and no re-probing on later inserts.

Also added the >= 1.15.2 + inference requirement to docs/components/vectordbs/dbs/qdrant.mdx, since it previously only existed in a code comment.

Verification

Live servers, both lanes, real Qdrant class, no mocks:

	v1.18.2 (inference on)	v1.14.1 (no inference)
insert()	no throw, 0 warnings	no throw, 1 warning
points persisted	3	3 (was 0 on your branch)
stored vector shape	["bm25", ""]	plain dense array
keywordSearch("espresso")	0.7868 / 0.7847	null
filtered keywordSearch	narrowed to alice	null
search() dense	[1, 0.6302, 0.2387]	identical
2nd insert	0 extra warnings	0 extra warnings
update() re-encode	old term drops, new term searchable	null, write lands
payload indexes	4 keyword indexes	4 keyword indexes
memory_migrations	{}	{}

The silent-loss path specifically. Replaying memory/index.ts's two-level insert fallback against v1.14.1:

before: => add() would now RETURN SUCCESS with ids: [...]
        => actual points stored: {"count":0}
after:  => add() reports success for: 2 ids
        => points ACTUALLY persisted: {"count":2}

Tests. 31 -> 36 in qdrant-keyword-search.test.ts. The 5 new ones cover: insert downgrades and retries with a plain array; update does the same; a legacy-collection network error still rejects with only one upsert attempt; the next insert succeeds first try; memory_migrations gets zero createPayloadIndex calls (the regression lock for my bad finding above). Stashing the fix turns exactly 3 of them red.

• npx jest src/oss -> 1224 passed, 78 suites
• npx prettier --check -> clean
• pnpm run build -> ok

Repo sandbox, end-to-end through Memory.add() + search() on the oss-qdrant lane:

main:       FAIL (3.4s): bm25Score is 0 for query "invoice" ...
this branch: PASS (3.8s)

Assessment

9/10, merging. The feature was right, the design was right, and the test coverage was already above average for a PR this size. The one real defect was a fallback that couldn't have shown up without a server that accepts the config but can't act on it. Thanks for the persistence through the review rounds, and again, sorry for the wrong minor finding and the delay.

[bmsvinci1729]

Thanks for pushing that fallback fix, and for this whole journey -- my first open source contribution, learned a lot along the way! Appreciate all the review effort here !!