Multi-field lexical arms: per-field MatchText gates with fusion weights - #132
Conversation
Title/heading payload fields are usually far more precise lexical signals
than chunk bodies, but the lexical arm could only gate on one field - and
duplicating it was broken at the root: everything downstream (fusion
weights, the adaptive Q-learning profile, latency/hit metrics, trace marks,
degraded tags) keys on retriever.name, a CLASS attribute, so two arms of
the same class merged into one weight, one learning profile, and
indistinguishable telemetry.
- Retriever subclasses accept a name= constructor override (instance
attribute shadows the class default; empty names fail loud).
- QdrantTextRetriever gains gate_key: the MatchText gate runs on that
field while the result text still comes from text_key - a hit surfaced
only by the title arm carries its chunk body, not its title, into
fusion and synthesis.
- recall.text_search_fields ({payload_field: fusion_weight}, env
MNEMOSTACK_TEXT_SEARCH_FIELDS="title:2.0,text:1.0") builds one arm per
field via the shared build_qdrant_text_arms factory (all three surfaces:
HTTP server, MCP, CLI). Arm names: qdrant_text for the text_key field
(an existing deployment adding a title arm keeps the body arm learned
profile), qdrant_text:<field> otherwise. Weight 1.0 is left to the
adaptive profile; anything else becomes a static fusion override.
- When set, the mapping REPLACES the arm set, and it requires
text_search: lexical - any other mode fails loud at startup
(ensure_text_fields_mode) instead of silently ignoring the fields;
doctor reports the same verdict.
- mnemostack text-index creates the full-text index for EVERY configured
gate field; doctor live-checks each one. --source bm25 still selects
the whole lexical family (suffixed names match by their base name in
the synthesis source filters).
- Real-server smoke gains a multi-field scenario (per-field text indexes,
title-gated match whose body never mentions the token) - verified
locally against Qdrant v1.18.3 with client 1.18.0 (8/8).
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a54e168f53
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…es, shared query embedding
- CLI _text_search_fields no longer swallows a malformed value into an
empty mapping - serve/text-index degrading to the body-only arm while
doctor shows green was exactly the silent no-op the fields contract
forbids; the error propagates like the servers refusing to boot.
- MCP validates the fields/mode pairing in build_server, not lazily in
the recaller factory: a misconfigured deployment fails before the
server starts advertising tools whose every recall would error.
- The YAML mapping form rejects keys that collide after trimming
("title" vs " title "), matching the env-string duplicate rule.
- QdrantSparseRetriever and BM25Retriever emit sources=[self.name] so a
name= override keys the feedback/Q-learning pipeline consistently with
fusion and telemetry (the text retriever already did).
- The direct-retrievers synthesize path reuses _source_enabled, so the
"bm25" umbrella selects suffixed lexical arms there too.
- _metric_name sanitizes every character outside the Prometheus
identifier grammar, not just the colon (field names are operator
input; a slash would break the scrape).
- Multi-field arm sets share one memoized query embedding
(_SharedQueryEmbedding): N fields cost one billable embed call per
recall instead of N concurrent identical ones.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 81f90c00a4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…s, bool YAML keys - _SharedQueryEmbedding is now single-flight: concurrent arms racing a fresh query all missed the memo and each called the provider - the duplicate work the wrapper exists to prevent. First caller leads, the rest wait on its event. Failures (empty vector, the providers own contract) are returned but never memoized, so a transient outage does not pin the lexical arms dead for that query. - _metric_name is injective again: plain character substitution merged names differing only by punctuation (metadata/title vs metadata-title) into one series - sanitized names now carry a short stable CRC suffix of the exact name. - The YAML mapping form rejects boolean field KEYS (unquoted on/off/ yes/no parse as booleans and would silently target a payload field literally named "True").
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a5fe3a9c42
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…tive metric encoding - build_qdrant_text_arms normalizes its fields through the same parse_text_search_fields as Config.load: programmatic callers (ServerConfig/build_server/library use) could pass weight 0/-1/NaN that satisfies the type annotation while RRF clamps the arm to zero - a silently omitted arm instead of a loud error. - _metric_name switched from a truncated-hash suffix to escape ENCODING (underscore doubles, invalid characters become _xx per UTF-8 byte): distinct sanitized names can never merge into one series, which a 16-bit digest could not guarantee. Names already inside the grammar pass through verbatim, so the historical qdrant_text/vector series keep their identifiers.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 813813fb68
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…spaces, text-index preflight
- MCP build_server now parses the fields mapping eagerly too (not just
the mode pairing): an invalid weight from a programmatic caller failed
only on the first tool call, after embedding/vector init could bury
the actual config error.
- _metric_name namespaces are now provably disjoint: the CLOSED set of
built-in names passes through verbatim (historical series keep their
identifiers); every other name is escape-encoded under an arm_ prefix.
Verbatim passthrough of anything-valid let a literal name collide with
the escape text of another ("a_2f" vs "a/"); no built-in starts with
arm_ and the encoding is injective, so no two names share a series.
- text-index validates the fields/mode pairing BEFORE creating any
index: under text_search=sparse/off it previously mutated Qdrant and
printed "lexical is ready" for a configuration the servers refuse to
boot.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9e37291935
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
An exactly-repeated YAML key (two title: entries under text_search_fields) is collapsed by yaml.safe_load BEFORE any duplicate check can see it - the last weight silently wins. This is a property of the whole config file, not of one field (a repeated collection: behaves identically), so the terminal fix is at the loader: Config.load now parses with a SafeLoader subclass that rejects duplicate mapping keys at any level, matching how every other malformed config value already fails the load.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: aee1330bea
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…bels, from_qdrant name - The strict YAML loader now scans for duplicates BEFORE merge-key flattening and skips << entries: an anchor-based config overriding one inherited key (<<: *defaults + explicit host) is valid YAML whose override must win - round-5 rejected it as a duplicate. Only keys repeated EXPLICITLY in the same mapping fail. - _prometheus_dump escapes label VALUES per the Prometheus text format (backslash, quote, newline): operator-configured arm names reach the degraded-reason label, and an unescaped newline could inject extra exposition lines into the scrape. - BM25Retriever.from_qdrant accepts and forwards name=, so the primary Qdrant-backed construction path can use the identity override too.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d4cdb96405
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
MNEMOSTACK_TEXT_SEARCH_FIELDS="" parses to an empty mapping - a meaningful override, not an absent one: it is the env-var way to CLEAR fields a YAML config sets (without it, an env that also switches text_search away from lexical inherits the file fields and refuses startup with no env-level escape). The override now checks presence, not truthiness.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5cae6585fa
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…amily name overrides
- A non-blank fields string with an empty comma-separated segment (",",
"title:2.0,", ",title") is a malformed template expansion, not intent:
it previously collapsed to an empty mapping and silently enabled the
default body-only arm. Only a genuinely blank string clears the
mapping now.
- A name= override may carry its own family name (suffixed or not) but
never another reserved family: the recaller keys semantics on exact
identities (vector drives the vector floor and the no-vector
fallback) and the source filters partition by base name - a BM25 arm
named "vector" would masquerade through both.
1 participant
Title/heading payload fields are usually far more precise lexical signals than chunk bodies, but the lexical arm could only gate on one field — and duplicating it was broken at the root: everything downstream (fusion weights, the adaptive Q-learning profile, latency/hit metrics, trace marks, degraded tags) keys on
retriever.name, previously a CLASS attribute, so two arms of the same class merged into one weight, one learning profile, and indistinguishable telemetry.What ships
name=constructor override on the lexical retrievers (instance attribute shadows the class default; empty names fail loud) — the general fix for the keying collision.gate_keyonQdrantTextRetriever: the MatchText gate runs on that field while the result text still comes fromtext_key— a hit surfaced only by the title arm carries its chunk body, not its title, into fusion and synthesis.recall.text_search_fields({payload_field: fusion_weight}, envMNEMOSTACK_TEXT_SEARCH_FIELDS="title:2.0,text:1.0"): one arm per field via the sharedbuild_qdrant_text_armsfactory, wired on all three surfaces (HTTP server, MCP, CLI). Arm names:qdrant_textfor thetext_keyfield (an existing deployment adding a title arm keeps the body arm's learned profile),qdrant_text:<field>otherwise. Weights apply at the FUSION level (the gate filters, it cannot score); weight 1.0 is left to the adaptive profile, anything else is a static override.text_search: lexical— any other mode refuses to boot (ensure_text_fields_mode), anddoctorreports the same verdict. The parser rejects malformed weights, duplicate fields, and YAML bool typos (title: yes).mnemostack text-indexcreates the full-text index for EVERY configured gate field;doctorlive-checks each one.--source bm25still selects the whole lexical family (suffixed names match by base name in the synthesis source filters). Metric identifiers sanitize the colon (Prometheus reserves it for recording rules); traces/weights keep exact names.Verification
Full suite 1466 passed (32 new in
tests/test_multi_field_lexical.py: config parsing incl. rejection cases, factory naming/weights, gate-vs-text semantics on in-memory Qdrant, tenant composition, fusion-weight keying, source-umbrella matching, CLI wiring incl. acmd_serve→ServerConfigforwarding regression test, multi-fieldtext-index). Ruff + mypy clean.