Skip to content

Commit d40ceba

Browse files
nickrociclaude
andcommitted
feat(recall): upgrade Tier-1 retrieval with nomic embedder + cross-encoder rerank
Replace the 2021-era all-MiniLM-L6-v2 embedder with nomic-embed-text-v1.5 (asymmetric query/document prompts, Matryoshka 768→128, 1024-token seq cap) and add a cross-encoder rerank stage (ms-marco-MiniLM-L-12-v2) between RRF fusion and the boost step. Three-stage retrieval — recall (BM25 + embeddings) → fusion (RRF) → precision (cross-encoder) — gives the priming snippet meaningful precision lift on a 169-entry library. Daemon eagerly loads both models + runs a full-pipeline warmup before the priming RPC socket opens, so the first hook request hits hot MPS kernels instead of a 6.8s graph-compile cliff. Steady-state latency ~300-500ms, within the bumped 2s hook budget (500ms accommodates slower hardware). Calibrations for the new embedder distribution: - EMBEDDING_NOISE_FLOOR raised 0.25 → 0.50 (nomic's cosine sits higher) - _RERANK_SCORE_FLOOR at -3.0 catches weak-but-real matches; the embedding floor handles outright gibberish at the source. Other adjustments: - Cross-encoder load failures now propagate (no silent degradation) - Hook budget 200ms → 2000ms; server-side request timeout 1.0s → 1.8s - Logging setup pins httpcore/httpx/transformers/sentence_transformers et al. at WARNING so -v doesn't drown the daemon log in HF wire traffic - einops added as runtime dep (required by nomic's custom RoPE module) README updated with HF model download expectations, ~25s boot time, ~2.5GB resident footprint, and a corrected Tier-1 description. Tests: 681 passing (439 daemon + 174 search + 68 hooks). Three existing tests updated to reflect the new "quality > quantity" rerank contract. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent d190539 commit d40ceba

15 files changed

Lines changed: 592 additions & 62 deletions

File tree

README.md

Lines changed: 49 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -54,7 +54,7 @@ The agent can't afford to query the library for everything it's about to do, and
5454

5555
| Tier | Cognitive analog | Latency | Trigger | What it does |
5656
|---|---|---|---|---|
57-
| **1. Ambient priming** | Familiarity / spreading activation (Collins & Loftus, 1975) | ~0 (already in prompt context) | Daemon refreshes `hot-context.md` after every batch | Top 5 most-relevant entries injected as `additionalContext` on every UserPromptSubmit. Agent has them "in mind" without asking. Hybrid retrieval: BM25 + sentence-transformer embeddings merged via RRF, boosted by `reinforced` counter. ≤500 char budget. |
57+
| **1. Ambient priming** | Familiarity / spreading activation (Collins & Loftus, 1975) | <2s (Unix-socket round trip to warm daemon; budget enforced by the hook) | Hook fires a priming RPC to the daemon on every UserPromptSubmit | Top 5 most-relevant entries injected as `additionalContext` on every UserPromptSubmit. Agent has them "in mind" without asking. Three-stage retrieval: BM25 + sentence-transformer embeddings (nomic-embed-text-v1.5) → RRF fusion → cross-encoder rerank (ms-marco-MiniLM-L-12-v2), boosted by `reinforced` counter + project-scope prior. ≤1500 char budget. |
5858
| **2. Deliberate recall** | Hippocampal recollection (Yonelinas, 2002) | 30-60s | `/ultan-advisor <question>` invocation | Sonnet Librarian searches deeply, Opus Scholar synthesises a referenced answer. The drill-down when the priming snippet isn't enough. |
5959
| **3. Acute notice** | Orbital-PFC stop signal (Aron et al., 2014) | <100ms synchronous | PreToolUse hook on every tool call | Default `advise`: pattern-matches the tool call against entries with `block_triggers`; on match, emits an `additionalContext` FYI — *"📚 Library note: [[X]] applies here"* — and the tool proceeds. **The agent decides.** Opt-in `severity: block` is reserved for genuinely dangerous actions (`rm -rf /`, force-push to main); only then does the hook hard-deny. Like a human noticing a relevant memory mid-action: noticed, not paralysed. |
6060

@@ -66,7 +66,7 @@ The library is structurally a graph: folder hierarchy gives the tree, wikilinks
6666

6767
Retrieval over the graph is split across tiers:
6868

69-
- **Tier 1 (priming) is text-only**hybrid BM25 + sentence-transformer embeddings merged via RRF, boosted by the `reinforced` counter. The retriever does not traverse the graph; it returns top-k entries by text similarity, with wikilinks appearing in the output as *hints*.
69+
- **Tier 1 (priming) is text-only**three-stage retrieval: BM25 + sentence-transformer embeddings (nomic-embed-text-v1.5) fused via RRF, then re-ordered by a cross-encoder reranker (ms-marco-MiniLM-L-12-v2) co-attending over each `(query, body)` pair, and finally boosted by the `reinforced` counter plus project-scope prior. The retriever does not traverse the graph; it returns top-k entries by relevance, with wikilinks appearing in the output as *hints*.
7070
- **Tier 2 (deliberate recall) is agent-driven graph traversal.** The agent sees wikilinks in priming context and follows them via the `ultan-search` skill, which returns the entry plus its local neighborhood — siblings, subfolders, parent README — in one read. The agent decides which edges to follow and when to stop, the same "memory as tools" pattern Letta and Wire have shown beats pre-baked retrieval expansion on cross-document queries.
7171

7272
The deliberate choice: deterministic-and-cheap text retrieval at always-on Tier 1, semantic-and-expensive graph traversal at on-demand Tier 2. PageRank-style structural expansion at Tier 1 is a possible addition (see *Roadmap*), not a current capability.
@@ -79,7 +79,9 @@ The deliberate choice: deterministic-and-cheap text retrieval at always-on Tier
7979
# 1. Sync the daemon's deps (uv-managed)
8080
cd daemon && uv sync --group dev
8181

82-
# 2. Sync the search CLI (separate venv, shared BM25 implementation)
82+
# 2. Sync the search CLI (separate venv, shared BM25 implementation).
83+
# Pulls in sentence-transformers + einops; HuggingFace model weights
84+
# are downloaded on first daemon start (see step 4 below).
8385
cd ../tools/search && uv sync
8486

8587
# 3. Install the slash commands and hooks
@@ -100,10 +102,47 @@ cd /path/to/ultan/daemon && uv run agent-mem-daemon -v
100102
# ~/.agent-mem/knowledge/ as the Scholar approves them.
101103
```
102104

103-
When the daemon is running, the ``UserPromptSubmit`` hook makes a sub-100 ms
104-
Unix-socket call into it to get a priming snippet keyed on your *current
105-
prompt* (not the last batch's curation). When the daemon is down, the hook
106-
falls back to a tiny in-process lexical scan so you still see relevant entries.
105+
### First-start expectations (model downloads)
106+
107+
The Tier-1 retrieval pipeline uses two open HuggingFace models, downloaded
108+
into your local cache on first daemon start (`~/.cache/huggingface/`,
109+
re-used by every subsequent run):
110+
111+
| Model | Role | Size on disk |
112+
|---|---|---|
113+
| [`nomic-ai/nomic-embed-text-v1.5`](https://huggingface.co/nomic-ai/nomic-embed-text-v1.5) | sentence-transformer embedder (asymmetric query/document, Matryoshka 768→128) | ~270 MB |
114+
| [`cross-encoder/ms-marco-MiniLM-L-12-v2`](https://huggingface.co/cross-encoder/ms-marco-MiniLM-L-12-v2) | reranker that scores (query, body) pairs after RRF fusion | ~130 MB |
115+
116+
Both are downloaded **anonymously** — no HuggingFace account or token is
117+
needed. If you have outbound network restrictions, pre-cache them on a
118+
machine with internet access and copy `~/.cache/huggingface/hub/`
119+
across. nomic-embed-v1.5 requires `trust_remote_code=True` because its
120+
RoPE attention module ships as custom code in the model repo; the daemon
121+
sets this only for the nomic model name, not as a global default.
122+
123+
Boot time on a warm M-series Mac (cold models, warm HF cache):
124+
125+
- ~1 s for the BM25 index
126+
- ~17 s for the embedding model to load + index the library + JIT-warm
127+
the query-side MPS kernels
128+
- ~5 s for the cross-encoder to load + run a full-pipeline warmup query
129+
130+
Total: ~25 s from `uv run agent-mem-daemon -v` to "priming RPC ready".
131+
The startup runs a single end-to-end priming search before opening the
132+
socket — by the time the hook can connect, every MPS kernel in the
133+
retrieval path is JIT-compiled, so the first real request is hot.
134+
135+
Steady-state resident footprint: **~2.5 GB physical memory** on Apple
136+
Silicon (unified memory, includes Metal device-side allocations); peak
137+
of ~3.4 GB during boot warmup. On a discrete-GPU box the split between
138+
RSS and GPU memory looks different; on CPU-only hardware the daemon
139+
still works but per-request rerank latency drifts up.
140+
141+
When the daemon is running, the `UserPromptSubmit` hook makes a Unix-socket
142+
call into it (hard cap 2 s; warm steady-state ~300-500 ms) to get a
143+
priming snippet keyed on your *current prompt* (not the last batch's
144+
curation). When the daemon is down, the hook falls back to a tiny
145+
in-process lexical scan so you still see relevant entries.
107146

108147
To save a memory explicitly: `/ultan never deploy to prod without my explicit OK`.
109148

@@ -224,12 +263,11 @@ Two known gaps relative to where the bio framing points.
224263

225264
### Tier 1 graph signal
226265

227-
Tier 1 priming is currently text-only (BM25 + embeddings + RRF + `reinforced`). Two cheap structural signals sit unused:
266+
Tier 1 priming uses BM25 + embeddings (nomic-embed-text-v1.5) + RRF + cross-encoder rerank (ms-marco-MiniLM-L-12-v2) + `reinforced` + project-scope prior. One cheap structural signal still sits unused:
228267

229-
- **Folder-scope prior** — entries under `projects/<current-cwd-slug>/` should rank higher on text-similarity ties.
230-
- **Wikilink-density boost** — heavily inbound-linked entries are load-bearing (PageRank's intuition, cheap to compute on a few-thousand-entry library). One extra term in the RRF fusion.
268+
- **Wikilink-density boost** — heavily inbound-linked entries are load-bearing (PageRank's intuition, cheap to compute on a few-thousand-entry library). One extra term added before the cross-encoder rerank (so densely-linked entries get a better chance of surfacing into the rerank's candidate pool).
231269

232-
Neither requires architectural change — both are reranker tweaks in `priming.py`. The agent-driven Tier 2 traversal stays as-is; this is purely about giving Tier 1 a structural prior.
270+
Doesn't require architectural change — it's a small addition in `priming.py` before `_rerank_candidates`. The agent-driven Tier 2 traversal stays as-is; this is purely about giving Tier 1 a structural prior on top of the relevance-only signal the cross-encoder already provides.
233271

234272
### Forgetting (LTD) with surprise-calibrated encoding strength
235273

daemon/agent_mem_daemon/__main__.py

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -275,9 +275,43 @@ def _prewarm_indexes(knowledge_root: Path, log: logging.Logger) -> None:
275275
from embeddings import load_or_build as emb_load # noqa: PLC0415
276276

277277
idx = emb_load(knowledge_root)
278+
# Dummy search to JIT-compile the MPS kernels for the query path.
279+
# ``load_or_build`` only loads weights and computes doc vectors — it
280+
# never runs the query-side forward pass, so the first real priming
281+
# request would pay a ~2-3s graph-compile cost without this.
282+
idx.search("warmup query for kernel compile", k=1)
278283
log.info("startup: embedding index ready (%d docs)", len(idx.docs))
279284
except Exception:
280285
log.exception("startup: embedding prewarm failed (lazy rebuild on first use)")
286+
try:
287+
# Full-pipeline warmup. Just loading model weights isn't enough —
288+
# MPS kernel caches are keyed on tensor shape, so a dummy predict
289+
# on tiny inputs JIT-compiles kernels for the wrong shape and the
290+
# first real request pays a recompile anyway. Running the actual
291+
# priming pipeline against a realistic prompt warms every kernel
292+
# the hook path will use: embedding query encode, cross-encoder
293+
# predict on ~10 pairs of full-body documents, the works.
294+
import time # noqa: PLC0415
295+
296+
from .priming import _hybrid_search as priming_hybrid_search # noqa: PLC0415
297+
298+
t0 = time.monotonic()
299+
priming_hybrid_search(
300+
knowledge_root,
301+
"warmup priming query for kernel-shape JIT cache",
302+
k=5,
303+
)
304+
log.info(
305+
"startup: priming pipeline warm (%dms full warmup, BM25+embedding+rerank)",
306+
int((time.monotonic() - t0) * 1000),
307+
)
308+
except Exception:
309+
# The reranker is essential for Tier-1 precision but the daemon
310+
# should still run without it — first priming request will surface
311+
# the load failure loudly via the propagating exception path.
312+
log.exception(
313+
"startup: priming-pipeline prewarm failed (will fail loudly on first request)"
314+
)
281315

282316

283317
def _install_signal_handlers(stop_event: threading.Event) -> None:

daemon/agent_mem_daemon/logging_setup.py

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -90,4 +90,29 @@ def configure(
9090
sh.setFormatter(fmt)
9191
root.addHandler(sh)
9292

93+
# Silence noisy third-party loggers regardless of root level. Without
94+
# this, running with ``-v`` (root = DEBUG) buries our own messages
95+
# under HTTP wire traffic — every HuggingFace request from
96+
# sentence-transformers emits dozens of DEBUG lines (full HTTP
97+
# headers, connection lifecycle) per call. Pinning these at WARNING
98+
# keeps the daemon log readable. Includes anything that's been seen
99+
# spamming the log during model load / warmup.
100+
_NOISY_LOGGERS = (
101+
"httpcore",
102+
"httpcore.http11",
103+
"httpcore.connection",
104+
"httpx",
105+
"urllib3",
106+
"huggingface_hub",
107+
"sentence_transformers",
108+
"sentence_transformers.SentenceTransformer",
109+
"sentence_transformers.util",
110+
"sentence_transformers.base.model",
111+
"transformers",
112+
"filelock",
113+
"markdown_it",
114+
)
115+
for name in _NOISY_LOGGERS:
116+
logging.getLogger(name).setLevel(logging.WARNING)
117+
93118
return logging.getLogger("agent_mem_daemon")

daemon/agent_mem_daemon/priming.py

Lines changed: 93 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,7 @@
5252
import yaml
5353
from bm25 import load_or_build as bm25_load_or_build
5454
from embeddings import load_or_build as embeddings_load_or_build
55+
from reranker import rerank as _cross_encoder_rerank
5556

5657
from .paths import home as _agent_mem_home
5758

@@ -317,12 +318,17 @@ def _embedding_search(
317318
log.exception("priming: embedding search raised")
318319
return []
319320

320-
# Filter out very weak semantic matches — cosine below ~0.25 is
321-
# noise from this model and would poison RRF with irrelevant
322-
# paths. Empirical: gibberish queries against unrelated entries
323-
# produce scores in the 0.02-0.05 range; genuine paraphrase
324-
# matches sit at 0.4+.
325-
EMBEDDING_NOISE_FLOOR = 0.25
321+
# Filter out very weak semantic matches before they reach RRF.
322+
# Calibrated for nomic-embed-text-v1.5 — its cosine distribution
323+
# sits noticeably higher than older models (unrelated ~0.40-0.50,
324+
# genuine paraphrase matches ~0.65+). A floor at 0.50 separates the
325+
# populations: nonsense queries against unrelated entries fall under
326+
# it (so gibberish in == gibberish-free out, the "no real match
327+
# clears stale priming" semantic), while real paraphrase matches
328+
# comfortably clear it. Re-tune if the embedder swaps again — see
329+
# test_search_returns_empty_for_unrelated_query in tools/search for
330+
# the empirical handle.
331+
EMBEDDING_NOISE_FLOOR = 0.50
326332
return [(Path(h.path), float(h.score)) for h in hits if float(h.score) >= EMBEDDING_NOISE_FLOOR]
327333

328334

@@ -353,31 +359,100 @@ def _rrf_merge(
353359
return ordered[:k_top]
354360

355361

362+
# Minimum cross-encoder logit for a candidate to count as a real match.
363+
# ms-marco-MiniLM-L-12-v2 outputs unnormalized relevance logits — strong
364+
# matches sit at +5 to +10, weak-but-real around -2 to +2, definite
365+
# non-matches at -5 and below. The "no real match" filter is mostly
366+
# handled upstream by ``EMBEDDING_NOISE_FLOOR``; this floor is the
367+
# rerank's own guard against an outright non-match slipping through
368+
# the embedding lane on lexical-only signal. A permissive value lets
369+
# weak-but-real matches through (a tightly-scoped lesson on a less
370+
# common topic can rerank around 0). Tighten toward 0 if noise leaks
371+
# through; loosen toward -5 if valid matches get dropped.
372+
_RERANK_SCORE_FLOOR = -3.0
373+
374+
375+
def _rerank_candidates(
376+
query: str,
377+
candidates: List[Tuple[Path, float]],
378+
*,
379+
k: int,
380+
) -> List[Tuple[Path, float]]:
381+
"""Cross-encoder rerank pass on ``candidates``; degrade to input on failure.
382+
383+
Reads each candidate's body once, hands the ``(query, body)`` pairs to
384+
the cross-encoder, and returns the top-``k`` by relevance score with
385+
the cross-encoder score in place of the RRF score. Candidates scoring
386+
below ``_RERANK_SCORE_FLOOR`` are dropped — see the constant's docstring
387+
for why. The reranker returns ``None`` on any internal failure (model
388+
missing, predict raised, empty input) — in that case we hand back the
389+
upstream order truncated to ``k`` so retrieval still works without
390+
rerank.
391+
"""
392+
if not candidates:
393+
return []
394+
395+
bodies: List[Tuple[Path, str]] = []
396+
for path, _ in candidates:
397+
try:
398+
bodies.append((path, path.read_text(encoding="utf-8")))
399+
except (OSError, UnicodeDecodeError):
400+
# Skip unreadable candidates rather than fail the whole rerank.
401+
continue
402+
if not bodies:
403+
return candidates[:k]
404+
405+
reranked = _cross_encoder_rerank(query, bodies)
406+
if reranked is None:
407+
return candidates[:k]
408+
above_floor = [(p, s) for p, s in reranked if s >= _RERANK_SCORE_FLOOR]
409+
return above_floor[:k]
410+
411+
356412
def _hybrid_search(
357413
knowledge_dir: Path,
358414
query: str,
359415
*,
360416
k: int,
361417
) -> List[Tuple[Path, float]]:
362-
"""Run BM25 and embedding search in parallel, merge via RRF.
363-
364-
Each lane pulls a few more than ``k`` so RRF has overlap to work
365-
with. If the embedding lane is unavailable (no model installed,
366-
fresh corpus, transient failure), gracefully degrades to BM25-only.
418+
"""Run BM25 + embedding search, fuse with RRF, then cross-encoder rerank.
419+
420+
Three-stage retrieval, low precision → high precision:
421+
422+
1. **Recall**: BM25 and embedding lanes each pull ``k * 4`` candidates.
423+
BM25 catches exact-token matches; embeddings catch paraphrase.
424+
2. **Fusion**: RRF merges the two ranked lists into ``k * 4``
425+
topically-relevant candidates (rank-based, robust to score scale
426+
differences between BM25 and cosine).
427+
3. **Rerank**: a cross-encoder co-attends over each ``(query, body)``
428+
pair and scores actual applicability — orthogonal signal to the
429+
overlap-based lanes upstream. Returns the top ``k`` reranked.
430+
431+
Graceful degradation: if the embedding lane is unavailable, we
432+
fall back to BM25-only feed into the reranker. If the reranker is
433+
unavailable, we fall back to the RRF order verbatim. The hot path
434+
must keep working when any single component breaks.
367435
"""
368-
# Pull 2× k from each lane so RRF has a chance to surface entries
369-
# that one lane ranks low but the other ranks high.
436+
# Pull 2× k from each lane. Wider over-fetch (e.g. 4×) would give
437+
# the reranker more variety, but cross-encoder predict on MPS scales
438+
# nonlinearly — measured ~60ms @ 5 candidates, ~170ms @ 10, ~600ms @
439+
# 20 (graph-cache effects above the small-tensor threshold). At 4×
440+
# the rerank stage alone overruns the 200ms hook budget. Stick to 2×
441+
# so end-to-end fits.
370442
per_lane_k = max(k * 2, k + 5)
371443
bm25_hits = _bm25_search(knowledge_dir, query, k=per_lane_k)
372444
emb_hits = _embedding_search(knowledge_dir, query, k=per_lane_k)
373445
if not bm25_hits and not emb_hits:
374446
return []
447+
375448
if not emb_hits:
376-
# Embedding lane unavailable — fall back to BM25 verbatim.
377-
return bm25_hits[:k]
378-
if not bm25_hits:
379-
return emb_hits[:k]
380-
return _rrf_merge([bm25_hits, emb_hits], k_top=k)
449+
fused = bm25_hits
450+
elif not bm25_hits:
451+
fused = emb_hits
452+
else:
453+
fused = _rrf_merge([bm25_hits, emb_hits], k_top=per_lane_k)
454+
455+
return _rerank_candidates(query, fused, k=k)
381456

382457

383458
# Scope bonuses are calibrated against the RRF top-rank score (~0.016 for

daemon/agent_mem_daemon/priming_rpc.py

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -82,10 +82,12 @@
8282
# ── Tunables ─────────────────────────────────────────────────────────
8383

8484

85-
# Hard cap on a single request handler's wall time. The daemon will
86-
# truncate / error rather than letting a runaway index rebuild block
87-
# the hook beyond its 200 ms total budget.
88-
SERVER_REQUEST_TIMEOUT_S = 1.0
85+
# Hard cap on a single request handler's wall time. Sits below the hook
86+
# client's 2s total budget so a slow handler still surfaces as a server
87+
# error rather than a client-side socket timeout (cleaner failure mode).
88+
# The cross-encoder rerank stage runs ~300ms warm on Apple Silicon; CPU
89+
# or older GPUs can push that to ~1s, hence the headroom here.
90+
SERVER_REQUEST_TIMEOUT_S = 1.8
8991

9092
# Length-prefix size (big-endian uint32). 4 GB ceiling per message — we
9193
# expect bodies in the kilobyte range; the limit exists to refuse

0 commit comments

Comments
 (0)