Stress-test latent memory claims against dirty corpus, vague queries, and strong RAG baseline

[Flagrare]

The current experiments make TardigradeDB look like a serious research prototype, especially after invalidating mean-pool injection and showing full KV / hidden-state paths can work.

The next useful validation should be a harsher benchmark:

• 1k+ memories, not 10/30/100 clean synthetic cases
• vague queries, not mostly lexical/specific recall
• contradictory memories
• stale memories with decay / recency pressure
• irrelevant near-neighbor memories
• strong RAG baseline with reranker, not weak baseline only
• modern model path with RoPE/GQA, beyond GPT-2/Qwen small
• report CSV/raw outputs/seeds, not only narrative summaries
• compare latency, storage, prompt tokens, prefill cost, and recall

Main question to resolve:

Is TardigradeDB beating text RAG as memory, or is it mainly trading disk/storage for context/prefill efficiency through persisted KV/hidden-state payloads?

Both are valuable, but they are different claims.

Suggested benchmark matrix:

1. Text RAG + reranker
2. Hidden states + Top5Avg
3. Full KV injection
4. Q4 quantized KV injection
5. Hybrid retrieval index + KV payload
6. Negative controls with irrelevant and contradictory memories

Success criteria should include:

• Recall@1 / Recall@5
• false positive rate
• “I don’t know” calibration
• stale-memory suppression
• cost per retrieved useful memory
• latency and storage per 1k memories

This would separate “promising research infrastructure” from “nice controlled demo”.

[Flagrare]

Context update (April 24, 2026):

Several items from this issue have been partially or fully addressed since it was filed:

Done:

• Vague queries tested — 87% latent vs 100% RAG (15 vague queries across 10 domains)
• Strong RAG baseline — e5-small-v2 at 100% recall, compared directly
• Modern model with RoPE/GQA — Qwen3-0.6B is the primary test model
• Compare latency, storage, prompt tokens — 97ms latency, 730KB/memory, 46% prompt token savings documented

Still open — the hard parts:

• 1k+ memories (current max tested: 100)
• Contradictory memories (memory A says X, later memory B says not-X)
• Stale memories with decay/recency pressure (AKL exists but untested under retrieval load)
• Irrelevant near-neighbor memories (adversarial distractors)
• RAG baseline with reranker (current baseline is embedding-only, no cross-encoder)
• CSV/raw outputs/seeds for reproducibility
• False positive calibration / "I don't know" threshold
• Cost per retrieved useful memory at scale

Current state for reference:

• Retrieval: hidden states + Top5Avg = 100% recall@5 at 100 memories (engine pipeline, Q4)
• Injection: byte-identical to text RAG (8/10 novel facts, same misses both paths)
• Pipeline: Q4 round-trip cosine similarity = 0.999

The core question from this issue remains the most important open question:

Is TardigradeDB beating text RAG as memory, or is it mainly trading disk/storage for context/prefill efficiency through persisted KV/hidden-state payloads?

Current honest answer: it matches RAG on retrieval quality and saves 46% prompt tokens through injection. It does not yet beat RAG on retrieval. The value proposition is zero-token memory delivery, not better search.

[Flagrare]

Retraction + scope update (2026-05-20).

Retraction. The earlier triage comment cited "87% latent vs 100% RAG (15 vague queries)" and "100% recall@5 at 100 memories" as evidence that latent retrieval matches text RAG. Those numbers came from the corpus corrupted by the dataset-prep bug that the 2026-05-14 bench audit retracted (see docs/experiments/2026-05-14-bench-audit.md). Treat them as void.

The honest native-engine number on the clean dataset is ~36% R@1 at 50-item scale; the full-corpus number has not yet been re-measured. All four RLS modes (keyword / multiphrasing / embedding / generative / agent) underperformed the no-RLS baseline on clean data; the DeepSeek agent reformulator lost 12.7pp. The "vague-query vocabulary overlap is the retrieval ceiling" framing drawn from the broken data is also retracted.

Synthetic-corpus measurements (100% recall at 5K memories, vague-query refinement on the 100-cell Sonia corpus, cross-model retrieval, KV injection on gibberish facts) are unaffected — those used controlled inputs, not LoCoMo.

Current status of the original checklist:

Genuinely addressed:

• Scale beyond 100 memories. 5K-cell synthetic scaling validated at 100% recall (Top5Avg, Q4) — but synthetic, not adversarial.
• RAG-with-reranker baseline is unblocked. Cross-encoder shipped at python/tardigrade_hooks/reranker.py::CrossEncoderReranker (default cross-encoder/ms-marco-MiniLM-L-6-v2). Centered + CE rerank lifts vague R@5 to 64% and moderate R@5 to 68% on the 100-cell Sonia corpus. The head-to-head on clean LoCoMo with a strong text-RAG-plus-reranker baseline has not been run.
• Modern model with RoPE/GQA. Qwen3-0.6B is the primary path; cross-model (Qwen3 → GPT-2) demoed via MLP adapter.

Still open (the hard parts):

• Contradictory memories — memory A says X, later memory B says ¬X
• Stale memories under retrieval pressure — AKL decay exists and is now auto-driven by MaintenanceWorker ([Medium][Rescope] Governance Sweep Scheduling — automate AKL decay (engine vs adapter) #12), but interaction with retrieval ranking is untested
• Adversarial near-neighbor distractors
• False-positive rate / "I don't know" calibration / answerability threshold
• Cost-per-useful-retrieval at scale (latency × recall × storage)
• CSV / raw outputs / seeds for reproducibility on every benchmark
• Full clean-LoCoMo head-to-head: native engine vs text-RAG-plus-reranker

Reframed central question. Per the 2026-05-16 positioning pivot (docs/positioning/latency_first.md), beating text-RAG on LoCoMo retrieval is no longer the primary product target — the engine pays no LLM call at ingest, the comparators do, and "who scores higher on a text-extraction benchmark" is the wrong axis for the architectures involved. The product axes are: latency (sub-ms p99 at 5K cells), footprint (~751 B/cell), determinism, and the KV-native API. This issue should refocus on what stresses the foundation as a primitive: the contradictory / stale / distractor / calibration matrix above, with raw artefacts checked in. LoCoMo Judge work continues as architecture (Track B in CLAUDE.md), not as the scoreboard this issue is graded on.

Recommend keeping this issue open as the foundation-stress matrix, decoupled from the LoCoMo-leaderboard framing.