Skip to content

REM: recursive consolidation into a maintained summary hierarchy (zoom-out recall) #716

Description

@tps-flint

Context

With #707 shipped (v0.22.0), REM executes distillation in-process — but consolidation is still flat: one pass over recent memories → staged candidates. Flat top-k retrieval and flat distillation share a structural blind spot: neither can answer zoom-out questions ("what does this corpus broadly know about X").

Direction: recursive consolidation into a maintained summary hierarchy

RAPTOR-style: cluster related memories (every Memory already carries an HNSW-indexed embedding) → distill each cluster into a parent memory (derivedFrom = members, parentId chain, higher durability per level) → recurse. HarperFast/harper#1244 measured this shape for document retrieval: ~2.4× thematic topic coverage at equal token budget, ~68% win rate on decisive thematic questions with no factoid regression (reference component: HarperFast/summary-tree). REM's slice-2 machinery (models.generate() path, schema-constrained output, all-or-nothing staging, staged-never-auto-promoted) is exactly the engine this rides on.

What makes it a real slice, not a weekend port

  1. Living corpus. RAPTOR assumes a static document set. A memory corpus is append-heavy with supersession — staleness must propagate up the tree incrementally (a superseded member invalidates its ancestors), with no nightly full rebuilds.
  2. Eval gate before any tree code. Thematic-coverage + blind-judge QA eval ported to a real agent-memory corpus, plus factoid non-regression on the existing recall benchmarks. Same discipline as Activate recall levers in production: enable BM25 hybrid retrieval + requant embeddings Q4 to Q8 #623/Embeddings: consider Harper's native model.embed instead of hand-rolled harper-fabric-embeddings + llama.cpp sandbox workarounds #504: no merge without the instrument.
  3. Trust/provenance: parent memories are machine-derived — model, timestamp, member set stamped; zoom-in from summary to sources must stay verifiable.

The ask — @kriszyp

You offered the #1244 harness/data — formally taking you up on it here:

  1. The eval harness + question-set generation from #1244 (the thematic-coverage measurement and the blind-judge end-to-end QA setup), so we can port the gate to a Flair memory corpus before designing the tree build.
  2. Anything #1244 learned about incremental maintenance — did you explore re-clustering under corpus growth/mutation, or was the tree built once over a static corpus? This is our main engineering delta and any negative results would save us real time.

Design doc follows once the gate instrument exists.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions