feat(decay): slice 1 — surfacing-aware memory decay#7
Merged
Conversation
…c sweep
Implements the first slice of the LTD design from
``[[projects/agent-mem/concepts/forgetting-ltd-decay-design]]``.
Deterministic, no-LLM bookkeeping that runs without curator cost.
**New module: ``decay.py``**
- ``stamp_last_surfaced(entry_path)`` — mirror of the Scholar's
``_bump_reinforced_counter``: parse frontmatter, set
``last_surfaced: <today>``, atomic temp+rename write. Idempotent
within a day (skips the write if the field is already current).
- ``run_sweep(knowledge_dir, now)`` — walks the library, archives
entries meeting all three conditions:
1. ``now - last_activity > 30d`` where
``last_activity = max(created, updated, last_reinforced,
last_surfaced)`` — the surfacing-aware activity signal,
using the per-session sent-cache to know when the agent has
seen an entry.
2. ``reinforced < 2`` — entries reasserted by the user survive.
3. ``not arousal_pinned`` — Scholar-stamped "identity-critical"
guard (field reserved; Scholar wiring lands in slice 2).
Archived = moved to ``_archive/<orig-path>``, ``status: stale``,
``archived: <today>`` stamped in the moved entry's frontmatter.
- ``maybe_run_sweep(knowledge_dir)`` — opportunistic trigger.
Self-skips inside the 24h cooldown or when another sweep holds
the module-level lock. Returns ``None`` when skipped,
``SweepResult`` when run.
**Trigger wiring**
- ``priming_rpc._handle_priming``: after a successful render, stamps
``last_surfaced`` on each entry in ``newly_sent`` and calls
``maybe_run_sweep`` synchronously. Bookkeeping extracted into
``_post_render_bookkeeping`` to keep ``_handle_priming`` under
the complexity ceiling.
- ``scholar._maybe_run_decay_sweep_safe``: called from both the
main-path and empty-packet branches of ``review()`` alongside the
existing priming refresh and invariants check. Failures swallowed
+ logged so decay can never break the curation pipeline.
**Failure semantics**
Every public entry point is best-effort: a bad entry skips that
entry but doesn't abort the sweep; a missing knowledge dir produces
an empty result and writes the sweep-state anyway (so the 24h
cooldown still ticks); the daemon never crashes on a decay failure.
**Tests**: 26 new tests (490 total daemon, 174 search, 68 hooks).
Covers frontmatter mutation, sweep-state I/O roundtrip + corruption
fallback, eligibility predicate corners, end-to-end archival with
each guard exercised in isolation, and cooldown-trigger mechanics.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…age gate Slice 1 added ~250 lines of new code (``decay.py``) plus the ``_post_render_bookkeeping`` helper in ``priming_rpc.py``. Adds 24 focused tests covering bookkeeping wiring and the deeper defensive paths so the 90% gate holds: missing knowledge dir, parse-iso-date invariants, reinforced-count input validation, both arousal-pin field-name variants, source-unlink failure during archive, destination resolution failure, mkdir failure, sweep continuing after a bad entry, lock-held early-bail in maybe_run_sweep, sweep-state file with wrong field type / non-dict / unparseable ISO date, atomic-write swallowing OSError, top-level admin files (``index.md``/``log.md``) excluded from sweep. Coverage: 89.04% (pre-slice-1) → 90.05%. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Forgetting section: rewrote the intro to acknowledge the slice 1 sweep is in place (was "Ultan models LTP but not LTD"), so the remaining work is correctly framed as "refinements" rather than the whole mechanism. - Mechanisms table: LTD row TODO → Partial (with explicit pointer at the half-life-from-encoding-strength formula being the part still outstanding). Decayed-not-deleted row Partial → Done, with the resurrection-on-contradiction pathway called out as the still-open follow-up. - Status: 225 → 756 (514 daemon + 174 search + 68 hooks). The 225 number was from before the recall + dedup + decay work landed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Editorial cleanup of issues flagged in review: - Reframed the framing: the library no longer "accumulates indefinitely" (PR #7's decay handles unused noise); reflection's actual contribution is consolidating *used* patterns that would otherwise stay as parallel leaves. Sharper claim, also still true. - Softened "single biggest delta from a mammalian system" to "one of the largest" — encoding_strength is arguably a comparable delta and "single biggest" overclaims. - Added the design fork (thematic summary vs schema inference) as an explicit decision to make before implementation. The current sketch describes thematic clustering while citing biology that's about transitive inference — distinct mechanisms, both useful, worth picking deliberately. - Cross-referenced A-MEM (already cited in the Reconsolidation row) as related prior art that does hierarchical evolution but doesn't propose the net-new-parent step. Expanded the open-questions list from three items to seven: - Folder placement when children span folders (new). - Composition with decay: what happens to a parent when its children archive? (new — addresses a real failure mode). - Recursive reflection: reflections-of-reflections, or stay shallow? (new). - Parent's encoding_strength derivation: max-of-children is the current sketch; flagged as under-specified with two alternatives. - Original three questions kept (cadence, minimum cluster size, contradicts-against-parent). No design decisions made — the section now surfaces the choices without prejudging them. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous framing proposed a periodic offline Scholar job doing
embedding-cosine clustering + per-cluster summarisation. User
pushback (and the existing
[[no-regex-seed-extraction-use-parallel-search]] convention) makes
clear the right shape is agent-driven, not algorithmic:
- The Librarian already does multi-entry-reading-with-judgment for
``merge_entries``, ``split_folder``, ``update_readme``,
``add_wikilink``. Adding "synthesise an abstract parent" is one
more verb in that vocabulary, not a separate periodic daemon.
- Cosine clustering would force a topical-summary shape (Park et
al. 2023's flavour); agent judgment gives the schema-inference
shape the cited Eichenbaum/Schlichting biology actually describes
("rule A about python + rule B about Go → 'always pin
dependency versions'", not "5 entries about python deps → one
themed summary").
Rewrites:
- Section prose: new framing introduces ``abstract_entries`` /
``propose_parent_entry`` as the proposed Librarian action;
contrasts cosine-summary vs schema-inference flavours and picks
the latter; cross-references the no-regex-seed-extraction
convention as the matching architectural pattern.
- Open questions: "cadence" reframed as "what triggers the
Librarian to look" (it's not a separate periodic job anymore).
"Minimum cluster size" reframed as "restraint heuristics for the
Librarian's prompt". Reinforcement cascade promoted to its own
question. Folder placement, decay composition, recursive
reflection, and contradicts-vs-parent kept.
- Mechanisms table row: "Periodic Scholar pass clusters
semantically-near entries..." → "New Librarian action that
proposes a parent abstraction over related leaves during its
normal scan — agent judgment, not cosine clustering."
Note: ``[[projects/agent-mem/concepts/neuro-gaps/reflective-abstraction-offline-integration-gap]]``
still carries the old framing and will need an ``update_entry``
proposal from the Librarian — left for the curator pipeline to
pick up from this conversation buffer rather than editing
directly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…pdates them Footer addition surfaces on every priming render: *Entries are living — the curator updates them on new info. You may cite the current text; no need to maintain them yourself.* Addresses a real gap the agent could hit: an entry it consulted last turn may have a slightly different framing this turn because the Librarian/Scholar updated it from the conversation buffer. Without this note, the agent might (a) think it mis-cited, (b) distrust the entry, or (c) try to edit memory directly to "fix" the drift — the last of which would violate the existing ``no-manual-memory-writes-by-agents`` convention. The new line preempts (c) and gently licenses (a) — past versions were correct at their time; cite the current text. Wording is permissive on user's correction (``You may cite`` rather than ``Cite``) — the previous header's ``cite or follow when applicable`` already opted for that tone. Same string lands in both renderers (daemon ``priming.py::_FOOTER`` and the hook-side fallback ``_priming_client.py::_FOOTER``) so the agent sees the same framing regardless of which path served the request. Tests: - ``test_refresh_respects_char_budget`` and ``test_priming_char_budget_trims_bullets`` use deliberately-tight budgets (~400-500 chars) to exercise trim behaviour. The new footer added ~140 chars to the unconditional framing, so both budgets bumped (500→700, 400→600) to leave room for the trim test's seeded content. Production budget (1500) is unaffected. - ``test_send_request_returns_none_when_socket_missing`` had a pre-existing pyright complaint (partial ``PrimingRequest`` dict after ``session_id`` was added to the TypedDict in PR #6); fixed properly by constructing a fully-typed request rather than suppressing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two changes to the README intro: - Inline image after the "It's your library — ls/cat/git it" line. 600px display, retina-scaled via 1200px source, ~1.5 MB after downsampling from the 9.5 MB / 2816×1536 original. Visual metaphor maps to the architecture: monk-as-Librarian (reading, classifying salience), scholar-at-desk-as-Scholar (deliberating, committing), and the brain-and-circuit-board hologram standing in for the bio-inspired memory model the system implements. - New "Why this exists" section before "Design, in one paragraph". Frames the gap that motivated the build (session amnesia + the existing memory features being shallow because they're cheap) and the central design choice (stop optimising for token cost first; take the neurology seriously). Bridges to the design paragraph that follows. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…hallow" line - Motivation section moved from after the divider to immediately after the Master Ultan quote, ahead of the "Ultan watches your conversations…" what-it-does paragraph. WHY now leads WHAT, matching the natural reader's question order. - Dropped the "Most current memory systems are shallow because they're cheap — last-N turns, keyword match, a fixed prompt suffix. None of them looks much like how memory actually works." sentence per user feedback. The token-cost framing in the surrounding text already carries that idea without naming specific competitor patterns. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Memory is plain markdown on disk and often git-tracked, so any
credential that ends up in a proposal's body / applies-when /
keywords / reasoning quote is a real leak. Defence in depth at
both curator stages:
- librarian_prompt.py (ground rule 7): the Librarian must not
quote API keys, auth tokens, bearer tokens, OAuth client secrets,
passwords, private-key blocks (BEGIN ... PRIVATE KEY),
connection strings with embedded passwords, GitHub PATs (ghp_*,
github_pat_*), AWS access keys (AKIA*), Anthropic/OpenAI keys
(sk-*), JWTs, session cookies, or similar.
- scholar_prompt.py (hierarchy invariant 6): VETO any proposal
whose body, applies-when, keywords, frontmatter, or reasoning
quote contains a literal secret matching the same patterns.
Fixed VETO reason ("contains-secret — would write credentials
to plain-markdown library") so post-mortems are greppable.
Tests pin the patterns. README Design-discipline list gains a
bullet; Status test count 514→516 / 756→758.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…orage location Pre-public-release hygiene pass. - ``daemon/README.md``: removed three hardcoded ``/Users/nicholasholden/`` paths (lines 4, 14, 158). The PLAN.md reference was dead anyway (file doesn't exist) and the install/run + dogfood snippets now use ``<repo>/daemon`` as the placeholder. - ``daemon/tests/test_llm_path_guard.py``: the path-guard regression test built a fake-real path using the developer's home directory for narrative reasons; swapped for ``/Users/example/...`` since the actual assertion doesn't care which absolute path it's given as long as it's outside the test's tmp_path boundary. - ``.gitignore``: added ``.bm25.idx`` and ``.embeddings.idx`` — derived search artefacts that are rebuilt on demand from the markdown corpus they index. Untracked ``tools/search/fixtures/.bm25.idx`` (was 12.9 KB of binary); the BM25 tests regenerate it in-tmpdir, verified 24/24 still pass. - ``README.md``: new ``### Where your memories live`` subsection inside Quick start. Spells out the ``~/.agent-mem/`` layout (knowledge/ vs events.jsonl vs daemon.log vs the derived indexes vs sweep-state.json vs pending-nudges.md vs runs/) so anyone trying Ultan for the first time knows exactly where the data lands. Cross-references the deeper "Storage on disk" table later in the README. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
First slice of the LTD design from
forgetting-ltd-decay-design. Deterministic, no-LLM bookkeeping that runs without curator cost.last_surfacedfrontmatter field stamped by the daemon every time an entry shows up in priming output. Uses the per-session sent-cache as the trigger so we know when the agent has actually seen an entry. Idempotent within a day.last_activity = max(created, updated, last_reinforced, last_surfaced). Entries withnow - last_activity > 30dANDreinforced < 2AND notarousal_pinned: trueget archived (moved to_archive/<orig-path>withstatus: stale+archived: <today>stamped on the moved file).decay.maybe_run_sweep()is called frompriming_rpc._handle_primingand fromscholar.review()(both main + empty-packet paths). It self-skips unless the 24h cooldown has elapsed and no other sweep is in flight. Sweep state lives at~/.agent-mem/sweep-state.json.Failure semantics
Every public entry point is best-effort: a bad entry skips that entry but doesn't abort the sweep; a missing knowledge dir produces an empty result and still updates the cooldown; the daemon never crashes on a decay failure. Stamping
last_surfacedfailures get logged and swallowed in the priming hot path.Test plan
uv run pytest --cov— 514 passing, coverage 90.05% (up from 89.04%)decaymodule tests — 45 cases covering frontmatter mutation, sweep-state I/O roundtrip + corruption fallback, eligibility predicate corners, end-to-end archival with each guard exercised in isolation, cooldown-trigger mechanics, and the deeper defensive paths (source-unlink failure, mkdir failure, destination-resolution failure, sweep continuing after a bad entry, lock-held early-bail)priming_rpcbookkeeping tests —_post_render_bookkeepingcalled for each newly_sent entry, no-session-id branch, swallowed stamp/sweep failuresWhat's not in this slice (intentionally deferred)
encoding_strengthas a write-time field set by Scholar — slice 2.encoding_strength— out of scope per the design's decision to keep retrieval signal coming from the cross-encoder, not from a self-reinforcing prior._archive/on new contradiction — slice 3.🤖 Generated with Claude Code