feat(memory): claude-mem feature parity — series of 5 PRs - #2
Closed
fazleelahhee wants to merge 2 commits into
Closed
feat(memory): claude-mem feature parity — series of 5 PRs#2fazleelahhee wants to merge 2 commits into
fazleelahhee wants to merge 2 commits into
Conversation
Captures the 6 design decisions from brainstorming: auto+explicit capture with source column, background compression worker in cce serve, per-turn + per-session rollup granularity, extended session_recall + new session_timeline/session_event MCP tools, one-shot cce sessions migrate command, three dashboard panels. Compressor is extractive using BAAI/bge-small-en-v1.5 already loaded for the index — no new dependencies, no extra RAM, no Ollama required. Ships as 5 sequential PRs off this branch; each independently reviewable.
PR 1 of 5 against feature/memory-claude-mem-parity. Lays the storage foundation: per-project SQLite at ~/.cce/projects/<name>/memory.db with the v1 schema (sessions, prompts, tool_events, tool_event_payloads, turn_summaries, decisions, code_areas, pending_compressions, migrated_files, schema_versions) plus FTS5 virtual tables and triggers for prompts / decisions / turn_summaries. Adds `cce sessions migrate` — idempotent one-shot importer for legacy per-session JSON files (current path and pre-rebrand ~/.claude-context-engine/...). Imported decisions and code areas are tagged source='migrated' so future session_recall can rank them. Consumed JSONs are archived into sessions/migrated.zip and removed. No behaviour change to the existing JSON capture path. Hooks and the compression worker land in PR 2 and PR 3. 10 new unit tests cover: schema bootstrap, idempotent reconnection, foreign-key enforcement, FTS triggers, migration of session JSON + decisions_log archive, idempotent rerun, archive-and-remove. Full suite: 311 passed, 1 skipped. See docs/specs/2026-04-28-memory-claude-mem-parity-design.md.
Contributor
There was a problem hiding this comment.
Pull request overview
Introduces the “Foundation” layer for memory ↔ claude-mem parity by adding a per-project SQLite memory.db (with FTS) plus an idempotent migrator and CLI command to import legacy JSON session artifacts.
Changes:
- Add
memory.dbbootstrap + schema versioning (incl. FTS tables/triggers). - Add JSON→SQLite migrator and wire it to
cce sessions migrate(with optional archiving). - Add unit tests covering schema bootstrap and migration behavior; add a design spec doc.
Reviewed changes
Copilot reviewed 7 out of 8 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
src/context_engine/memory/db.py |
Creates/opens per-project SQLite DB and bootstraps schema + FTS. |
src/context_engine/memory/migrate.py |
Implements idempotent importer from legacy session JSON + optional archiving. |
src/context_engine/cli.py |
Adds cce sessions migrate CLI subcommand. |
tests/memory/test_db.py |
Validates schema creation, FK pragma, FTS behavior, constraints. |
tests/memory/test_migrate.py |
Validates migrate import counts, idempotency, archiving, and decisions_log shape. |
src/context_engine/memory/__init__.py |
Package docstring describing new storage approach. |
tests/memory/__init__.py |
Marks tests.memory as a package. |
docs/specs/2026-04-28-memory-claude-mem-parity-design.md |
Design/spec documentation for the 5-PR series and schema/architecture. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+118
to
+127
| # decisions_log.json is a top-level list of decision dicts, not a session. | ||
| if source.name == _DECISIONS_LOG_NAME and isinstance(data, list): | ||
| for d in data: | ||
| _insert_decision( | ||
| conn, | ||
| session_id=None, | ||
| decision=d.get("decision", ""), | ||
| reason=d.get("reason", ""), | ||
| timestamp=d.get("timestamp"), | ||
| ) |
There was a problem hiding this comment.
In decisions_log.json import, each entry includes a session_id (see session_capture.prune_old_sessions), but the migrator always inserts session_id=None. This drops linkage that can be preserved when the referenced session already exists in the DB; consider using d.get('session_id') and only keeping it when _session_exists(conn, ...) is true (otherwise NULL to satisfy the FK).
Comment on lines
+161
to
+163
| def _insert_decision(conn, *, session_id, decision, reason, timestamp): | ||
| epoch = int(timestamp) if timestamp else int(time.time()) | ||
| iso = time.strftime("%Y-%m-%dT%H:%M:%S", time.gmtime(epoch)) |
There was a problem hiding this comment.
timestamp is treated as falsy, so a valid 0/0.0 (which the legacy pruner writes as the default when missing) will be replaced with time.time(). This makes migrated rows look “new” and can distort ordering/ranking. Consider checking timestamp is not None (and possibly allowing 0 → epoch 0) instead of a truthiness check.
Comment on lines
+135
to
+144
| session_id = data.get("id") | ||
| # We do not synthesise a sessions row from migrated data — the legacy | ||
| # JSON files predate the new sessions schema and miss timestamps in a | ||
| # form we can trust. Importing decisions with session_id=None is the | ||
| # safer choice; the FK is nullable on purpose. | ||
| for d in data.get("decisions", []) or []: | ||
| _insert_decision( | ||
| conn, | ||
| session_id=session_id if _session_exists(conn, session_id) else None, | ||
| decision=d.get("decision", ""), |
There was a problem hiding this comment.
_session_exists(conn, session_id) is executed repeatedly inside the decisions/code_areas loops even though session_id is constant per file. This results in extra DB queries per imported row; compute it once (or memoize by session_id) and reuse the result for both loops.
Comment on lines
+91
to
+97
| summary.files_imported += 1 | ||
| _mark_imported(conn, f) | ||
| consumed.append(f) | ||
| if consumed: | ||
| consumed_per_dir[sessions_dir] = consumed | ||
|
|
||
| conn.commit() |
There was a problem hiding this comment.
Archiving happens after conn.commit(), but files are recorded in migrated_files before archiving/removal. If _archive_and_remove raises (e.g., zip write error), the DB will be committed and the file will be permanently considered “already imported”, so reruns won’t be able to archive/delete it. Consider reordering so archive write failures don’t leave files stuck in an imported-but-unarchived state (e.g., only mark imported after successful archive, or add an archived flag and treat already-imported-but-present files as cleanup candidates).
Comment on lines
+145
to
+147
| Eight tables in `~/.cce/projects/<name>/memory.db`. Timestamps stored as | ||
| both ISO text and epoch int (text for humans, int for sorting). FTS5 | ||
| virtual tables shown last. |
There was a problem hiding this comment.
This section says “Eight tables in memory.db”, but the implemented schema in memory/db.py includes additional tables (e.g., tool_event_payloads, migrated_files, schema_versions) plus FTS virtual tables and triggers. Please align the table count and the SQL snippet with the actual v1 schema so the spec remains a reliable reference.
Suggested change
| Eight tables in `~/.cce/projects/<name>/memory.db`. Timestamps stored as | |
| both ISO text and epoch int (text for humans, int for sorting). FTS5 | |
| virtual tables shown last. | |
| The v1 schema in `~/.cce/projects/<name>/memory.db` includes the core | |
| memory tables plus support tables such as `tool_event_payloads`, | |
| `migrated_files`, and `schema_versions`, along with FTS5 virtual tables | |
| and their triggers. Timestamps are stored as both ISO text and epoch int | |
| (text for humans, int for sorting). |
2 tasks
fazleelahhee
added a commit
that referenced
this pull request
Apr 28, 2026
Consolidates the 11 actionable items from the 5-PR memory stack into one
fixup commit on the unified ai-memory branch.
migrate.py
- Archive *before* mark-imported + commit, with rollback on zip failure
so a failed archive no longer leaves files stuck imported-but-not-archived.
- Preserve session_id linkage from decisions_log.json when the referenced
session already exists (was unconditionally NULL).
- Memoise _session_exists per-file (constant per archive entry).
- Use `timestamp is not None` so legacy 0/0.0 timestamps keep their original
ordering instead of being stamped to "now".
compressor.py
- Drop unused `Any` import.
- Cap raw_input at _TOOL_INPUT_CHAR_CAP (4 KB) before json.loads so multi-MB
patch payloads don't stall the compression worker.
- Yield the asyncio loop after every drained item, with a short breath every
5 items, so a backlog doesn't monopolise mcp.run_stdio().
mcp_server.py
- Use FTS5 MATCH on decisions_fts / turn_summaries_fts (and a LIKE filter on
code_areas) to prefilter recall candidates instead of embedding the latest
600 rows on every session_recall call.
- Clamp `limit` in session_timeline to 1..200 with a helper that handles bad
input cleanly.
- Wrap the sessions metadata query and the session_event payload query in
try/except so DB errors return a specific message.
- Handle NULL raw_input/raw_output in session_event without rendering the
string "None".
- Update the dual-write comment to reflect that recall now goes through FTS5.
design spec
- Drop the inaccurate "Eight tables" sentence; describe the support and
FTS5 tables that ship in v1 instead.
Tests: 43/43 memory + dashboard pass; 344/345 across the full suite (the
single failure is test_ollama_client hitting httpx.ReadTimeout because no
Ollama is running in this environment — unrelated).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
5 tasks
Contributor
Author
fazleelahhee
added a commit
that referenced
this pull request
Apr 28, 2026
Adds full conversation memory to cce as a peer feature alongside code search:
per-project SQLite store, lifecycle capture hooks, hybrid FTS5 + sqlite-vec
recall, automatic SessionStart resume injection, and a recall benchmark.
## What's in
**Foundation** — schema v2 in `~/.cce/projects/<name>/memory.db` with sessions,
prompts, tool_events, turn_summaries, decisions, code_areas, plus FTS5 virtual
tables and sqlite-vec `vec0` tables for semantic search. v1→v2 migrates in place.
**Capture** — 5 lifecycle hooks (SessionStart, UserPromptSubmit, PostToolUse,
Stop, SessionEnd) post to a loopback aiohttp server inside `cce serve`. Hook
shell script is platform-aware (cce_hook.sh on POSIX, cce_hook.cmd on Windows)
and shell-quoted correctly for both shells. Capture fails closed.
**Compress** — extractive worker drains `pending_compressions` on a thread,
using bge-small (already loaded for the index — no LLM, no API tokens).
**Recall** — `session_recall(topic)` runs hybrid retrieval: FTS5 + sqlite-vec
with reciprocal rank fusion, content-keyed dedup, threshold-tuned against an
empirical bge-small noise floor. TL;DR header (top-3 by centroid similarity)
+ 20 tagged matches with relative-time hints and drill affordances
(`→ session_timeline("<sid>")` / `→ session_event(id=N)`).
**Resume** — SessionStart hook returns last session's rollup + recent
decisions as plain text. Claude Code captures hook stdout and injects it
into the model's context at session start. Matcher is `startup|clear|compact`
so the resume re-injects after `/clear` or `/compact` too. *This is what
solves "decisions you made last week have to be re-explained today."*
**Operational** — daemon-thread vec backfill (no startup stall), payload
retention via `cce sessions prune` + automatic background `auto_prune_loop`,
cleanup triggers for orphaned vec rows, FTS stop-word filter, init-time
reachability probe, `cce sessions status` CLI command, recall benchmark
script (`scripts/bench_recall.py`).
**Tooling** — `cce sessions migrate` imports legacy JSON; idempotent;
archives consumed files. CLAUDE.md template (block v3) documents the full
MCP tool surface including `session_timeline` and `session_event`.
## Bench
R@5=0.75, P@5=0.46, MRR=0.74 on the 14-query benchmark; off-topic queries
("how is the weather today") reject cleanly. Token cost per recall:
~600-900 tokens (5-7× cheaper than competing memory tools that LLM-curate).
## Tests
149 memory + dashboard + integration + CLI tests; 387 of full suite pass.
CI green on Python 3.11 / 3.12 / 3.13.
## Stats
39 files changed, +6,663 / -189 LOC over 28 commits. Supersedes the
original 5-PR stack (#2-#6, all closed).
fazleelahhee
added a commit
that referenced
this pull request
Apr 28, 2026
Wire-up audit on main surfaced 8 items. All addressed. #1 production: RRF dedup broken in dual-write window (live-confirmed duplicate in session_recall). Fixed by canonicalising _content_key through grammar.compress(lite). #2 ux: dashboard returned raw compressed bytes. /api/memory/* now applies grammar.expand on read. #3 cce sessions migrate now compresses on insert (homogeneous storage). #4-7 test gaps closed. #8 confirmed reserved-for-future column. Tests: 603/603 (+25 new). CI green on Python 3.11 / 3.12 / 3.13. Live-verified the dedup fix on /home/fazle/trading/v3.
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Series
This is the umbrella draft PR for the memory ↔ claude-mem parity feature. The branch ships in 5 sequential, independently reviewable commits/PRs.
memory/db.pyschema +cce sessions migrate+ 10 testscce servesession_recall+ newsession_timeline,session_eventMCP toolsSpec
Design + decisions:
docs/specs/2026-04-28-memory-claude-mem-parity-design.mdon this branch.Six brainstorming decisions locked:
source='manual'|'auto'|'migrated')cce serve(per-project MCP server)session_recall(topic)(back-compat); addsession_timeline(session_id)andsession_event(event_id)cce sessions migrate(idempotent, user-invoked)Compressor tiers: extractive default using
BAAI/bge-small-en-v1.5already loaded for the index. No Ollama required, no embedded LLM, no extra deps.What's in PR 1
src/context_engine/memory/db.py— schema bootstrap, version table, connection helper (PRAGMAforeign_keys=ON,journal_mode=WAL)src/context_engine/memory/migrate.py— idempotent JSON-to-SQLite importer (current path + legacy~/.claude-context-engine/...)cce sessions migrateCLI subcommand with--no-archivetests/memory/test_db.py(5 tests),tests/memory/test_migrate.py(5 tests)Test plan
pytest tests/memory/— 10 new tests passcce sessions migrate --helpregisters and routes correctlycce sessions migrateagainst the sonin install once PR is reviewed (real-data shakeout)