feat(memory) PR 3/5: compress — extractive worker in cce serve by fazleelahhee · Pull Request #4 · elara-labs/code-context-engine

fazleelahhee · 2026-04-27T23:36:18Z

PR 3 of 5 — Compress

Stacked on PR 2 (capture). Base: `feature/memory-pr2-capture`.

Implements the background compression worker that drains `pending_compressions` and writes turn summaries + session rollups.

What's added

`memory/extractive.py` — sentence splitter + centroid-based extractive summarisation. No new dependencies. Operates on any embedder exposing `embed_query(str)`. Always returns real source text — no synthesis, no hallucination.
`memory/compressor.py` — `compress_turn()` and `compress_session_rollup()` plus the `compression_loop()` async drain. Uses the bge-small embedder already loaded for the index. Falls back to truncation if the embedder is unavailable.
`_run_serve` (cli.py) — spawns the worker alongside the hook server, with its own sqlite connection (avoids cross-thread issues with sqlite3). Cancelled and closed gracefully on shutdown.

Tier behaviour

Tier	When	Quality
`extractive`	default — bge-small loaded	top-K centroid sentences
`truncation`	embedder None or extractive raises	first 200 chars + ellipsis
`empty`	no candidate text (e.g. SessionEnd before any turn drained)	""

Out of scope (lands later)

Recall: `session_recall` still reads from JSON. Lands in PR 4.
Dashboard panels: PR 5.

Test plan

`pytest tests/memory/` — 39 tests (14 new for PR 3)
Full suite: 340 passed, 1 skipped
Live shakeout post-merge: verify extractive summaries land in `turn_summaries` after a real Claude Code turn

PR 3 of 5 against feature/memory-claude-mem-parity, stacked on PR 2. Adds the background compression half: - src/context_engine/memory/extractive.py — sentence splitter + centroid-based extractive summariser. No new dependencies; takes any embedder exposing embed_query(str) -> Iterable[float]. Pure source text; no synthesis means no hallucination. - src/context_engine/memory/compressor.py — compress_turn() and compress_session_rollup() build the candidate text from prompts + tool_events (+payloads), run the extractive summariser using the bge-small embedder already loaded for the index, and persist to turn_summaries / sessions.rollup_summary with tier='extractive'. Falls back to truncation when embedder is None or extractive raises. compression_loop() drains pending_compressions every 5 s, oldest first, single-flight by design. Queue rows that error are kept with attempts++ and last_error stamped for retry. - _run_serve in cli.py spawns the compression worker alongside the hook server. Owns its own sqlite connection to avoid cross-thread use; gracefully cancelled and closed on MCP server exit. Tests (14 new, full suite 340 passed): - tests/memory/test_extractive.py: sentence split, centroid neighbour selection, source-order preservation, truncation fallback shape. - tests/memory/test_compressor.py: turn compression with extractive + truncation tiers, session rollup combining turn summaries, empty rollup when no turns, _drain_one queue pop, compression_loop with stop_event for graceful shutdown. Recall surface unchanged from main; PR 4 retires the JSON path.

Copilot

Pull request overview

Implements the “compress” stage of the memory pipeline by adding an extractive summariser and a background worker that drains pending_compressions, producing per-turn summaries and session rollups during cce serve.

Changes:

Added centroid-based extractive summarisation (memory/extractive.py) with truncation fallback.
Added a background compression worker (memory/compressor.py) to process queued compressions and persist summaries/rollups.
Wired the worker into cce serve lifecycle and added tests + dev dependency updates.

Reviewed changes

Copilot reviewed 5 out of 6 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
uv.lock	Adds `pytest-aiohttp` to dev dependencies / lock resolution.
tests/memory/test_extractive.py	New unit tests for sentence splitting, extractive selection, ordering, and truncation.
tests/memory/test_compressor.py	New tests covering turn compression, rollups, queue draining, and loop behavior.
src/context_engine/memory/extractive.py	New extractive summariser implementation (centroid + cosine ranking).
src/context_engine/memory/compressor.py	New compression worker + DB write paths for turn summaries and session rollups.
src/context_engine/cli.py	Starts/stops the compression loop alongside the hook server in `_run_serve`.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-27T23:43:48Z

+import logging
+import sqlite3
+import time
+from typing import Any


Any is imported but not used. Please remove the unused import to keep the module clean (and avoid lint failures if the repo enforces unused-import checks).

Suggested change

from typing import Any

Copilot · 2026-04-27T23:43:49Z

+def _describe_input(tool_name: str, raw_input: str) -> str:
+    """One-line descriptor of a tool invocation for the summary candidates."""
+    if not raw_input:
+        return tool_name
+    try:
+        data = json.loads(raw_input)
+    except (json.JSONDecodeError, ValueError):
+        return f"{tool_name}: {raw_input[:120]}"
+    if not isinstance(data, dict):
+        return f"{tool_name}: {raw_input[:120]}"
+    # Surface common high-signal fields explicitly.
+    for key in ("file_path", "command", "pattern", "path", "query"):
+        if key in data and data[key]:
+            return f"{tool_name} {key}={data[key]!r}"
+    keys = list(data.keys())[:2]
+    return f"{tool_name} {keys}"


_describe_input calls json.loads(raw_input) on the full tool-input payload. In hooks.py, raw_input is stored as json.dumps(tool_input) and can be very large (e.g., patch payloads), which makes this parse potentially expensive and can stall the compression worker. Consider adding an input size cap (similar to _TOOL_OUTPUT_CHAR_CAP) so large inputs skip JSON parsing and fall back to a truncated descriptor.

Copilot · 2026-04-27T23:43:49Z

+    """Run forever, draining the queue between sleeps. Cancellable via task.cancel()."""
+    while True:
+        if stop_event is not None and stop_event.is_set():
+            return
+        try:
+            did_work = await _drain_one(conn, embedder)
+            if not did_work:
+                await asyncio.sleep(interval_seconds)
+        except asyncio.CancelledError:
+            raise
+        except Exception:
+            log.exception("compression_loop iteration crashed; backing off")
+            await asyncio.sleep(interval_seconds)


compression_loop can become a tight loop when the queue is non-empty: when did_work is True there is no await in the iteration, and compress_turn() / compress_session_rollup() are fully synchronous (embedding inference + SQLite I/O). This can monopolize the event loop running mcp.run_stdio() and degrade serve responsiveness when there’s a backlog. Consider yielding/backing off even when work was done (e.g., await asyncio.sleep(0) per item, or drain up to N items then sleep briefly), or running the compression work in a dedicated thread/process with its own SQLite connection created in that worker context.

Consolidates the 11 actionable items from the 5-PR memory stack into one fixup commit on the unified ai-memory branch. migrate.py - Archive *before* mark-imported + commit, with rollback on zip failure so a failed archive no longer leaves files stuck imported-but-not-archived. - Preserve session_id linkage from decisions_log.json when the referenced session already exists (was unconditionally NULL). - Memoise _session_exists per-file (constant per archive entry). - Use `timestamp is not None` so legacy 0/0.0 timestamps keep their original ordering instead of being stamped to "now". compressor.py - Drop unused `Any` import. - Cap raw_input at _TOOL_INPUT_CHAR_CAP (4 KB) before json.loads so multi-MB patch payloads don't stall the compression worker. - Yield the asyncio loop after every drained item, with a short breath every 5 items, so a backlog doesn't monopolise mcp.run_stdio(). mcp_server.py - Use FTS5 MATCH on decisions_fts / turn_summaries_fts (and a LIKE filter on code_areas) to prefilter recall candidates instead of embedding the latest 600 rows on every session_recall call. - Clamp `limit` in session_timeline to 1..200 with a helper that handles bad input cleanly. - Wrap the sessions metadata query and the session_event payload query in try/except so DB errors return a specific message. - Handle NULL raw_input/raw_output in session_event without rendering the string "None". - Update the dual-write comment to reflect that recall now goes through FTS5. design spec - Drop the inaccurate "Eight tables" sentence; describe the support and FTS5 tables that ship in v1 instead. Tests: 43/43 memory + dashboard pass; 344/345 across the full suite (the single failure is test_ollama_client hitting httpx.ReadTimeout because no Ollama is running in this environment — unrelated). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

fazleelahhee · 2026-04-28T00:11:49Z

Superseded by #7, which consolidates this 5-PR stack into a single branch (ai-memory) against main with Copilot's review feedback addressed in 353dc27. Closing — please review #7.

Wire-up audit on main surfaced 8 items. All addressed. #1 production: RRF dedup broken in dual-write window (live-confirmed duplicate in session_recall). Fixed by canonicalising _content_key through grammar.compress(lite). #2 ux: dashboard returned raw compressed bytes. /api/memory/* now applies grammar.expand on read. #3 cce sessions migrate now compresses on insert (homogeneous storage). #4-7 test gaps closed. #8 confirmed reserved-for-future column. Tests: 603/603 (+25 new). CI green on Python 3.11 / 3.12 / 3.13. Live-verified the dedup fix on /home/fazle/trading/v3.

fazleelahhee requested a review from Copilot April 27, 2026 23:40

Copilot started reviewing on behalf of fazleelahhee April 27, 2026 23:40 View session

Copilot AI reviewed Apr 27, 2026

View reviewed changes

fazleelahhee mentioned this pull request Apr 27, 2026

feat(memory) PR 5/5: dashboard — sessions list, timeline, decisions search #6

Closed

2 tasks

fazleelahhee mentioned this pull request Apr 28, 2026

feat(memory): claude-mem feature parity — 5-step series consolidated #7

Merged

5 tasks

fazleelahhee closed this Apr 28, 2026

AZagatti mentioned this pull request May 12, 2026

Setup / first-run issues: broken post-commit hook, model download stalls, /tmp cache wipe, silent lazy index #67

Closed

rajkumarsakthivel deleted the feature/memory-pr3-compress branch May 19, 2026 20:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(memory) PR 3/5: compress — extractive worker in cce serve - #4

feat(memory) PR 3/5: compress — extractive worker in cce serve#4
fazleelahhee wants to merge 1 commit into
feature/memory-pr2-capturefrom
feature/memory-pr3-compress

fazleelahhee commented Apr 27, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 27, 2026

Uh oh!

Copilot AI Apr 27, 2026

Uh oh!

Copilot AI Apr 27, 2026

Uh oh!

fazleelahhee commented Apr 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

fazleelahhee commented Apr 27, 2026

PR 3 of 5 — Compress

What's added

Tier behaviour

Out of scope (lands later)

Test plan

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 27, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 27, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 27, 2026

Choose a reason for hiding this comment

Uh oh!

fazleelahhee commented Apr 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants