MCP-1: Add an MCP server layer for agentic-memory

[Atharva-Kanherkar]

MCP-1: Add an MCP server layer for agentic-memory

Summary

Add a Python MCP server that exposes agentic-memory as persistent memory infrastructure for external agents such as Claude Code, Cursor, Windsurf, and custom MCP clients.

The MCP layer should be a thin wrapper only:

• validate tool inputs
• call core memory services
• return structured outputs that help the host LLM decide the next action

All memory behavior stays in the core Python modules. The MCP server must not import or call any generative LLM SDK. The only model dependency remains Gemini Embedding 2 for embeddings.

Repo Reality Check

The current repo mostly matches the intended architecture, with a few important corrections:

• procedural memory is already implemented in code, even though the README still marks parts of it as planned
• semantic contradiction detection and supersession resolution already exist in forgetting/contradiction.py
• procedural outcome tracking and Wilson-score ranking already exist in stores/procedural_store.py
• there is no cognitive_memory/ package today; the business logic lives in top-level modules such as stores/, retrieval/, forgetting/, models/, and utils/
• PDF support exists explicitly via embed_pdf() and multimodal records
• event emission is broader than the initial brief: the code already emits memory.accessed, memory.ranked, memory.supersession_resolved, memory.faded, and forgetting-cycle events
• MemoryAPIService.__init__() currently mutates config.CHROMA_DB_PATH during runtime construction; that is not thread-safe and must be removed as part of the shared-runtime extraction

Why This Matters

Today the project has:

• a core memory engine
• a FastAPI wrapper
• a Next.js playground

What it does not yet have is the agent-facing protocol boundary that lets any MCP-capable host reuse the memory system directly without going through the web app.

That MCP boundary is the product surface for long-lived agent memory.

Internet-Researched Constraints

1. Prefer FastMCP from the official Python SDK

The official mcp Python SDK currently recommends:

• FastMCP for most server implementations
• Streamable HTTP as the preferred production transport
• stdio as the simplest local transport for IDE and CLI integrations

Implication:

• implement the server with FastMCP
• support both stdio and streamable-http
• default the README examples to stdio for Claude Code / Cursor / Windsurf local setups

2. Do not rely on MCP resources for primary memory retrieval

Multiple public MCP issues show that some hosts list resources correctly but do not reliably read them when answering questions.

Implication:

• make all primary memory operations tools, not resources
• resources can be added later for optional media access, but recall must not depend on them

3. Keep the tool surface tight

Current host constraints matter:

• Claude Code warns when MCP output exceeds 10,000 tokens
• Windsurf limits available MCP tools and also caps tool calls per prompt

Implication:

• keep the tool inventory small and high-signal
• prefer store-specific write tools and a small number of retrieval/admin tools
• bound result counts aggressively

4. Avoid rich binary output contracts in v1

Current MCP Python SDK and client issues show rough edges around:

• binary/base64 content handling
• resource encoding mismatches
• schema generation around image output types

Implication:

• accept media as local file paths, with base64 input as a fallback
• return structured JSON plus durable media_ref paths
• do not make raw binary/image tool outputs part of the critical path

5. Similar memory servers already hit config and race-condition problems

Public MCP memory-server issues include:

• path/env handling bugs
• write failures under concurrent usage
• search normalization bugs

Implication:

• validate all storage paths at startup
• add a core-level run lock for forgetting cycles before exposing mutating admin tools
• keep write paths single-owner and deterministic

Main Problems To Solve Before or During MCP Work

P0. The core service container is trapped inside api/app.py

MemoryAPIService currently lives inside the FastAPI module. The MCP server should not depend on FastAPI wiring.

Required change:

• extract a shared core service container used by both the API and MCP entrypoints
• stop mutating config.CHROMA_DB_PATH during runtime construction
• pass chroma_path directly into each store or its PersistentClient constructor instead of routing through module-level global config
• make embedding dimensions runtime-injectable instead of relying only on module-level config so future reindex migrations can stand up a parallel runtime cleanly

P1. Superseded semantic memories can still leak into normal retrieval

The current retriever fans out across stores but does not exclude semantic records that have been superseded.

Required change:

• add an active-record filter in SemanticStore so the store contract itself does not surface superseded facts by default

P2. Contradiction detection failures are currently swallowed in the API

_safe_contradiction_lookup() catches broad exceptions and silently returns [].

Required change:

• MCP write responses must expose contradiction-check status explicitly as completed, skipped, or error
• a failure to run contradiction detection must not be indistinguishable from “no contradictions found”

P3. Forgetting-cycle execution is not concurrency-guarded

The current forgetting service has no visible single-run guard.

Required change:

• add a core-level asyncio.Lock and return already_running for overlapping requests

P4. Manual single-record forgetting is not yet safe to expose

Low-level store delete() methods do not provide one coordinated path for:

• record deletion
• owned media cleanup
• event emission
• store-type resolution

Implication:

• do not expose a generic forget_memory MCP tool in v1
• expose forgetting-cycle preview/run first
• add a core admin delete service later if needed

Proposed MCP Tool Surface

Write tools

• remember_fact
• remember_episode
• remember_procedure
• resolve_contradiction
• record_procedure_outcome

Recall tools

• recall_memories
• recall_procedures
• recall_episodes
• get_memory

Admin and observability tools

• get_memory_overview
• preview_forgetting_cycle
• run_forgetting_cycle

Notably absent in v1:

• raw embedding tools
• raw Chroma collection tools
• EventBus subscription tools
• generic delete/forget-by-id tool
• resource-dependent primary recall flow

Tool-shape clarifications:

• get_memory should require both memory_type and record_id rather than probing all three stores
• recall_procedures should call ProceduralStore.get_best_procedure_matches() directly, not UnifiedRetriever.query()
• recall_episodes should cover recent, session, and time_range modes in one tool rather than splitting temporal recall into additional tools

Child Issues

• MCP-1A: Extract a shared runtime factory and remove global config mutation #58 MCP-1A: Extract a shared runtime factory and remove global config mutation
• MCP-1B: Harden core retrieval, contradiction status, and forgetting concurrency #59 MCP-1B: Harden core retrieval, contradiction status, and forgetting concurrency
• MCP-1C: Define MCP tool schemas, routing, and bounded response contracts #60 MCP-1C: Define MCP tool schemas, routing, and bounded response contracts
• MCP-1D: Implement the FastMCP server and transport startup #61 MCP-1D: Implement the FastMCP server and transport startup
• MCP-1E: Add multimodal MCP input handling for local paths and inline payloads #62 MCP-1E: Add multimodal MCP input handling for local paths and inline payloads
• MCP-1F: Package the server and document setup, ops, and migration guidance #63 MCP-1F: Package the server and document setup, ops, and migration guidance

Delivery Plan

MCP-1A: Extract a shared core runtime for API and MCP

Owns:

• new shared service-container module
• API import cleanup so api/app.py becomes an adapter, not the owner of runtime composition
• removal of the thread-unsafe config.CHROMA_DB_PATH mutation during runtime construction
• moving EventRecorder into the shared runtime if it remains part of the common observability surface
• injecting embedding dimensions through runtime construction so a migration can run 768-dim and 1536-dim runtimes side by side against different Chroma paths

Acceptance criteria:

• FastAPI and MCP entrypoints construct the same stores, retriever, contradiction detector, forgetting service, and media store from one shared runtime factory
• runtime construction never mutates module-level config state to swap Chroma paths
• chroma_path is injected directly into store/client construction, so concurrent runtime creation cannot observe another runtime's database path
• embedding dimensions can be supplied at runtime rather than only through module-level config, enabling parallel re-embedding into a fresh database path during future dimension upgrades

MCP-1B: Harden the core for MCP-safe behavior

Owns:

• active-only semantic retrieval filtering
• explicit contradiction-check status reporting
• forgetting-cycle lock / single-flight guard
• consistent exception mapping hooks
• event emission when ProceduralStore.record_outcome() updates a procedure
• removal of duplicate ContradictionDetector construction paths so the runtime owns one shared detector instance

Acceptance criteria:

• a superseded semantic record is not returned by default recall
• contradiction lookup failure is visible in the write response
• a second forgetting-cycle request receives already_running instead of overlapping execution
• procedural outcome recording emits an event that observability surfaces can show
• the forgetting guard uses asyncio.Lock rather than an ad hoc sync primitive in the async MCP runtime

MCP-1C: Define MCP schemas and serializers

Owns:

• typed request/response models for all MCP tools
• shared success/error envelope
• serialization of memory records, procedural matches, contradiction candidates, and forgetting reports
• get_memory(memory_type, record_id) routing in the request model
• explicit store-first-then-contradiction-check ordering in remember_fact
• configurable truncation for large forgetting reports

Acceptance criteria:

• all tools return machine-readable structured output
• errors are stable and typed
• responses stay below practical host token limits for common calls
• forgetting preview/run responses cap decisions to a configurable maximum and report when output was truncated
• remember_fact documentation makes it explicit that contradiction detection runs only after the new semantic record has been stored

MCP-1D: Implement the MCP transport layer

Owns:

• FastMCP server
• one runtime entrypoint that selects stdio or streamable-http
• startup validation and health exposure

Acceptance criteria:

• local clients can run the server over stdio
• remote/self-hosted clients can connect over /mcp via streamable HTTP
• the server can switch transport via a single CLI/env-controlled startup path, for example MEMORY_MCP_TRANSPORT=stdio|streamable-http

MCP-1E: Add multimodal input handling for MCP calls

Owns:

• file-path input support
• base64 input fallback
• temporary-file handling for inline payloads
• size limits and validation errors
• explicit validation that local file paths are readable from the server filesystem, not merely meaningful to the client

Acceptance criteria:

• screenshot, audio, and PDF-backed memories can be stored via MCP without the FastAPI upload endpoints
• text recall can find media-backed memories
• remote deployments fail clearly when a caller provides a local path that is not accessible from the MCP server host

MCP-1F: Document client setup and integration behavior

Owns:

• README setup for Claude Code, Cursor, and Windsurf
• env var reference
• startup modes
• deployment notes for local-path media behavior when the server is remote
• tested mcp package version and pinning guidance
• embedding-dimension migration guidance so users understand that changing vector size requires a re-embedding cutover, not an in-place live switch
• troubleshooting section for common MCP client issues

Acceptance criteria:

• a user can install dependencies, start the server, and connect one of the three supported hosts without guessing config structure

Defensive Engineering Rules

• Use structured output for every tool. Do not rely on free-form strings.
• Keep tool descriptions short and trigger-oriented so hosts call the right tool without orchestration glue.
• Prefer local file paths for media input. Base64 is fallback-only.
• Bound all list-style results with explicit defaults and max values.
• Exclude superseded semantic memories from default recall.
• Never silently swallow contradiction-check failures.
• Return typed retryable metadata on dependency failures.
• Keep the MCP layer free of business logic. If logic is missing, add it in core first.
• reject or loudly warn on relative storage paths in non-local deployments so the server does not accidentally point at a different Chroma database because of working-directory drift
• treat embedding-dimension changes as reindex migrations: never point an existing Chroma collection at a new vector size and hope it works

Packaging Plan

Add a real Python package boundary so users can run only the MCP server if they do not need the API or web UI.

Recommended shape:

• add pyproject.toml
• add a console script such as agentic-memory-mcp
• make MCP dependencies installable as an extra
• keep API dependencies installable separately
• pin the tested mcp version explicitly instead of floating across point releases

Suggested dependency split:

• core dependencies in [project.dependencies]: chromadb, google-genai, numpy, python-dotenv
• API extra in [project.optional-dependencies].api: fastapi, uvicorn, python-multipart
• MCP extra in [project.optional-dependencies].mcp: mcp[cli]

Recommended runtime/config additions:

• allow EMBEDDING_DIMENSIONS to be overridden via env and runtime constructor parameters
• keep the default at 768 for now
• treat 1536 as a supported migration target, not a hot-swappable live toggle on existing collections

Future Migration Note: 768 to 1536

gemini-embedding-2-preview supports both 768-dimensional and 1536-dimensional embeddings. Moving from 768 to 1536 is a valid production migration, but it is a one-time re-embedding and reindexing operation because Chroma collection dimensions are fixed once data has been written.

Guidance:

• do not change embedding size in place on an existing Chroma path
• stand up a new runtime with the target embedding size and a fresh Chroma path
• read records from the old collections
• re-embed all records at the new size
• write them into new collections
• verify retrieval quality and record counts
• switch traffic to the new runtime/path
• retire the old collections after validation

When this work happens, MCP-1A should already have made the necessary runtime changes:

• chroma_path injected directly instead of via global config mutation
• embedding dimensions injected at runtime
• parallel runtimes possible during migration without restart-coupled global state

Operational recommendation:

• keep 768 as the default unless retrieval quality justifies the migration cost
• support 1536 cleanly once the runtime/config injection refactor lands

Acceptance Surface

The MCP server is complete when:

• an MCP host can store semantic, episodic, and procedural memories
• contradiction candidates are returned on semantic writes and can be resolved explicitly
• procedural recall uses similarity plus Wilson-score ranking
• procedural outcomes can be recorded and affect future ranking
• text recall can retrieve media-backed episodic and semantic memories
• forgetting cycles can be previewed and run safely
• the server runs without the FastAPI app
• setup docs exist for Claude Code, Cursor, and Windsurf

Architecture

flowchart LR
    Host[Claude Code / Cursor / Windsurf / Custom MCP Client]
    MCP[FastMCP Server]
    Runtime[Shared Memory Runtime]
    RET[UnifiedRetriever]
    CD[ContradictionDetector]
    FGT[ForgettingService]
    SS[SemanticStore]
    ES[EpisodicStore]
    PS[ProceduralStore]
    MS[MediaStore]
    EMB[GeminiEmbedder]
    CH[(ChromaDB)]
    BUS[EventBus]

    Host --> MCP
    MCP --> Runtime
    Runtime --> RET
    Runtime --> CD
    Runtime --> FGT
    Runtime --> SS
    Runtime --> ES
    Runtime --> PS
    Runtime --> MS
    SS --> EMB
    ES --> EMB
    PS --> EMB
    SS --> CH
    ES --> CH
    PS --> CH
    SS -. emit .-> BUS
    ES -. emit .-> BUS
    PS -. emit .-> BUS
    RET -. emit .-> BUS
    FGT -. emit .-> BUS

Loading

Sources

• Official MCP Python SDK: https://github.com/modelcontextprotocol/python-sdk
• MCP Python SDK on PyPI: https://pypi.org/project/mcp/
• Claude Code MCP docs: https://docs.anthropic.com/en/docs/claude-code/mcp
• Cursor MCP docs: https://docs.cursor.com/context/mcp
• Windsurf MCP docs: https://docs.windsurf.com/windsurf/cascade/mcp
• MCP Python SDK issue on resource auto-use: Claude Desktop never uses resources from my MCP server modelcontextprotocol/python-sdk#1016
• MCP Python SDK issue on binary/resource encoding: MCP Server Base64 encode/decode mismatch for Resources modelcontextprotocol/python-sdk#342
• MCP Python SDK issue on image/schema output rough edges: PydanticSchemaGenerationError: Unable to generate pydantic-core schema for Image type modelcontextprotocol/python-sdk#1060
• Official MCP servers repo and memory-server issue tracker: https://github.com/modelcontextprotocol/servers

[Atharva-Kanherkar]

@claude please review this plan throughly, including the sources mentioned.

[claude]

Claude finished @Atharva-Kanherkar's task in 3s —— View job

---

Plan Review: MCP-1 Add an MCP server layer for agentic-memory

Todo List

• Initialize review task
• Explore current codebase structure
• Review core modules (stores, retrieval, forgetting, models)
• Check current API layer (api/app.py)
• Analyze plan accuracy against codebase reality
• Provide comprehensive feedback

---

Overall Assessment

The plan is well-researched and architecturally sound. The problem diagnoses (P0–P4) are accurate, the delivery sequencing is logical, and the tool surface decisions are defensible. A few gaps and one unmentioned bug are worth flagging before implementation starts.

---

Accuracy Check Against the Codebase

All four pre-existing bug claims check out:

P0 — MemoryAPIService is in api/app.py: Confirmed at api/app.py:312. The class is well-structured and already accepts chroma_path, media_root, and embedder constructor params, so extraction is mostly a file-move. ✅

P1 — Superseded records leak into retrieval: Confirmed. retrieval/retriever.py:147 iterates all stores and calls store.retrieve(text, top_k=fetch_k) with no post-filter. SemanticMemory has a superseded_by field (models/semantic.py:15) but SemanticStore.retrieve_by_vector() does not filter on it. ChromaDB supports where filters on metadata, so the fix should go inside SemanticStore.retrieve() and retrieve_by_vector() using where={"superseded_by": ""} — not in the retriever itself. The plan correctly identifies the problem but leaves the fix strategy open. Worth specifying. ✅

P2 — Contradiction failures swallowed: Confirmed at api/app.py:398-405. The except (ValueError, Exception): clause is redundant (since Exception already covers ValueError) and effectively catches everything silently. ✅

P3 — No forgetting-cycle lock: Confirmed. forgetting/service.py:148 — run_cycle() has no guard. Two concurrent calls will both run _plan_cycle() and _execute_prunes() on the same store state. ✅

---

Unmentioned Bug: Thread-Unsafe Global Config Mutation (MCP-1A Risk)

MemoryAPIService.__init__() at api/app.py:323-343 does this:

original_chroma_path = config.CHROMA_DB_PATH
try:
    if chroma_path is not None:
        config.CHROMA_DB_PATH = chroma_path
    self.semantic_store = SemanticStore(...)
    ...
finally:
    config.CHROMA_DB_PATH = original_chroma_path

This mutates a module-level global and relies on the finally block to restore it. It is not thread-safe — if two service containers are initialized concurrently (e.g., during testing), they can observe each other's path. When MCP-1A extracts this into a shared runtime factory, fix this by passing chroma_path directly to the PersistentClient constructor inside each store rather than going through config. This should be called out explicitly in MCP-1A's acceptance criteria.

---

Tool Surface Review

The 13-tool surface is reasonable. A few design questions worth settling before MCP-1C:

get_memory ambiguity: The three stores are type-specific — there is no cross-store get_by_id. get_memory will need to either (a) require a memory_type parameter and route to the right store, or (b) try all three stores in sequence. Option (a) is simpler and avoids triple-lookup overhead. Recommend adding memory_type as a required param and documenting it clearly.

recall_procedures vs UnifiedRetriever.query(): ProceduralStore.get_best_procedure_matches() (stores/procedural_store.py:157) already handles the similarity + Wilson-score blending that recall_procedures needs. Do not route this through UnifiedRetriever.query() — that would lose the Wilson-score weighting. Call ProceduralStore.get_best_procedure_matches() directly.

recall_episodes time-range: UnifiedRetriever.query_time_range() exists and routes to EpisodicStore.get_by_time_range(). Consider adding optional start / end ISO timestamp params to recall_episodes rather than making time-range a separate tool. This keeps the surface at 13 instead of 14 and avoids a proliferation of recall tools.

record_procedure_outcome event gap: ProceduralStore.record_outcome() (stores/procedural_store.py:140) updates and replaces the record but emits no event. For MCP observability, an event should fire here. This is a minor hardening item that fits naturally into MCP-1B.

---

Delivery Plan Sequencing

The MCP-1A → 1B → 1C → 1D → 1E → 1F ordering is correct. A few notes per phase:

MCP-1A (shared runtime): MemoryAPIService extraction is straightforward. The main risk is the global config mutation mentioned above. Also consider whether EventRecorder stays FastAPI-only or moves to the shared runtime — it depends only on EventBus, so it can move.

MCP-1B (harden core):

• The superseded-record filter belongs in SemanticStore (not the retriever) to keep the store contract self-consistent.
• The contradiction status enum (completed | skipped | error) needs to be a first-class return type — not inferred from an empty list.
• The forgetting lock can be a simple threading.Lock or asyncio.Lock depending on whether the MCP server is sync or async. FastMCP is async, so prefer asyncio.Lock.

MCP-1C (schemas): The plan correctly flags that responses must stay under ~10,000 tokens for Claude Code. For preview_forgetting_cycle / run_forgetting_cycle, the decisions list in ForgettingReport can be very long. Consider capping it at a configurable max_decisions with a count summary when truncated.

MCP-1D (transport): Streamable HTTP (/mcp) as the production transport is correct for the current SDK. The stdio default for local IDE integrations is also correct. One nuance: FastMCP supports both transports via mcp.run() with a transport argument — no need for separate entrypoints. A single __main__ that reads a MEMORY_MCP_TRANSPORT env var (stdio or streamable-http) keeps the deployment model simple.

MCP-1E (multimodal): The plan correctly defers binary output. Accepting local file paths is the right v1 choice. One thing to clarify: for the remember_episode and remember_fact tools that accept media_path, the MCP server will need to validate that the path is accessible from the server's filesystem — not the client's. This matters when the server runs remotely. Document this clearly in MCP-1F.

---

Packaging

requirements.txt has no mcp package. The pyproject.toml plan is the right shape. For the console script and dependency extras, a rough structure:

[project.optional-dependencies]
api = ["fastapi", "uvicorn", "python-multipart"]
mcp = ["mcp[cli]"]

This lets users install only what they need. mcp[cli] includes the fastmcp CLI runner. The core dependencies (chromadb, google-genai, numpy, python-dotenv) go in [project.dependencies].

---

Minor Issues

• forgetting/contradiction.py:29 requires the record to be already stored before calling find_potential_contradictions. The MCP remember_fact flow must therefore store first, then call the contradiction detector — the same order as the current API. Make this ordering explicit in MCP-1C's request model documentation.
• The ForgettingService initializes its own ContradictionDetector if none is passed (forgetting/service.py:146). When both the API/MCP layers and ForgettingService use a shared ContradictionDetector instance from the runtime factory, this default-construction fallback should be removed to avoid two detectors holding separate state.
• config.py uses ./chroma_db as a relative path default. When the MCP server is launched from a different working directory than the API, both will resolve to different Chroma databases unless an absolute path is configured. Add a startup validation step (already mentioned in the plan's defensive rules) that logs or errors on relative paths in production.

---

Sources

External source fetching was not available in this environment. Based on SDK knowledge current to the plan's writing date, the SDK claims (FastMCP preferred, streamable HTTP as production transport, stdio for local) are accurate. One nuance worth double-checking before MCP-1D: the mcp PyPI package version may have changed its FastMCP API between point releases — pin to a specific version in pyproject.toml and note the tested version in the README.

---

Summary

The plan is accurate, well-scoped, and ready to implement with these additions:

1. Flag and fix the global config mutation in MCP-1A acceptance criteria
2. Specify where P1 filter lives (in SemanticStore, not the retriever)
3. Add memory_type to get_memory tool or document the resolution strategy
4. Route recall_procedures through ProceduralStore.get_best_procedure_matches() directly
5. Cap decisions list in forgetting tools to avoid token-limit issues
6. Emit an event from record_outcome() as part of MCP-1B hardening
7. Use asyncio.Lock for forgetting cycle guard given FastMCP's async runtime
8. Pin mcp version in pyproject.toml

[claude]

Claude finished @Atharva-Kanherkar's task in 4m 32s —— View job

---

Plan Review: MCP-1 Add an MCP server layer for agentic-memory

Todo List

• Initialize review task
• Explore codebase structure (stores, retrieval, forgetting, models, api)
• Verify plan's claims against actual code
• Analyze plan completeness, correctness, and risks
• Write comprehensive review

---

Overall Assessment

The plan is well-researched, architecturally sound, and its diagnosis of the codebase is accurate. The phased delivery (MCP-1A → 1F) is the right sequencing — hardening core before exposing it over MCP is the correct order of operations. The issues below are refinements, not blockers.

---

Codebase Reality Check — Verified

I read the actual source files to verify the plan's claims:

Claim	Verified
Procedural memory with Wilson-score ranking implemented	✅ stores/procedural_store.py + models/procedural.py
Supersession/contradiction in forgetting/contradiction.py	✅ resolve_supersession() sets importance=0.0, emits memory.supersession_resolved
No cognitive_memory/ package	✅ Correct — top-level stores/, retrieval/, forgetting/
embed_pdf() exists	✅ utils/embeddings.py, delegating to _embed_media()
Richer event set	✅ memory.stored, memory.retrieved, memory.ranked, memory.accessed, memory.contradiction_flagged, memory.supersession_resolved, memory.faded, memory.forgotten, forgetting.cycle_completed, forgetting.cycle_dry_run
No MCP code today	✅ No mcp import anywhere
No pyproject.toml	✅ Only requirements.txt

---

Problem Diagnoses — All Accurate

P0 (MemoryAPIService trapped in api/app.py) — Confirmed. MemoryAPIService.__init__ at api/app.py:312 is where all stores, the retriever, the contradiction detector, and the forgetting service are wired. The FastAPI create_app() function owns runtime construction. The MCP server has no safe import path to these today.

One additional smell to address during MCP-1A: lines 323–343 mutate the global config.CHROMA_DB_PATH inside a try/finally to inject a custom path. This is thread-unsafe and should be replaced with proper constructor-level path passing when the shared runtime is extracted.

P1 (Superseded records leak into retrieval) — Confirmed and serious. retrieval/retriever.py:148 calls store.retrieve(text, top_k=fetch_k) with no post-filter. rank_results() in retrieval/ranking.py also has no supersession awareness. Records with a non-null superseded_by field are scored and returned identically to active records. The fix belongs in _rank_and_touch() or as a pre-rank filter for semantic results.

P2 (Contradiction failures silently swallowed) — Confirmed. api/app.py:398–405:

def _safe_contradiction_lookup(...):
    try:
        return active_service.contradiction_detector.find_potential_contradictions(record)
    except (ValueError, Exception):
        return []

except Exception catches everything including network/embedding errors. A caller receiving [] cannot distinguish "no contradictions" from "detection failed." The plan's proposed completed / skipped / error status envelope is the right fix.

P3 (Forgetting cycle not concurrency-guarded) — Confirmed. ForgettingService.run_cycle() at forgetting/service.py:148 has no lock whatsoever. Two concurrent calls will both scan all records, plan independently, and execute overlapping fades/prunes. This is a real data integrity risk.

P4 (No coordinated delete path) — Confirmed. No single method coordinates record deletion + media cleanup + event emission across store types. Correct to defer a generic forget_memory tool to v1 post.

---

Technical Concerns and Recommendations

1. MCP-1A: Specify where the shared runtime lives

The plan says "extract a shared core service container" but doesn't name the module. Recommend core/runtime.py (or memory_runtime.py at the project root) exporting a MemoryRuntime class and a create_runtime() factory. Both api/app.py and the MCP entrypoint import only from there. Without a concrete location decision, MCP-1A risks being interpreted differently by different contributors.

2. P3: Specify the lock mechanism

The plan requires a "core-level lock" but doesn't specify what kind. Two options:

• asyncio.Lock — correct if FastMCP runs async and the forgetting service is only called from one process
• File lock (fcntl.flock or filelock) — required if API and MCP run as separate processes sharing the same ChromaDB on disk

Since both transports (stdio + HTTP) will be supported and may run alongside the FastAPI server, a file-based lock is the safer default. This should be decided in the MCP-1B spec, not left implicit.

3. P1 filter scope

The supersession filter should only apply to semantic records — episodic and procedural stores don't use superseded_by. The fix in UnifiedRetriever should filter based on record.memory_type == "semantic" and record.superseded_by is not None. An include_superseded: bool = False parameter on the recall tools gives hosts an explicit escape hatch if they need the full history.

4. Missing: Tests in the delivery plan

MCP-1B involves behavioral changes to the core (supersession filter, contradiction status, forgetting lock). MCP-1C adds new schema models. Neither phase mentions tests. At minimum, MCP-1B's acceptance criteria should include:

• A test that queries after supersession and confirms the old record is excluded
• A test that simulates an embedding failure and confirms error status propagates
• A test that calls run_cycle() concurrently and confirms only one executes

5. Token bounding for preview_forgetting_cycle

ForgettingReport.decisions is an unbounded list — one entry per record scanned. On a large memory store this could easily exceed Claude Code's 10,000-token MCP output warning. The MCP-1C schema for preview_forgetting_cycle should cap decision details (e.g., max_decisions: int = 50) and always include the aggregate counts (scanned, faded, pruned, etc.) regardless of the cap.

6. get_memory tool is underspecified

The tool surface lists get_memory under recall tools but the plan body never describes it. This is presumably a single-record lookup by ID. Should it resolve across all stores, or require memory_type? What does it return for a superseded record — the record with a superseded_by pointer, or a redirect to the successor? This needs a concrete spec in MCP-1C.

7. resolve_contradiction needs guard rails

ContradictionDetector.resolve_supersession(superseded_id, kept_id) takes two IDs. There's no validation that the IDs actually form a contradiction pair or even exist in the same store. The MCP tool should validate:

• Both records exist
• superseded_id != kept_id
• The semantic store owns both records (procedural/episodic supersession is not defined)

8. Multi-user / session isolation not addressed

The plan is silent on whether multiple hosts (e.g., two Claude Code windows) sharing one MCP server share one flat memory namespace. This is likely intentional (shared persistent memory is the product), but it should be explicitly stated in MCP-1F documentation so users don't create accidental cross-project pollution. A namespace or agent_id parameter on write tools can be deferred, but the decision should be documented.

9. Temp file cleanup for multimodal inputs (MCP-1E)

The existing save_query_upload() in api/app.py:376 creates a temp file via NamedTemporaryFile(delete=False) and never cleans it up — this is an existing leak. MCP-1E should not repeat this pattern. For base64 inline inputs, use a context manager or explicit cleanup after the embedding call completes.

---

Minor Points

• Transport naming: The plan correctly calls out "Streamable HTTP" as the preferred production transport. Worth noting that SSE (Server-Sent Events) is still the transport used by most Cursor and Windsurf versions today. The MCP-1D and MCP-1F work should test against both if broad client compatibility is a goal.
• Architecture diagram: Shows SS → EMB, ES → EMB, PS → EMB as separate arrows implying three embedder instances. The actual code uses one shared GeminiEmbedder passed to all three stores. MCP-1A should preserve this single-instance pattern — don't instantiate embedders per-store in the new runtime factory.
• recall_episodes tool: The plan lists this but doesn't distinguish between text-search episodic retrieval (goes through UnifiedRetriever.query() with memory_types=["episodic"]) and time-range episodic retrieval (query_time_range()). These are meaningfully different operations and may warrant separate tools or a query_mode parameter.

---

Source Review Note

WebFetch was not granted in this run, so the referenced GitHub issue URLs (python-sdk/issues/1016, 342, 1060) could not be fetched directly.

Based on the MCP Python SDK as of early 2026: FastMCP is the correct choice, Streamable HTTP is the right production transport, and the constraints cited in the plan around resource reliability and binary output are accurate reflections of known SDK rough edges. The plan's mitigations (tools-only primary recall, no binary output contracts in v1, local file paths for media) are all sound.

---

Summary

The plan is accurate, well-motivated, and sequenced correctly. The five identified problems (P0–P4) are all real and verified in the code. The main gaps to address before or during execution:

1. Name the shared runtime module explicitly in MCP-1A
2. Decide file-lock vs. asyncio-lock for P3 before writing code
3. Add test requirements to MCP-1B acceptance criteria
4. Bound preview_forgetting_cycle output in MCP-1C schemas
5. Spec get_memory and resolve_contradiction more precisely in MCP-1C
6. Fix the config.CHROMA_DB_PATH global mutation during MCP-1A extraction

[Atharva-Kanherkar]

Accepted review points and plan amendments:

• Added the thread-unsafe config.CHROMA_DB_PATH mutation as an explicit MCP-1A risk; MCP-1A now requires passing chroma_path directly into store/client construction and forbids module-level config mutation during runtime creation.
• Clarified that the superseded-record filter belongs in SemanticStore, not the retriever.
• Clarified tool-shape decisions: get_memory(memory_type, record_id), recall_procedures calls ProceduralStore.get_best_procedure_matches() directly, and recall_episodes covers recent|session|time_range in one tool.
• Added the record_procedure_outcome() event gap to MCP-1B hardening.
• Specified asyncio.Lock for the forgetting-cycle guard.
• Added configurable truncation for large forgetting reports.
• Made the remember_fact ordering explicit: store first, then contradiction lookup.
• Updated transport notes so MCP-1D uses one runtime entrypoint with transport selected by CLI/env.
• Added remote-path validation notes for multimodal file inputs.
• Expanded packaging to pin a tested mcp version and split extras as api and mcp.

The repo-local draft has been updated accordingly in docs/issues/mcp-server-plan.md.

[Atharva-Kanherkar]

@claude updated. re review. also. explain the whole concepts to me in layman terms. use mermaid diagrams if needed, and simple language.

[claude]

Claude finished @Atharva-Kanherkar's task in 2m 24s —— View job

---

Re-review + Layman Explanation

Todo List

• Initialize task
• Read updated plan (docs/issues/mcp-server-plan.md)
• Re-review the updated plan against the codebase
• Write layman explanation with mermaid diagrams

---

Re-review of Updated Plan

Note: docs/issues/mcp-server-plan.md doesn't exist as a file in the repo yet. The update was described in the issue comment at 17:39. Re-reviewing based on those accepted changes.

What Changed — and Whether It's Right

✅ config.CHROMA_DB_PATH mutation flagged as MCP-1A risk
Good. The actual code at api/app.py:323–343 does exactly the unsafe pattern: it saves/restores a module-level global inside try/finally. If two threads hit MemoryAPIService.__init__() at the same time, they corrupt each other's path. The fix (pass chroma_path into store constructors directly) is now in MCP-1A's scope. But note: SemanticStore.__init__ at stores/semantic_store.py:25 still reads config.CHROMA_DB_PATH directly — that line is the other half of the fix and must also change when stores are extracted.

✅ Superseded-record filter moved to SemanticStore
Correct placement. retrieval/retriever.py:148 calls store.retrieve() — if the filter lives in the store, it's automatic everywhere. Using ChromaDB's where={"superseded_by": ""} metadata filter inside SemanticStore.retrieve() and retrieve_by_vector() is the right implementation path.

✅ get_memory(memory_type, record_id) — required param added
Good call. Avoids triple-store probing. One remaining spec gap: what should the tool return if memory_type="semantic" but the record is superseded? Recommend returning the record with its superseded_by pointer visible, so the caller knows a newer record exists.

✅ recall_procedures calls ProceduralStore.get_best_procedure_matches() directly
Correct. UnifiedRetriever.query() would strip the Wilson-score weighting. Direct call preserves ranking behavior.

✅ recall_episodes covers recent|session|time_range in one tool
Good. Avoids proliferation. The query_mode parameter should be explicit in the schema with an enum: recent | session | time_range. time_range mode needs start and end ISO timestamps as optional params.

✅ record_procedure_outcome() event gap added to MCP-1B
Confirmed needed. stores/procedural_store.py:140 updates and replaces the record with no event emission. This is observable via MCP so it matters.

✅ asyncio.Lock for forgetting cycle
Right choice since FastMCP runs async. One nuance: if the API (uvicorn) and MCP server run as separate processes sharing the same ChromaDB on disk, an asyncio.Lock does NOT protect across processes — only within one. If the deployment is ever two separate OS processes, a file lock (filelock) is needed. For v1 where they likely share a process or run independently, asyncio.Lock is fine. Worth a comment in the code when implementing.

✅ Configurable truncation for forgetting reports
Good. ForgettingReport.decisions is unbounded (forgetting/service.py:53). A max_decisions: int = 50 with a truncated: bool + total_decisions: int in the response envelope is the right shape.

✅ remember_fact ordering: store first, then contradiction lookup
Confirmed correct by the code. forgetting/contradiction.py:29 requires the record to already exist in ChromaDB before similarity search can find potential contradictions against it.

✅ Single entrypoint, transport via MEMORY_MCP_TRANSPORT env var
Clean. FastMCP supports both stdio and streamable-http via mcp.run(transport=...) — no need for separate scripts.

✅ Remote path validation for multimodal
Important. The MCP server must reject media_path values that are not readable from the server's filesystem, with a clear error like "path not accessible from server: /Users/...".

✅ mcp version pinning
Essential. The MCP SDK has been moving fast. Pin to the specific version tested, e.g. mcp[cli]==1.x.y.

---

Still-Open Gaps (Not Addressed in Updated Plan)

Gap	Risk	Suggested Fix
No test requirements in MCP-1B acceptance criteria	Medium — behavioral changes with no regression safety net	Add: supersession filter test, contradiction-error propagation test, concurrent forgetting test
resolve_contradiction tool has no guard rails spec	Medium — caller can pass non-existent or same-store-mismatched IDs	Specify: validate both IDs exist, both are semantic, superseded_id != kept_id
Multi-user namespace isolation not documented	Low for v1, confusing later	Add one line to MCP-1F: "all memories share a flat namespace by default"
save_query_upload() temp file leak (api/app.py:376)	Low — existing bug, MCP-1E must not repeat it	Use context manager or explicit cleanup in MCP-1E's file-path handling

---

Overall Re-review Verdict

The updated plan is ready to implement. All eight prior recommendations were addressed correctly. The four remaining gaps above are refinements, not blockers — the most important one is adding test acceptance criteria to MCP-1B before writing code.

---

---

Layman Explanation: What is This All About?

The Big Picture in One Sentence

This project gives AI agents (like Claude, Cursor's AI, etc.) a persistent memory system — so they remember things across conversations — and we're adding a standard plug (MCP) so any AI tool can use it without special wiring.

---

Part 1: The Three Types of Memory

Think of it like a human brain:

mindmap
  root((Agent Memory))
    Semantic
      Facts & knowledge
      "The sky is blue"
      "User prefers TypeScript"
    Episodic
      Past events & experiences
      "Yesterday we debugged auth"
      "User uploaded a screenshot at 3pm"
    Procedural
      How to do things
      "Steps to deploy to prod"
      "How to fix this type of bug"

Loading

• Semantic = the "what is true" brain. Facts. Knowledge. Things that can be superseded ("I thought X, but actually Y").
• Episodic = the "what happened" brain. A diary of events, time-stamped.
• Procedural = the "how to do it" brain. Recipes. Ranked by success rate (if a procedure works well, it ranks higher).

---

Part 2: How the System Currently Works

flowchart TD
    User["User / Web Browser"]
    API["FastAPI Web Server\n(api/app.py)"]
    SS["SemanticStore\n(facts)"]
    ES["EpisodicStore\n(events)"]
    PS["ProceduralStore\n(procedures)"]
    RET["UnifiedRetriever\n(search across all stores)"]
    CD["ContradictionDetector\n(finds conflicting facts)"]
    FGT["ForgettingService\n(prunes old/weak memories)"]
    CH[("ChromaDB\n(vector database on disk)")]
    EMB["Gemini Embeddings\n(turns text into numbers)"]
    BUS["EventBus\n(logs what's happening)"]

    User --> API
    API --> SS & ES & PS
    API --> RET
    API --> CD
    API --> FGT
    SS & ES & PS --> EMB
    SS & ES & PS --> CH
    SS & ES & PS -. events .-> BUS
    RET --> SS & ES & PS

Loading

How a memory gets stored:

1. You send text to the API
2. Gemini converts it to a vector (a list of 768 numbers that captures meaning)
3. That vector + metadata gets saved to ChromaDB (a database that can find similar vectors fast)

How recall works:

1. You ask a question
2. Gemini converts the question to a vector
3. ChromaDB finds the stored memories whose vectors are most similar
4. The retriever ranks them by recency, importance, and access count
5. Results come back

How forgetting works:
Memory importance decays over time (like human memory). Old, unaccessed, or superseded memories fade and eventually get pruned. The forgetting service runs this cleanup cycle.

---

Part 3: The Problem — No Standard Plug

Right now, only the web UI can use this memory system. If Claude Code wants to use it, or Cursor, or any other AI tool — they all have to talk to the FastAPI web server. That's messy and non-standard.

MCP (Model Context Protocol) is like a USB-C standard for AI tools. Any AI host that speaks MCP can plug into any MCP server without custom wiring.

flowchart LR
    subgraph "Without MCP (today)"
        CC1["Claude Code"] --"custom HTTP"--> API1["FastAPI"]
        CUR1["Cursor"] --"can't connect"--> API1
        WS1["Windsurf"] --"can't connect"--> API1
    end

    subgraph "With MCP (after this issue)"
        CC2["Claude Code"] --"MCP"--> MCP["FastMCP Server"]
        CUR2["Cursor"] --"MCP"--> MCP
        WS2["Windsurf"] --"MCP"--> MCP
        MCP --> CORE["Shared Memory Runtime\n(same core engine)"]
    end

Loading

---

Part 4: The Four Bugs to Fix First (P0–P3)

Before the plug can be added safely, four problems in the core need fixing:

P0 — The kitchen is inside the restaurant (not reusable)
All the "wire everything together" code lives inside the FastAPI app. The MCP server can't import it cleanly. Fix: extract a shared "runtime factory" that both API and MCP use.

P1 — Old wrong facts still show up in search results
When a fact gets superseded ("I was wrong, the answer is actually X"), the old record is marked but NOT excluded from search. Fix: SemanticStore.retrieve() should filter out superseded records by default.

P2 — Silent failures look like success
When contradiction detection fails (e.g., Gemini is down), the API returns an empty list — same as "no contradictions found." The MCP layer must tell the difference. Fix: return completed / skipped / error status explicitly.

P3 — Two forgetting cycles can run at the same time
If two requests trigger run_forgetting_cycle simultaneously, they'll both scan and prune independently, potentially corrupting state. Fix: add an asyncio.Lock so only one runs at a time; the second gets "already_running".

---

Part 5: The 13 Tools the MCP Server Will Expose

Think of these as buttons an AI can press:

mindmap
  root((MCP Tools))
    Write
      remember_fact
      remember_episode
      remember_procedure
      resolve_contradiction
      record_procedure_outcome
    Recall
      recall_memories
      recall_procedures
      recall_episodes
      get_memory
    Admin
      get_memory_overview
      preview_forgetting_cycle
      run_forgetting_cycle

Loading

Write tools = save new memories
Recall tools = search and fetch memories
Admin tools = see what's stored, simulate/run cleanup

---

Part 6: The Delivery Order (Why This Sequence)

flowchart LR
    A["MCP-1A\nExtract shared runtime\n(fix P0 + thread-unsafe config)"]
    B["MCP-1B\nHarden core\n(fix P1, P2, P3, P4)"]
    C["MCP-1C\nDefine tool schemas\n(what goes in, what comes out)"]
    D["MCP-1D\nBuild transport layer\n(stdio + HTTP server)"]
    E["MCP-1E\nMultimodal inputs\n(images, audio, PDF via file path)"]
    F["MCP-1F\nDocument setup\n(Claude Code, Cursor, Windsurf)"]

    A --> B --> C --> D --> E --> F

Loading

You can't build the plug (1D) before you know the shape of the data (1C). You can't define shapes (1C) before the core is safe (1B). You can't harden the core (1B) before it's extractable (1A).

---

Part 7: What "Done" Looks Like

A user adds this to their Claude Code config:

{
  "mcpServers": {
    "agentic-memory": {
      "command": "agentic-memory-mcp",
      "args": []
    }
  }
}

Then Claude Code automatically:

• Calls remember_fact when it learns something important
• Calls recall_memories when it needs context about past work
• Calls run_forgetting_cycle to keep the memory store healthy

No web browser. No manual API calls. Just the AI using memory like a built-in capability — persistent across every conversation.

---

[Atharva-Kanherkar]

@claude do you think the point of adding 700ish dimensions could be increased in production?

[claude]

Claude finished @Atharva-Kanherkar's task in 58s —— View job

---

Answering: Can embedding dimensions be increased in production?

Todo List

• Understand the question
• Check current embedding setup in the codebase
• Provide a thorough answer

---

Short Answer: Yes — but it's a breaking migration, not a config bump.

The comment in config.py:10 says it all:

EMBEDDING_DIMENSIONS = 768  # Matryoshka truncation — can bump to 1536 later

The model in use is gemini-embedding-2, which is a Matryoshka embedding model. This means the full vector is 3072 dimensions, and you can truncate it to a smaller size at query time. Currently the code requests 768-dimensional vectors via output_dimensionality=768 in utils/embeddings.py:466,475.

---

What Matryoshka embeddings mean

Matryoshka Representation Learning (MRL) trains a single model so that the first N dimensions of a larger vector are already a valid, coherent embedding at that smaller size. It's like Russian nesting dolls — the 768-dim vector is "inside" the 1536-dim vector, which is "inside" the 3072-dim vector.

[  768-dim  ]
[  768-dim  |  +768  ] = 1536-dim
[  768-dim  |  +768  |  +1024  ] = 3072-dim (full)

Increasing dimensions gives more expressive representations — better at distinguishing subtle meaning differences — at the cost of more memory and slower cosine similarity searches.

---

What changing this would break

The dimension is baked into the ChromaDB collection at creation time. ChromaDB stores vectors in a fixed-size index. If you change EMBEDDING_DIMENSIONS and point at the same ./chroma_db, you will get errors on insert because the new vectors don't match the stored index shape.

Concretely:

• utils/embeddings.py:297 validates every incoming vector: if len(vector) != EMBEDDING_DIMENSIONS: raise
• ChromaDB itself will reject writes if the vector dimension doesn't match the collection's configured space

This means a dimension change requires a full data migration — re-embed every stored memory at the new dimension and write to a fresh ChromaDB collection.

---

Production trade-offs

Dimension	Relative quality	Memory per vector	Notes
768 (current)	Good	3 KB	Current setting
1536	Better	6 KB	What config.py suggests as next step
3072	Best	12 KB	Full model capacity

For an agent memory use case with potentially thousands of stored memories, the practical considerations are:

• 768 → 1536: Noticeable improvement in retrieval precision for semantically similar memories (e.g., two facts that are related but different). Storage cost doubles. Recommended if recall quality is a problem.
• 768 → 3072: Maximum quality. Worth it only if you have a large corpus and are seeing meaningful retrieval failures at 1536.
• Latency: Cosine similarity search in ChromaDB scales roughly linearly with dimensions. At thousands of records this is not a bottleneck; at millions it becomes one.

---

How to do it safely

Since MCP-1A is extracting a shared runtime and already fixing the chroma_path injection pattern, that's the right time to also make EMBEDDING_DIMENSIONS an injectable parameter rather than a global — so the migration can spin up a new runtime at 1536 alongside the old one at 768, re-embed, verify, then switch over.

A rough migration plan when the time comes:

1. Set EMBEDDING_DIMENSIONS=1536 (or via env var once that's wired)
2. Create a new ChromaDB path (e.g. ./chroma_db_1536)
3. Read all records from the old collections, re-embed at the new dimension, write to the new collections
4. Swap CHROMA_DB_PATH to point at the new store
5. Discard the old store

The refactor in MCP-1A (passing chroma_path into constructors instead of mutating globals) will make step 4 safe to do without a restart.

---

Bottom line

• Can you increase to 1536 in production? Yes. The model supports it natively.
• Should you? Only if you're seeing retrieval quality issues. 768 is solid for most agent memory workloads.
• How hard is it? A one-time re-embedding migration. Not a live config change — you need to rebuild the ChromaDB collections.
• The comment in config.py is correct: 1536 is the natural next step, and the architecture will support it cleanly once MCP-1A lands.