feat(memory): agent memory v2 — second brain with hybrid search, LLM extraction, and observability dashboard

[kovtcharov]

Summary

Comprehensive agent memory system that serves as a second brain — storing, recalling, and learning from every interaction. Built on proven patterns from Mem0, Zep, and Hindsight.

Architecture (v2)

• Hybrid search: Vector (FAISS) + BM25 (FTS5) + RRF fusion + cross-encoder reranking (ms-marco-MiniLM-L-6-v2)
• Mem0-style extraction: LLM decides ADD/UPDATE/DELETE/NOOP against existing memory after each conversation turn
• Zep-inspired fact lineage: superseded_by column preserves history when facts are corrected
• Hindsight-inspired reconciliation: Background pairwise similarity check detects contradictions across sessions
• Complexity-aware recall: Adaptive top_k (3/5/10) based on query complexity heuristics
• Temporal search: time_from/time_to on all search methods for time-based recall
• Conversation consolidation: Auto-distill old sessions into durable knowledge before 90-day prune
• No silent fallback: Embeddings are a hard requirement — system fails loudly on misconfiguration

Schema v2

Three tables (knowledge, conversations, tool_history) with new columns:

• knowledge.embedding BLOB — 768-dim vector (nomic-embed-text-v2)
• knowledge.superseded_by TEXT — fact lineage chain
• conversations.consolidated_at TEXT — consolidation tracking

Memory Tools (5 LLM-facing tools)

Tool	Purpose
remember	Store facts, notes, reminders with category/domain/entity/due_at
recall	Hybrid semantic+keyword search with temporal filtering
update_memory	Modify existing items, set reminded_at
forget	Delete a memory item
search_past_conversations	Search conversation history with temporal filtering

Use Cases

• Note-taking, journaling, meeting notes capture
• Reminders with due dates and wake-up scheduling
• Personal knowledge management (research, articles)
• Contact profiles via entity linking (person:sarah_chen)
• Error learning and skill capture from tool usage
• Recurring commitments (LLM advances due_at)

Observability Dashboard

Full-page Memory Dashboard in Agent UI with:

• Header stats cards (memories, sessions, tool calls, success rate)
• Activity timeline (30-day heatmap)
• Knowledge browser (filterable, sortable, paginated table)
• Tool performance stats
• Conversation history browser with consolidation status
• Upcoming/overdue reminders panel
• Maintenance actions (consolidate, rebuild embeddings, reconcile)
• Embedding coverage indicator

Startup Sequence

1. Validate Lemonade → 2. Backfill embeddings → 3. Rebuild FAISS → 4. Confidence decay → 5. Reconcile memory → 6. Consolidate sessions → 7. Prune → 8. Generate session

Files

Component	Files
Data layer	src/gaia/agents/base/memory_store.py
Agent mixin	src/gaia/agents/base/memory.py
System discovery	src/gaia/agents/base/discovery.py
REST API	src/gaia/ui/routers/memory.py
Agent UI	src/gaia/apps/webui/src/pages/MemoryDashboard.tsx
Architecture spec	docs/spec/agent-memory-architecture.md
Unit tests	tests/unit/test_memory_*.py
Integration tests	tests/integration/test_memory_*.py

Design References

System	Pattern adopted
Mem0	LLM-in-the-loop extraction (ADD/UPDATE/DELETE/NOOP)
Zep/Graphiti	Fact lineage via superseded_by, temporal search
Hindsight	Cross-encoder reranking, background reconciliation
ENGRAM	Memory typing (category-based) over knowledge graphs
CoALA	Four-tier cognitive architecture (working/episodic/semantic/procedural)

Test plan

• Unit tests pass: pytest tests/unit/test_memory_store.py test_memory_mixin.py test_memory_router.py
• Integration tests pass: pytest tests/integration/test_memory_integration.py test_memory_api_integration.py
• Schema v2 migration works on existing v1 databases
• Hybrid search returns semantically relevant results (not just keyword matches)
• Mem0 extraction correctly handles ADD/UPDATE/DELETE operations
• Superseded items excluded from search and system prompt
• Temporal filtering works with time_from/time_to on recall
• Consolidation distills old sessions into knowledge items
• Reconciliation detects contradictory facts across sessions
• Memory Dashboard renders all 6 sections with real data
• Dashboard knowledge browser supports filter/sort/paginate/edit/delete
• Lemonade unavailable at startup raises RuntimeError (no silent fallback)
• Cross-encoder reranking improves precision on ambiguous queries
• Complexity-aware recall uses adaptive top_k (3/5/10)
• Frontend build succeeds: cd src/gaia/apps/webui && npm run build

[kovtcharov]

🛑 STOP-THE-LINE — coordination notice

This PR is foundational (38k+ LOC, 77 files) and currently CONFLICTING with main. Until this lands, no PRs may merge changes to the frozen paths below.

Frozen paths

• src/gaia/agents/base/agent.py
• src/gaia/agents/base/memory*.py (all memory-related files)
• src/gaia/agents/base/discovery.py
• src/gaia/ui/routers/memory.py
• src/gaia/apps/webui/src/pages/MemoryDashboard.tsx
• Anything in this PR's file diff (run gh pr diff 606 --name-only for the full list)

Why

Every parallel agent merging to main makes this PR staler and less mergeable. With ~10 parallel agents potentially producing PRs/day for v0.20.0 + v0.18.2, without coordination this PR becomes unmergeable and v0.20.0 (June 2 consumer launch) slips by 4+ weeks.

What's allowed

• PRs touching ONLY non-frozen file paths (most v0.18.2 mobile work, Telegram, scheduler, MCP catalogue)
• Documentation changes
• CI/test infrastructure
• Hotfixes to v0.17.5/v0.18.0

Active focus on landing this PR

Per issue A5: rebase + review acceleration. Target land date: May 5 (latest acceptable May 12). If slipping beyond May 12, escalate.

For coding agents

Read AGENTS.md (newly added) for the full rule + check command (gh pr list --label stop-the-line).

[github-actions]

Summary

Massive, well-structured PR (~41k LOC) introducing memory v2 with hybrid retrieval (FAISS + FTS5 + RRF + cross-encoder), Mem0-style LLM extraction, Zep-inspired lineage, an observability dashboard, 26 eval scenarios, and a credible suite of unit + integration tests. Architecture, schema migration, FTS5 sanitization, parameterized SQL, admin gating via GAIA_MEMORY_ADMIN, and the locking strategy (WAL + threading.Lock + busy_timeout=5000) all look sound.

The main concern is a direct contradiction between the PR description and the implementation: the description (and test plan) claim "No silent fallback — system fails loudly on misconfiguration" and "Lemonade unavailable at startup raises RuntimeError," but memory.py:init_memory actually swallows the failure and silently disables memory. Per CLAUDE.md "No Silent Fallbacks — Fail Loudly," this should be reconciled before merge.

Issues

🟡 Silent fallback contradicts PR description and CLAUDE.md

src/gaia/agents/base/memory.py:347-361 — when the Lemonade embedding probe fails, the code catches a broad Exception, logs a warning, tears state down, and returns. The session continues with memory disabled. The PR description explicitly promises the opposite ("No silent fallback") and the test plan has a checkbox for "Lemonade unavailable at startup raises RuntimeError (no silent fallback)" — that checkbox cannot pass against this code path.

This also conflicts with the project rule in CLAUDE.md: "Either the operation succeeds as intended, or it raises an actionable error."

Two acceptable resolutions:

Option A — actually fail loudly (matches PR description):

        except Exception as e:
            raise RuntimeError(
                "Lemonade embedding service unreachable — memory v2 cannot "
                "initialize. Start lemonade-server (e.g. `lemonade-server serve`) "
                "and ensure the embedding model is available, or set "
                "GAIA_MEMORY_DISABLED=1 to opt out. Reason: "
                f"{e}"
            ) from e

Option B — keep the degrade, but make it an explicit opt-in and update the PR description. If the intent is genuinely "graceful degrade for users without Lemonade," gate it behind an env flag (e.g. GAIA_MEMORY_DEGRADE_ON_NO_EMBEDDINGS=1) so it's an opt-in, not a default. Then either delete the "No silent fallback" claim from the PR body or qualify it ("fails loudly unless GAIA_MEMORY_DEGRADE_ON_NO_EMBEDDINGS=1").

Either way is fine — the current state is "ships with the docs and the code disagreeing."

🟡 Bare except Exception: pass in mixin prompt auto-discovery

src/gaia/agents/base/agent.py:313-318 — the auto-discovery loop in _get_mixin_prompts swallows any failure from a mixin's prompt-fragment method without even a debug log. If a mixin author writes a buggy _get_xxx_prompt it will silently produce no fragment, with no diagnostic. At minimum log at debug level so the failure is traceable when someone is debugging "why isn't my mixin prompt showing up?":

                try:
                    fragment = getattr(self, attr_name)()
                    if fragment:
                        prompts.append(fragment)
                except Exception as e:
                    logger.debug(
                        "[Agent] mixin prompt fragment %s.%s raised: %s",
                        type(self).__name__, attr_name, e,
                    )

🟢 QueueFull swallowed silently in agent loop trigger enqueue

src/gaia/ui/agent_loop.py:137-142 — the comment says "queue is unbounded; this should never happen" which is true today, but if anyone ever adds a maxsize= to _trigger_queue the trigger will be silently dropped. Cheap fix:

        try:
            self._trigger_queue.put_nowait(
                AgentTrigger("user_message_followup", session_id)
            )
        except asyncio.QueueFull:
            logger.warning(
                "AgentLoop trigger queue full; dropping user_message_followup for session %s",
                session_id,
            )

🟢 Repeated except Exception: pass (or near-equivalents) in discovery / system_context

src/gaia/agents/base/discovery.py and src/gaia/agents/base/system_context.py use broad try/except extensively. For best-effort OS/registry/browser-history scanning this is mostly appropriate — those code paths must not crash a user's session because Chrome's history DB is locked or winreg returns garbage. But several of those handlers don't log at all, which makes "why didn't system context populate field X?" undiagnosable. Per the same CLAUDE.md rule, prefer "log at debug + continue" over silent pass, so the failure is visible when someone runs with --debug. Not blocking, but worth a sweep.

🟢 Minor: PR test-plan checkbox is unverifiable

The "Lemonade unavailable at startup raises RuntimeError (no silent fallback)" checkbox is a useful contract test — please add it as an actual pytest case (tests/unit/test_memory_mixin.py) once the silent-fallback issue is resolved one way or the other, so the regression is locked in.

Strengths

• Test coverage is genuinely solid. 8 new unit-test files (test_memory_store.py, test_memory_mixin.py, test_memory_router.py, test_goal_store.py, test_goals_router.py, test_memory_discovery.py, test_sdk_tool_messages.py) plus 3 integration files. New tests/unit/conftest.py for shared fixtures. This is exemplary for a feature of this scope.
• 26 eval scenarios under eval/scenarios/memory/ covering cross-session persistence, conflict resolution, adversarial poisoning, capacity stress, retrieval-under-noise, etc. — exactly the right shape for a "second brain" feature.
• Documentation is comprehensive and lands in the right places: docs/guides/memory.mdx, docs/sdk/sdks/memory.mdx, docs/spec/agent-memory-architecture.md, plus docs/docs.json nav updates. Matches the CLAUDE.md "every new feature must be documented" rule.
• SQL hygiene is excellent. Parameterized queries throughout memory_store.py. _sanitize_fts5_query correctly strips FTS5 special chars and caps at MAX_FTS_QUERY_LENGTH=500. No injection vectors spotted.
• Schema migration is idempotent via ALTER TABLE ADD COLUMN with duplicate-column tolerance — clean upgrade path from v1 → v2.
• Concurrency is handled deliberately: threading.Lock + WAL + busy_timeout=5000 in memory_store.py; double-checked-locking singletons in routers/memory.py and routers/goals.py.
• Admin endpoints are properly gated by GAIA_MEMORY_ADMIN=1 (memory clear/seed in both REST and MCP surfaces). CI gating via GAIA_MEMORY_DISABLED=1 is a sensible kill-switch.
• MCP subprocess management is safe: subprocess.Popen([sys.executable, ...]) with no shell=True, terminate→kill timeout, in routers/mcp.py.
• Pydantic ISO 8601 validation factored into a shared _validate_iso8601 helper — no DRY violations on the API surface.
• Goal store is correctly isolated in ~/.gaia/goals.db with its own PRAGMA foreign_keys=ON and state machine, decoupled from memory storage.
• Pending-approval route ordering in routers/goals.py (literal path before {goal_id}) shows the author thought about FastAPI's path-priority semantics.
• Single source of truth for VALID_CATEGORIES prevents drift between the store, the mixin, and the API layer.

Verdict

Approve with suggestions. The architecture, tests, docs, and security posture are all well above the bar for a feature this size. The one item that should be addressed before merge is the silent-fallback in memory.py:347-361 — either make it actually raise (so the PR description and test plan are honest) or document the degrade as an explicit, env-gated opt-in. Everything else is polish.

cc @kovtcharov-amd — flagging the CLAUDE.md "no silent fallback" rule violation; not a security issue, but it's the kind of thing that bites later when an outage masks itself as "memory just stopped working." No 🔒 security concerns.