Feature Request: Embedding-Based Memory Search for Persistent Agent Learning

[henrikrexed]

Problem Statement

Currently, SympoziumInstance memory uses a ConfigMap with a flat MEMORY.md file (max 256KB). While this provides basic persistence between agent runs, it has significant limitations:

1. No search capability — Agents must read the entire MEMORY.md at the start of each run, consuming context window tokens regardless of relevance
2. Doesn't scale — As memory grows, the full file becomes too large for the context window
3. No semantic retrieval — Agents can't find past investigations based on similarity to the current problem
4. Each run starts from scratch — Even if the agent diagnosed a nearly identical issue 2 weeks ago, it has no efficient way to find and reuse that knowledge

Use Case: Learning from Past Investigations

In SRE troubleshooting, many issues are recurring or similar. An agent that investigated a Kafka queue problem last week should be able to:

1. Receive a new alert about Kafka consumer lag
2. Search memory for past Kafka-related investigations
3. Find the previous root cause analysis and resolution steps
4. Skip the dead ends from the first investigation
5. Resolve the issue faster

Without semantic search, the agent either:

• Reads the entire memory (expensive, hits context limits)
• Starts from zero every time (wasteful, slower)

This is especially critical for Sympozium because agents are ephemeral (pod-per-run) — there's no conversation history to fall back on. Memory is the only continuity mechanism.

Proposed Solution

Option A: Built-in Embedding Support (Recommended)

Add an optional vector search layer to the existing memory system:

apiVersion: sympozium.ai/v1alpha1
kind: SympoziumInstance
spec:
  memory:
    enabled: true
    maxSizeKB: 1024
    search:
      enabled: true
      provider: ollama                    # or "openai"
      embeddingModel: nomic-embed-text    # lightweight, runs on Ollama
      baseUrl: http://ollama:11434        # embedding model endpoint
      vectorStore: persistent-volume      # or "qdrant", "chromadb"
      chunkSize: 512                      # tokens per chunk
      topK: 5                             # results per search
    systemPrompt: |
      You have access to a search_memory tool.
      Before starting any investigation, search memory for similar past issues.
      After completing an investigation, store key findings for future reference.

How it works:

1. Controller watches MEMORY.md ConfigMap for changes
2. On update: chunks the text → generates embeddings via configured provider → stores vectors
3. Agent pods get a search_memory tool injected (similar to how MCP tools are mounted)
4. Agent searches before investigating, writes findings after

Option B: MCP-Based Memory Server

A dedicated MCP server that handles memory storage and retrieval:

spec:
  mcpServers:
    - name: memory
      toolsPrefix: memory
      url: http://memory-server.sympozium-system:8080

The memory MCP server would expose:

• memory_search(query, topK) — semantic search over past entries
• memory_store(content, tags) — store new findings
• memory_list(tags, limit) — list recent entries

Option C: SkillPack Approach

A reusable SkillPack that any instance can mount:

spec:
  skills:
    - skillPackRef: memory-search
      params:
        EMBEDDING_MODEL: nomic-embed-text
        OLLAMA_URL: http://ollama:11434
        STORAGE: /data/memory-vectors

Architecture Considerations

Embedding Model Options

Model	Size	Speed	Quality	Where
nomic-embed-text	274MB	Fast	Good	Ollama (local)
mxbai-embed-large	670MB	Medium	Better	Ollama (local)
text-embedding-3-small	API	Fastest	Good	OpenAI (cloud)

For air-gapped / local-model users, Ollama-based embeddings are essential — this aligns with Sympozium's strength of working with local models.

Vector Storage Options

1. PersistentVolume with embedded DB (simplest) — Use something like SQLite with sqlite-vss or hnswlib bundled into the controller
2. In-cluster Qdrant/ChromaDB — More scalable, but adds infrastructure
3. ConfigMap-based (current pattern extended) — Store serialized vectors in a ConfigMap; simplest but limited by ConfigMap size (1MB)

Data Flow

Agent Run Completes
      ↓
Controller detects MEMORY.md update
      ↓
Chunk text → Generate embeddings → Store vectors
      ↓
Next Agent Run starts
      ↓
Agent calls search_memory("kafka consumer lag")
      ↓
Vector similarity search → Top-K results returned
      ↓
Agent uses past findings to accelerate investigation

Benchmark Evidence

From our AI Agent Benchmark comparing kagent, Sympozium, and HolmesGPT across 13 scenarios:

• Agents frequently encounter similar failure patterns across scenarios (e.g., multiple scenarios involve feature flag misconfigurations, Kafka issues, or pod crashloops)
• An agent with memory search could have reused diagnostic steps from scenario 1 (basic crashloop) when encountering scenario 7 (complex crashloop with timeout)
• Token consumption could be reduced by 30-50% on recurring patterns if the agent finds relevant past context instead of re-discovering it

Prior Art

• OpenClaw uses MEMORY.md + daily notes with Voyage AI embeddings for semantic search across memory files. Agents call memory_search(query) before answering questions about prior work.
• LangChain/LangGraph have built-in memory stores with vector retrieval
• CrewAI supports long-term memory with embedding-based search

Summary

Approach	Complexity	Scalability	Local Model Support
A: Built-in	Medium	High	✅ Ollama embeddings
B: MCP Server	Medium	High	✅ Any embedding API
C: SkillPack	Low	Medium	✅ Ollama embeddings

[m13v]

This is a well-scoped problem. We built something similar for extracting and searching user knowledge from browser data, and a few implementation details might be useful.

For the embedding model choice, nomic-embed-text via Ollama is a solid default. We tested it against OpenAI ada-002 and the retrieval quality was within ~5% for our use case, while being completely local. The tradeoff is that initial indexing is slower on CPU, but for SRE memory sizes (thousands of entries, not millions) it's fine.

One thing we found critical was chunking strategy. Naive fixed-size chunks (512 tokens) lose context at boundaries. We switched to a paragraph-aware chunker that respects section headers and code blocks, which improved retrieval relevance noticeably for technical content like investigation notes.

For the ConfigMap-based storage, you might want to consider SQLite with FTS5 instead of a separate vector DB. It keeps the dependency footprint small and you get keyword + semantic search in one query. We store the embedding alongside the text in the same table and do cosine similarity via sqlite-vec extension.

The ephemeral pod constraint is interesting. We handle a similar problem by writing the SQLite DB to a persistent volume and loading it at pod start. The index stays warm across runs.

[m13v]

here's our embedding + search implementation if it helps as a reference: https://github.com/m13v/ai-browser-profile/blob/main/ai_browser_profile/embeddings.py

handles the chunking, embedding generation, and hybrid retrieval (semantic + keyword) in one module. the db layer with the SQLite schema is here: https://github.com/m13v/ai-browser-profile/blob/main/ai_browser_profile/db.py

[henrikrexed]

Thanks @m13v . this is incredibly useful, especially the implementation details around chunking strategy and SQLite + cosine similarity. Your experience validates the approach i proposed.

A few observations from looking at your code:

1.⁠ ⁠The ONNX-based embedding approach (⁠ nomic-embed-text-v1.5 ⁠ via ONNX Runtime) is exactly what is needed for Sympozium. No Ollama dependency, no PyTorch , just a self-contained ~181MB model that works in any K8s pod. This is much lighter than what i originally proposed.

2.⁠ ⁠SQLite with BLOBs for vectors is the right call for our scale. SRE investigation memory is thousands of entries, not millions. No need for Qdrant or ChromaDB.

3.⁠ ⁠The semantic dedup (cosine ≥ 0.92 → supersede) is critical for the use case -> agents investigating similar Kafka issues shouldn't create 5 nearly-identical memory entries.

4.⁠ ⁠Paragraph-aware chunking is something I did not considered but it's essential . Our memory entries contain kubectl output, YAML configs, and stack traces that would lose context with naive 512-token splits.

I would love to implement this as a Sympozium SkillPack (or MCP sidecar) . Essentially adapting ⁠ embeddings.py ⁠ + SQLite schema for K8s-native agent memory. The key difference is that Sympozium agents are ephemeral pods, so the SQLite DB would live on a PersistentVolume.

Would you be open to let me use your ⁠ embeddings.py ⁠ module as a starting point? I would like to adapt the schema from browser knowledge to SRE investigation memory (keys like ⁠ root_cause ⁠, ⁠ resolution ⁠, ⁠ service ⁠, ⁠ namespace ⁠ instead of ⁠ email ⁠, ⁠ phone ⁠, etc.) but the core embedding + search engine would be the same.

Also curious , have you run into any concurrency issues with SQLite when multiple processes read/write? In Sympozium, agent pods are sequential per instance (one at a time), but the controller might also need to write during embedding generation.

[AlexsJones]

I just shipped persistent memory for Sympozium agents (#45). Each SympoziumInstance now gets its own memory server backed by SQLite with FTS5 full-text search, and agents talk to it over HTTP using three simple tools: memory_search, memory_store, and memory_list. This replaces the old ConfigMap-basedapproach and gives agents real recall across runs while keeping the door open for vector search down the road.

[m13v]

glad the ONNX approach clicks for your use case. 181MB self-contained is way better than pulling in the full PyTorch stack for inference. one thing to watch with nomic-embed-text in ONNX - make sure you're using the correct prefix tokens (search_query: vs search_document:) since the model was trained with those and skipping them drops retrieval quality significantly. happy to share our embedding pipeline config if that'd be useful.

[AlexsJones]

Thanks for your input and guidance on this @m13v , our work isn't done here at all.

[m13v]

That's awesome, congrats on shipping the persistent memory! The checkpoint-based approach is solid for agents that need to resume across sessions. If you run into issues with memory retrieval scaling as the context grows, semantic search over the stored checkpoints (embedding + cosine similarity) can help surface only the relevant memories rather than loading everything.

Looking forward to seeing how the agent behavior evolves with persistent state.

[henrikrexed]

That was fast @AlexsJones! SQLite + FTS5 with HTTP tools is the right first step , agents get real recall today, and the foundation is ready for vector search when needed.

A few questions on the implementation:

1.⁠ ⁠Deployment model — Is the memory server a sidecar per agent pod, or a shared Deployment per SympoziumInstance? Sidecar keeps it simple (no network hop) but a shared service means memory persists even when no agent is running.

2.⁠ ⁠Migration path — Does the controller seed the new memory server with existing MEMORY.md ConfigMap content? Smooth migration for existing users would help adoption.

3.⁠ ⁠Memory-aware system prompts — Are agents automatically instructed to use ⁠ memory_search ⁠ before investigating and ⁠ memory_store ⁠ after completing? Or is that left to the system prompt?

4.⁠ ⁠Observability — Are the memory HTTP calls traced with OTel? Having ⁠ memory_search ⁠ and ⁠ memory_store ⁠ as spans in the agent trace would let us measure hit rates and search latency.

For the vector search next step — @m13v's ONNX approach (⁠ nomic-embed-text ⁠, ~181MB, no Ollama dependency) would drop right into the same SQLite DB alongside FTS5 for hybrid search. The ⁠ search_query: ⁠ / ⁠ search_document: ⁠ prefix tokens are critical for retrieval quality with nomic.

I'd love to test this I can come up to few scenarios that involves the usage of memory to measure the improvements.

[m13v]

Great questions. From our experience:

1. Sidecar vs shared - we started with sidecar for simplicity but moved to a shared Deployment pretty quickly. The persistence argument is real, but the bigger win is that agents running in different pods can query each other's memories. If your agents are mostly independent, sidecar works. If they need shared recall across a conversation or workflow, go shared.

2. Migration path - seeding from the controller is cleaner. We had the agent bootstrap its own memory store on first run and it created a race condition when multiple agents spun up simultaneously. Having the controller provision the SQLite DB (or at least the schema) before the agent starts avoids that.

One thing to watch with FTS5 in a shared setup - concurrent writes from multiple agents need WAL mode enabled or you'll hit SQLITE_BUSY under load.

[henrikrexed]

Really useful insights @m13v, especially the sidecar → shared migration. That maps perfectly to Sympozium's upcoming PersonaPack orchestration (#47) where a lead agent and multiple worker agents would need shared recall within the same workflow.

The controller-provisioned DB makes total sense — @AlexsJones since you already have the controller managing the memory server lifecycle, provisioning the schema before the first agent pod starts should be straightforward.

The WAL mode tip is critical. For the Sympozium case, agents are sequential per instance (one at a time), but with PersonaPack orchestration a lead + parallel workers could hit the DB concurrently. WAL should be the default.

One question :Did you run into any size issues with the SQLite DB over time? For SRE memory, we'd potentially accumulate thousands of investigation entries. Did you need to implement any pruning/eviction strategy, or did SQLite handle the scale fine?

[m13v]

Yeah, the controller-provisioned DB pattern works really well for that kind of orchestration. One thing we learned the hard way - make sure you provision the schema and run any migrations before the first agent pod starts. We had a race condition where two agents tried to auto-migrate simultaneously and corrupted the WAL.

For PersonaPack with shared recall across a lead + workers, you might also want to consider read replicas if the worker count gets high. SQLite WAL mode handles concurrent reads well, but we saw contention above ~8 simultaneous readers doing embedding lookups. A simple connection pool with a read timeout solved it for us.