Skip to content

multi-turn-persistent-memory.md

Latest commit

 

History

History
401 lines (293 loc) · 15.7 KB

multi-turn-persistent-memory.md

File metadata and controls

401 lines (293 loc) · 15.7 KB

MassGen v0.1.5: Persistent Memory with Semantic Retrieval

MassGen is focused on case-driven development. This case study demonstrates the introduction of persistent memory with semantic retrieval, enabling agents to build cumulative knowledge across multi-turn sessions and achieve true self-evolution through research-to-implementation workflows.

Table of Contents

📋 PLANNING PHASE

📝 Evaluation Design

Prompt

Two-turn research-to-implementation workflow:

Turn 1 (Research):

Use crawl4ai to research the latest multi-agent AI papers and techniques from 2025.
Focus on: coordination mechanisms, voting strategies, tool-use patterns, and architectural innovations.

Turn 2 (Implementation):

Based on the multi-agent research from earlier, which techniques should we implement in MassGen
to make it more state-of-the-art? Consider MassGen's current architecture and what would be most impactful.

This prompt tests whether agents can:

  1. Research external sources and store findings
  2. Retrieve relevant research in a follow-up turn
  3. Apply research to self-improvement recommendations

Baseline Config

Multi-turn conversations were already supported in MassGen, but without persistent memory. Turn 2 only had access to:

  • The full conversation history from Turn 1 (in context window)
  • Any workspace files created during Turn 1

Limitation: No semantic search, no fact extraction, no persistent knowledge base across sessions.

🔧 Evaluation Analysis

Results & Failure Modes

Before Persistent Memory (multi-turn only):

✅ What worked:

  • Agents could have multi-turn conversations
  • Turn 2 could reference Turn 1's full output in context
  • Workspace files persisted across turns

❌ What was missing:

  1. No semantic augmentation: Turn 2 had Turn 1's answer but no additional extracted facts
  2. No structured knowledge: Research stored only as raw conversation text
  3. Generic recommendations: Without structured facts, recommendations lacked specificity

Baseline Turn 2 Result (session log_20251029_064846, 132 lines):

  • Generic architectural proposals: "add massgen/voting.py"
  • Theoretical recommendations: "pluggable voting", "layered memory"
  • Less grounded in current codebase structure
  • More abstract: "CoordinationStrategy interface and implementations"

Success Criteria

Persistent memory should enable:

  1. Automatic Fact Extraction: System extracts structured facts from Turn 1 research
  2. Semantic Augmentation: Turn 2 gets Turn 1's answer PLUS relevant extracted facts automatically
  3. Persistent Storage: Facts stored in vector database
  4. More Specific Recommendations: Turn 2 provides concrete file paths, implementation steps, grounded in both research and current architecture

🎯 Desired Features

To enable self-evolution through research-to-implementation:

  1. Fact Extraction: Automatically extract important facts from conversations
  2. Vector Storage: Store facts with embeddings in persistent vector database
  3. Semantic Retrieval: Automatically retrieve relevant facts based on context
  4. Cross-Turn Continuity: Facts from Turn 1 available in Turn 2
  5. Quality Extraction: Custom prompts to ensure useful, self-contained facts
  6. Multi-Agent Support: Concurrent fact storage from multiple agents

🚀 TESTING PHASE

📦 Implementation Details

Version

v0.1.5 - Introduction of persistent memory system

✨ New Features

  1. PersistentMemory Integration (massgen/memory/_persistent.py)

    • Wraps mem0's AsyncMemory with MassGen-specific logic
    • Automatic fact extraction on turn completion
    • Semantic retrieval via vector search
    • Metadata tracking (session_id, agent_id, turn number)

  2. Custom Fact Extraction Prompts (massgen/memory/_fact_extraction_prompts.py)

    • MASSGEN_UNIVERSAL_FACT_EXTRACTION_PROMPT designed for quality facts
    • Intended to filter out: agent comparisons, voting details, file paths, system internals
    • Focuses on: domain knowledge, insights, capabilities, recommendations
    • Enforces self-contained facts (understandable without original context)

  3. Qdrant Vector Store Integration

    • Server mode support for multi-agent concurrency
    • Vector similarity search with metadata filtering
    • Persistent storage across sessions

  4. Memory Configuration YAML

    • memory.persistent_memory section for mem0 configuration
    • LLM and embedding model settings
    • Qdrant connection parameters
    • Retrieval and compression settings

  5. Automatic Recording & Retrieval (in ChatAgent)

    • Records facts after each turn completion
    • Retrieves relevant facts when context window approaches limit
    • Injects as system message: "Relevant memories: ..."

New Config

massgen/configs/memory/gpt5mini_gemini_research_to_implementation.yaml

Key Memory Settings:

memory:
  enabled: true

  persistent_memory:
    enabled: true  # 🆕 NEW: Persistent memory
    session_name: "research_to_implementation"  # Cross-turn continuity
    vector_store: "qdrant"

    llm:
      provider: "openai"
      model: "gpt-4.1-nano-2025-04-14"  # Fact extraction

    embedding:
      provider: "openai"
      model: "text-embedding-3-small"  # Vector embeddings

    qdrant:
      mode: "server"
      host: "localhost"
      port: 6333

  retrieval:
    limit: 10  # Number of facts to retrieve

Command

Prerequisites:

# Start Qdrant server
docker run -d -p 6333:6333 -p 6334:6334 \
  -v $(pwd)/.massgen/qdrant_storage:/qdrant/storage:z \
  qdrant/qdrant

# Start crawl4ai (for web scraping)
docker run -d -p 11235:11235 --name crawl4ai \
  --shm-size=1g unclecode/crawl4ai:latest

Run Session:

uv run massgen --config @examples/memory/gpt5mini_gemini_research_to_implementation.yaml

Turn 1 Prompt:

Use crawl4ai to research the latest multi-agent AI papers and techniques from 2025.
Focus on: coordination mechanisms, voting strategies, tool-use patterns, and architectural innovations.

Turn 2 Prompt (in same session):

Based on the multi-agent research from earlier, which techniques should we implement in MassGen
to make it more state-of-the-art? Consider MassGen's current architecture and what would be most impactful.

🤖 Agents

  • Agent A: gpt-5-mini (with crawl4ai tools)
  • Agent B: gemini-2.5-flash (with crawl4ai tools)

Session: session_20251029_072105 Duration: 11 minutes across 2 turns Memory Stats:

  • Facts stored (Turn 1): 54
  • Facts retrieved (Turn 2): 10

🎥 Demo

Watch the recorded demo:

MassGen Case Study

📊 EVALUATION & ANALYSIS

Results

Persistent memory dramatically improved Turn 2's ability to provide specific, actionable recommendations by retrieving relevant research findings from Turn 1.

The Collaborative Process

Turn 1 - Research Phase (5 minutes):

  1. Agents used crawl4ai to scrape arXiv
  2. Retrieved 20+ papers on multi-agent systems from late 2025
  3. Analyzed coordination mechanisms, voting strategies, tool patterns, architectures
  4. Generated comprehensive research summary (~133 lines)
  5. 🆕 Memory recorded 54 facts automatically

Example facts stored:

"Multi-layer memory folding that includes short-term windows, episodic timelines, and semantic summaries allows agents to manage large contexts efficiently, reducing token usage while maintaining factual recall, which is crucial for long-horizon tasks and fine-tuning."

"In 2025, multi-agent and agentic-AI systems evolved from ad-hoc multi-LLM setups to using structured workflows including hierarchical planning, task graphs, and planner-executor separations, which improve coherence, scalability, and fault tolerance."

Turn 2 - Implementation Phase (6 minutes):

  1. Agents have Turn 1's full answer in context (standard multi-turn)
  2. 🆕 System automatically retrieves 10 relevant facts from Turn 1 via semantic search
  3. 🆕 Facts injected as system message: "Relevant memories: ..."
  4. Read MassGen codebase (massgen/ and docs/ directories)
  5. Cross-referenced Turn 1 answer + retrieved facts + current architecture
  6. Generated prioritized implementation plan (~110 lines)

Example automatic memory retrieval:

When Turn 2 starts, system automatically searches memories and injects relevant facts:

Retrieved fact:

"Using argumentation frameworks with evidence scoring, proficiency or reputation-weighted voting, multi-stage consensus, and human-in-the-loop arbitration are advanced voting strategies in 2025..."

This fact (from Turn 1 research) was automatically added to Turn 2's context, enabling:

"1) Evidence‑aware, proficiency‑weighted voting + Judge (High impact, low→medium effort)

  • Replace naive majority with weighted aggregation using per‑agent proficiency scores plus evidence strength..."

The Voting Pattern

No changes to voting in this release - standard MassGen voting applied. The improvement came from what agents could reference during answer generation, not how they voted.

The Final Answer

Turn 2 Quality Comparison (Both sessions have Turn 1 answer in context):

Without Persistent Memory (log_20251029_064846, 132 lines):

  • Generic architectural proposals: "add massgen/voting.py (or massgen/voting/ package)"
  • Theoretical interfaces: "CoordinationStrategy", "Aggregator base"
  • Broad phases: "Phase 0 (1-2 sprints)", "Phase 1 (2-6 weeks)"
  • Less grounded: Treats implementation like greenfield architecture

Sample from baseline:

1) Pluggable Voting & Aggregation + Adaptive Early Stopping
- Where to change: add massgen/voting.py (or massgen/voting/ package)
- Suggested API / design sketch:
  - Aggregator (base)
    - add_vote(agent_id, result, confidence, trajectory, metadata)

With Persistent Memory (log_20251029_072105, 110 lines):

  • ✅ Specific existing file paths: workflow_toolkits/vote.py, coordination_tracker.py, orchestrator.py
  • ✅ Concrete implementation steps: Numbered steps for each feature
  • ✅ Test metrics: Specific KPIs for measuring success
  • ✅ Sprint planning: Concrete deliverables per sprint
  • ✅ Grounded in current architecture: References actual MassGen files

Sample from memory-enabled:

Top recommendations (what + where to change in repo)

1) Evidence‑aware, proficiency‑weighted voting + Judge (High impact, low→medium effort)
Where to implement (explicit paths):
  - workflow_toolkits/vote.py — extend to accept evidence payloads and compute weighted scores
  - message_templates.py — add evidence schema to agent message format
  - coordination_tracker.py — track per‑agent proficiency/calibration
  - orchestrator.py — surface evidence into coordinator logs and call Judge

Concrete implementation steps:
  1. Extend message template with evidence: {claims:[...], tool_outputs:[...], confidence:float}
  2. Implement per‑agent scoreboard (moving average success) in coordination_tracker.py
  3. Update vote.py: compute final_score = α*proficiency + β*evidence_score + γ*vote_strength
  4. Create Judge agent that can (a) fetch supporting sources, (b) re-run tool calls...

Key Difference:

The memory-enabled version provides:

  • Actual file paths that exist (workflow_toolkits/vote.py vs. "add massgen/voting.py")
  • Numbered implementation steps
  • Specific integration points
  • Grounded in both research AND current codebase

This specificity comes from:

  1. Turn 1 research stored as structured facts (automatic)
  2. Turn 2 has Turn 1 answer PLUS 10 relevant facts (automatic semantic retrieval)
  3. Facts provide additional semantic context beyond raw conversation history
  4. Agents combine: Turn 1 answer + extracted facts + codebase analysis = concrete actionable plan

Memory System Performance

Memory example:

"Coordination mechanisms that improve long-term coherence include hierarchical recursive planning, task decomposition with DAG structures, and planner-executor systems that maintain shared memory and intermediate artifacts."

Retrieval Performance:

  • Turn 2 context triggers automatic semantic search
  • System found 10 most relevant facts from 54 stored
  • Latency: < 100ms
  • Facts augment Turn 1 answer already in context

Cost Analysis:

  • Fact extraction: gpt-4.1-nano @ $0.15/M tokens
  • Embeddings: text-embedding-3-small @ $0.020/M tokens
  • 54 facts extracted + embedded: < $0.001
  • Storage: ~108 KB (54 × 2KB per fact)

🎯 Conclusion

Why Persistent Memory Improves Self-Evolution

Before (multi-turn with conversation history):

  • Turn 2 had Turn 1's full answer in context ✓
  • But: No additional semantic augmentation
  • Result: Generic architectural proposals (132 lines, mostly abstract interfaces)

After (persistent memory + conversation history):

  • Turn 2 has Turn 1's answer PLUS 10 automatically retrieved facts
  • Facts provide additional semantic context extracted from Turn 1
  • System automatically searches and injects relevant memories
  • Result: Specific file paths, numbered steps, grounded in actual architecture (110 lines, more concrete)

The Compound Effect

Within this session, memory enabled:

  • Turn 1: Research 20+ papers → Extract and store 54 structured facts
  • Turn 2: Automatically retrieve 10 relevant facts → More specific recommendations

The architecture supports future cross-session retrieval, though not demonstrated in this case study.

Broader Implications

Persistent memory enables:

  1. Self-Evolution: Agents can learn about themselves through research-to-implementation
  2. Research-to-Implementation: Bridge external research to internal development
  3. Semantic Augmentation: Additional structured facts supplement conversation history
  4. Knowledge Storage: Facts persist in vector database for future retrieval
  5. Improved Specificity: Extracted facts lead to more concrete, actionable recommendations

Future Improvements

Memory Quality (Current: 72% good, 28% system internals):

The custom fact extraction prompts significantly improve memory quality, but ~28% of stored facts are still system internals (voting details, agent comparisons, meta-instructions). Planned improvements:

  • Stricter pattern matching for procedural language
  • Two-pass extraction (extract, then validate against exclusion rules)
  • Domain-specific prompts for research vs. implementation tasks
  • Active learning from user feedback on memory relevance

Cross-Session Loading:

  • Load facts from previous sessions by session_name
  • Session management UI
  • Memory pruning and maintenance

Retrieval Intelligence:

  • Multi-query retrieval (expand queries to multiple search vectors)
  • Temporal weighting (recent facts ranked higher)
  • Cross-session memory fusion (merge related facts)

📌 Status Tracker

  • Planning phase completed
  • Features implemented
  • Testing completed
  • Demo recorded
  • Results analyzed
  • Case study reviewed