Skip to content

Phase 5 Integration

Jump to bottom
Rick Hightower edited this page Feb 1, 2026 · 1 revision

Phase 5: Integration

This page aggregates all Phase 5 documentation for the Integration phase.

Phase Overview

Hook handler connection, query CLI, and admin commands.

05-01-PLAN

Plan 05-01: Client Library and Hook Mapping

Goal

Create a client library that hook handlers can use to call IngestEvent RPC, with event type mapping from hook events to memory events.

Requirements Covered

  • HOOK-02: Hook handlers call daemon's IngestEvent RPC
  • HOOK-03: Event types map 1:1 from hook events

Tasks

Task 1: Create memory-client Crate

File: crates/memory-client/Cargo.toml

[package]
name = "memory-client"
version = "0.1.0"
edition = "2021"
description = "Client library for Agent Memory daemon"

[dependencies]
memory-service = { path = "../memory-service" }
memory-types = { path = "../memory-types" }
tonic = { workspace = true }
tokio = { workspace = true }
thiserror = { workspace = true }
tracing = { workspace = true }

File: crates/memory-client/src/lib.rs

//! Client library for Agent Memory daemon.
//!
//! Per HOOK-02: Hook handlers call daemon's IngestEvent RPC.

pub mod client;
pub mod error;
pub mod hook_mapping;

pub use client::MemoryClient;
pub use error::ClientError;
pub use hook_mapping::{HookEvent, HookEventType, map_hook_event};

Task 2: Implement MemoryClient

File: crates/memory-client/src/client.rs

use memory_service::proto::{
    memory_service_client::MemoryServiceClient,
    IngestEventRequest,
};
use memory_types::Event;
use tonic::transport::Channel;

use crate::error::ClientError;

pub struct MemoryClient {
    inner: MemoryServiceClient<Channel>,
}

impl MemoryClient {
    pub async fn connect(endpoint: &str) -> Result<Self, ClientError> {
        let inner = MemoryServiceClient::connect(endpoint.to_string())
            .await
            .map_err(ClientError::Connection)?;
        Ok(Self { inner })
    }

    pub async fn ingest(&mut self, event: Event) -> Result<String, ClientError> {
        // Convert event to proto and call RPC
    }
}

Task 3: Implement Hook Event Mapping

File: crates/memory-client/src/hook_mapping.rs

use memory_types::{Event, EventType, EventRole};

#[derive(Debug, Clone)]
pub enum HookEventType {
    SessionStart,
    UserPromptSubmit,
    AssistantResponse,
    ToolUse,
    Stop,
}

#[derive(Debug, Clone)]
pub struct HookEvent {
    pub session_id: String,
    pub event_type: HookEventType,
    pub content: String,
    pub timestamp: Option<i64>,
    pub tool_name: Option<String>,
}

pub fn map_hook_event(hook: HookEvent) -> Event {
    let event_type = match hook.event_type {
        HookEventType::SessionStart => EventType::SessionStart,
        HookEventType::UserPromptSubmit => EventType::UserMessage,
        HookEventType::AssistantResponse => EventType::AssistantMessage,
        HookEventType::ToolUse => EventType::ToolUse,
        HookEventType::Stop => EventType::SessionEnd,
    };

    let role = match hook.event_type {
        HookEventType::UserPromptSubmit => EventRole::User,
        HookEventType::AssistantResponse => EventRole::Assistant,
        HookEventType::ToolUse => EventRole::System,
        _ => EventRole::System,
    };

    Event::new(hook.session_id, event_type, role, hook.content)
        .with_timestamp(hook.timestamp.map(|ts|
            chrono::DateTime::from_timestamp_millis(ts).unwrap_or_default()
        ).unwrap_or_else(chrono::Utc::now))
}

Task 4: Implement ClientError

File: crates/memory-client/src/error.rs

use thiserror::Error;

#[derive(Error, Debug)]
pub enum ClientError {
    #[error("Connection failed: {0}")]
    Connection(#[from] tonic::transport::Error),

    #[error("RPC failed: {0}")]
    Rpc(#[from] tonic::Status),

    #[error("Serialization failed: {0}")]
    Serialization(String),
}

Task 5: Update Workspace Cargo.toml

Add memory-client to workspace members.

Task 6: Add Tests

Test client connection (mock server), event mapping.

Acceptance Criteria

  • memory-client crate compiles
  • MemoryClient can connect to daemon endpoint
  • ingest() method sends events via gRPC
  • HookEvent maps to Event with correct types
  • Tests pass for event mapping

Dependencies

  • Phase 4 complete (IngestEvent RPC exists)

Plan created: 2026-01-30

05-01-SUMMARY

Phase 05-01 Summary: Client Library and Hook Mapping

Completed Tasks

Task 1: Created memory-client Crate

  • New crate at crates/memory-client/
  • Added to workspace members and dependencies
  • Dependencies: memory-service, memory-types, tonic, tokio, thiserror, tracing, chrono, ulid

Task 2: Implemented MemoryClient

  • MemoryClient::connect(endpoint) - Connect to daemon
  • MemoryClient::connect_default() - Connect to default endpoint
  • MemoryClient::ingest(event) - Ingest single event via gRPC
  • MemoryClient::ingest_batch(events) - Ingest multiple events
  • Type conversion from domain Event to proto Event

Task 3: Implemented Hook Event Mapping

  • HookEventType enum with variants: SessionStart, UserPromptSubmit, AssistantResponse, ToolUse, ToolResult, Stop, SubagentStart, SubagentStop
  • HookEvent struct with builder pattern methods
  • map_hook_event(hook) function maps to domain Event

Task 4: Implemented ClientError

  • Connection errors (tonic transport)
  • RPC errors (tonic status)
  • Serialization errors
  • Invalid endpoint errors

Key Artifacts

File Purpose
crates/memory-client/Cargo.toml Crate manifest
crates/memory-client/src/lib.rs Module exports
crates/memory-client/src/client.rs MemoryClient implementation
crates/memory-client/src/error.rs Error types
crates/memory-client/src/hook_mapping.rs Hook event mapping

Verification

  • cargo build --workspace compiles
  • cargo test --workspace passes (107 tests)
  • 11 new tests for memory-client

Requirements Coverage

  • HOOK-02: Hook handlers can call daemon's IngestEvent RPC via MemoryClient
  • HOOK-03: Event types map 1:1 from hook events via map_hook_event()

Completed: 2026-01-30

05-02-PLAN

Plan 05-02: Query CLI

Goal

Add query commands to the CLI for manual TOC navigation and testing.

Requirements Covered

  • CLI-02: Query CLI for manual TOC navigation and testing

Tasks

Task 1: Add Query Subcommand to CLI

File: crates/memory-daemon/src/cli.rs

Add Query variant to Commands enum with nested subcommands:

#[derive(Subcommand, Debug)]
pub enum Commands {
    // existing...

    /// Query the memory system
    Query {
        #[command(subcommand)]
        command: QueryCommands,
    },
}

#[derive(Subcommand, Debug)]
pub enum QueryCommands {
    /// List root TOC nodes (year level)
    Root,

    /// Get a specific TOC node
    Node {
        /// Node ID to retrieve
        node_id: String,
    },

    /// Browse children of a node
    Browse {
        /// Parent node ID
        parent_id: String,

        /// Maximum results
        #[arg(short, long, default_value = "20")]
        limit: u32,

        /// Continuation token for pagination
        #[arg(short, long)]
        token: Option<String>,
    },

    /// Get events in time range
    Events {
        /// Start time (Unix ms or ISO)
        #[arg(long)]
        from: i64,

        /// End time (Unix ms or ISO)
        #[arg(long)]
        to: i64,

        /// Maximum results
        #[arg(short, long, default_value = "50")]
        limit: u32,
    },

    /// Expand a grip to show context
    Expand {
        /// Grip ID to expand
        grip_id: String,
    },
}

Task 2: Implement Query Command Handler

File: crates/memory-daemon/src/commands/query.rs

use memory_client::MemoryClient;
use anyhow::Result;

pub async fn handle_query(command: QueryCommands, endpoint: &str) -> Result<()> {
    let mut client = MemoryClient::connect(endpoint).await?;

    match command {
        QueryCommands::Root => {
            let nodes = client.get_toc_root().await?;
            for node in nodes {
                println!("{}: {} ({})", node.node_id, node.title, node.level);
            }
        }
        QueryCommands::Node { node_id } => {
            let node = client.get_node(&node_id).await?;
            print_node(&node);
        }
        QueryCommands::Browse { parent_id, limit, token } => {
            let response = client.browse_toc(&parent_id, limit, token).await?;
            for child in response.children {
                println!("{}: {}", child.node_id, child.title);
            }
            if let Some(token) = response.continuation_token {
                println!("\nMore results available. Use --token {}", token);
            }
        }
        QueryCommands::Events { from, to, limit } => {
            let events = client.get_events(from, to, limit).await?;
            for event in events {
                println!("[{}] {}: {}", event.timestamp, event.role, &event.text[..80.min(event.text.len())]);
            }
        }
        QueryCommands::Expand { grip_id } => {
            let context = client.expand_grip(&grip_id).await?;
            println!("=== Before ===");
            for e in context.events_before { println!("{}", e.text); }
            println!("\n=== Excerpt ===\n{}", context.excerpt);
            println!("\n=== After ===");
            for e in context.events_after { println!("{}", e.text); }
        }
    }
    Ok(())
}

fn print_node(node: &TocNode) {
    println!("ID: {}", node.node_id);
    println!("Title: {}", node.title);
    println!("Level: {:?}", node.level);
    println!("Summary: {}", node.summary.as_deref().unwrap_or("-"));
    println!("\nBullets:");
    for bullet in &node.bullets {
        println!("  • {}", bullet.text);
    }
    println!("\nChildren: {}", node.child_node_ids.len());
}

Task 3: Add Query Methods to MemoryClient

File: crates/memory-client/src/client.rs

Add methods for query RPCs:

impl MemoryClient {
    pub async fn get_toc_root(&mut self) -> Result<Vec<TocNode>, ClientError>;
    pub async fn get_node(&mut self, node_id: &str) -> Result<TocNode, ClientError>;
    pub async fn browse_toc(&mut self, parent_id: &str, limit: u32, token: Option<String>) -> Result<BrowseResponse, ClientError>;
    pub async fn get_events(&mut self, from: i64, to: i64, limit: u32) -> Result<Vec<Event>, ClientError>;
    pub async fn expand_grip(&mut self, grip_id: &str) -> Result<GripContext, ClientError>;
}

Task 4: Wire Query Command in main.rs

Update main.rs to handle Query command.

Task 5: Add Tests

Test CLI parsing for query commands.

Acceptance Criteria

  • memory-daemon query root lists year nodes
  • memory-daemon query node <id> shows node details
  • memory-daemon query browse <id> paginates children
  • memory-daemon query events --from --to retrieves events
  • memory-daemon query expand <grip_id> shows context
  • Helpful error messages on connection failure

Dependencies

  • Plan 05-01 complete (MemoryClient)

Plan created: 2026-01-30

05-02-SUMMARY

Phase 05-02 Summary: Query CLI

Completed Tasks

Task 1: Add Query RPCs to Proto

Updated proto/memory.proto with:

  • TocLevel enum (Year, Month, Week, Day, Segment)
  • TocBullet message (text, grip_ids)
  • TocNode message (full node structure)
  • Grip message (excerpt with event pointers)
  • GetTocRoot, GetNode, BrowseToc RPCs
  • GetEvents, ExpandGrip RPCs

Task 2: Implement Query Module in Service

Created crates/memory-service/src/query.rs:

  • get_toc_root() - Returns year-level nodes
  • get_node() - Fetches node by ID
  • browse_toc() - Paginated child navigation
  • get_events() - Time range event retrieval
  • expand_grip() - Context around grip excerpt
  • Type conversion functions (domain ↔ proto)
  • 8 unit tests

Task 3: Wire Query RPCs to Service

Updated MemoryServiceImpl in ingest.rs to implement all query RPCs.

Task 4: Add Query Methods to MemoryClient

Extended memory-client/src/client.rs:

  • get_toc_root() - Get year nodes
  • get_node() - Get specific node
  • browse_toc() - Browse children with pagination
  • get_events() - Get events in time range
  • expand_grip() - Expand grip context

Task 5: Add Query Subcommand to CLI

Updated memory-daemon/src/cli.rs:

  • QueryCommands enum with Root, Node, Browse, Events, Expand subcommands
  • Endpoint flag for specifying daemon address

Task 6: Implement Query Command Handler

Created handle_query() in commands.rs:

  • Formatted output for all query types
  • Pagination support with continuation tokens
  • Error handling for connection failures

Key Artifacts

File Purpose
proto/memory.proto TOC navigation and query messages
memory-service/src/query.rs Query RPC implementations
memory-service/src/ingest.rs Service trait implementation
memory-client/src/client.rs Client query methods
memory-daemon/src/cli.rs Query subcommand definitions
memory-daemon/src/commands.rs Query command handler

CLI Usage

# List root nodes
memory-daemon query root

# Get specific node
memory-daemon query node <node_id>

# Browse children with pagination
memory-daemon query browse <parent_id> --limit 20 --token <token>

# Get events in time range
memory-daemon query events --from <ms> --to <ms> --limit 50

# Expand grip context
memory-daemon query expand <grip_id> --before 3 --after 3

Verification

  • cargo build --workspace compiles
  • cargo test --workspace passes (117 tests)
  • 8 new query tests in memory-service
  • 2 new CLI tests in memory-daemon

Requirements Coverage

  • CLI-02: Query CLI for manual TOC navigation and testing

Completed: 2026-01-30

05-03-PLAN

Plan 05-03: Admin Commands

Goal

Add administrative commands for TOC rebuilding, database compaction, and status reporting.

Requirements Covered

  • CLI-03: Admin commands: rebuild-toc, compact, status

Tasks

Task 1: Add Admin Subcommand to CLI

File: crates/memory-daemon/src/cli.rs

Add Admin variant to Commands enum:

#[derive(Subcommand, Debug)]
pub enum Commands {
    // existing...

    /// Administrative commands
    Admin {
        #[command(subcommand)]
        command: AdminCommands,
    },
}

#[derive(Subcommand, Debug)]
pub enum AdminCommands {
    /// Rebuild TOC from raw events
    RebuildToc {
        /// Start from this date (YYYY-MM-DD)
        #[arg(long)]
        from_date: Option<String>,

        /// Dry run - show what would be done
        #[arg(long)]
        dry_run: bool,
    },

    /// Trigger RocksDB compaction
    Compact {
        /// Compact only specific column family
        #[arg(long)]
        cf: Option<String>,
    },

    /// Show database statistics
    Stats,
}

Task 2: Add Admin RPCs to Proto

File: proto/memory.proto

// Admin messages
message RebuildTocRequest {
    optional int64 from_timestamp_ms = 1;
    bool dry_run = 2;
}

message RebuildTocResponse {
    int32 segments_created = 1;
    int32 nodes_updated = 2;
    string message = 3;
}

message CompactRequest {
    optional string column_family = 1;
}

message CompactResponse {
    string message = 1;
}

message StatsRequest {}

message StatsResponse {
    int64 event_count = 1;
    int64 toc_node_count = 2;
    int64 grip_count = 3;
    int64 disk_usage_bytes = 4;
    map<string, int64> cf_sizes = 5;
}

service MemoryService {
    // existing RPCs...

    // Admin RPCs
    rpc RebuildToc(RebuildTocRequest) returns (RebuildTocResponse);
    rpc Compact(CompactRequest) returns (CompactResponse);
    rpc GetStats(StatsRequest) returns (StatsResponse);
}

Task 3: Implement Admin RPC Handlers

File: crates/memory-service/src/admin.rs

use crate::proto::{RebuildTocRequest, RebuildTocResponse, CompactRequest, CompactResponse, StatsRequest, StatsResponse};
use memory_storage::Storage;
use std::sync::Arc;

pub async fn rebuild_toc(
    storage: Arc<Storage>,
    request: RebuildTocRequest,
) -> Result<RebuildTocResponse, tonic::Status> {
    // 1. Query events from storage starting from from_timestamp
    // 2. Re-run segmentation
    // 3. Re-generate TOC nodes
    // 4. Return counts
}

pub async fn compact(
    storage: Arc<Storage>,
    request: CompactRequest,
) -> Result<CompactResponse, tonic::Status> {
    // Call storage.compact() or storage.compact_cf(cf_name)
}

pub async fn get_stats(
    storage: Arc<Storage>,
) -> Result<StatsResponse, tonic::Status> {
    // Gather counts from each CF
    // Get disk usage via std::fs
}

Task 4: Add Stats/Compact Methods to Storage

File: crates/memory-storage/src/db.rs

impl Storage {
    pub fn compact(&self) -> Result<(), StorageError> {
        self.db.compact_range::<&[u8], &[u8]>(None, None);
        Ok(())
    }

    pub fn compact_cf(&self, cf_name: &str) -> Result<(), StorageError> {
        let cf = self.db.cf_handle(cf_name).ok_or(...)?;
        self.db.compact_range_cf::<&[u8], &[u8]>(&cf, None, None);
        Ok(())
    }

    pub fn get_stats(&self) -> Result<StorageStats, StorageError> {
        // Count entries in each CF
        // Get disk usage
    }
}

Task 5: Implement Admin Command Handler

File: crates/memory-daemon/src/commands/admin.rs

pub async fn handle_admin(command: AdminCommands, endpoint: &str) -> Result<()> {
    let mut client = MemoryClient::connect(endpoint).await?;

    match command {
        AdminCommands::RebuildToc { from_date, dry_run } => {
            let from_ts = from_date.map(|d| parse_date(&d));
            let result = client.rebuild_toc(from_ts, dry_run).await?;
            println!("Segments created: {}", result.segments_created);
            println!("Nodes updated: {}", result.nodes_updated);
        }
        AdminCommands::Compact { cf } => {
            let result = client.compact(cf).await?;
            println!("{}", result.message);
        }
        AdminCommands::Stats => {
            let stats = client.get_stats().await?;
            println!("Events:     {}", stats.event_count);
            println!("TOC Nodes:  {}", stats.toc_node_count);
            println!("Grips:      {}", stats.grip_count);
            println!("Disk Usage: {} MB", stats.disk_usage_bytes / 1024 / 1024);
        }
    }
    Ok(())
}

Task 6: Add Admin Methods to MemoryClient

impl MemoryClient {
    pub async fn rebuild_toc(&mut self, from_ts: Option<i64>, dry_run: bool) -> Result<RebuildTocResponse, ClientError>;
    pub async fn compact(&mut self, cf: Option<String>) -> Result<CompactResponse, ClientError>;
    pub async fn get_stats(&mut self) -> Result<StatsResponse, ClientError>;
}

Task 7: Wire Admin Command in main.rs

Task 8: Add Tests

Test CLI parsing, storage stats method.

Acceptance Criteria

  • memory-daemon admin rebuild-toc rebuilds TOC from events
  • memory-daemon admin compact triggers compaction
  • memory-daemon admin stats shows database statistics
  • --dry-run flag shows plan without executing
  • Error handling for connection failures

Dependencies

  • Plan 05-01 complete (MemoryClient)
  • Plan 05-02 complete (query module pattern)

Plan created: 2026-01-30

05-03-SUMMARY

Phase 05-03 Summary: Admin Commands

Completed Tasks

Task 1: Add Admin Subcommand to CLI

Updated memory-daemon/src/cli.rs:

  • AdminCommands enum with Stats, Compact, RebuildToc subcommands
  • Database path override option
  • Dry-run flag for rebuild-toc

Task 2: Add Storage Admin Methods

Added to memory-storage/src/db.rs:

  • compact() - Full compaction on all column families
  • compact_cf(cf_name) - Compact specific column family
  • get_stats() - Returns StorageStats with counts and disk usage
  • StorageStats struct exported from crate

Task 3: Implement Admin Command Handler

Created handle_admin() in commands.rs:

  • Stats: Shows event/node/grip counts and disk usage
  • Compact: Triggers RocksDB compaction (all or specific CF)
  • RebuildToc: Placeholder for TOC rebuild with dry-run support

Key Artifacts

File Purpose
memory-daemon/src/cli.rs Admin subcommand definitions
memory-daemon/src/commands.rs Admin command handler
memory-storage/src/db.rs compact(), get_stats() methods
memory-storage/src/lib.rs StorageStats export

CLI Usage

# Show database statistics
memory-daemon admin stats

# Trigger full compaction
memory-daemon admin compact

# Compact specific column family
memory-daemon admin compact --cf events

# Rebuild TOC (dry run)
memory-daemon admin rebuild-toc --dry-run

# Rebuild TOC from specific date
memory-daemon admin rebuild-toc --from-date 2026-01-01

Verification

  • cargo build --workspace compiles
  • cargo test --workspace passes (116 tests)
  • CLI help displays correctly

Requirements Coverage

  • CLI-03: Admin commands: rebuild-toc, compact, status

Notes

  • RebuildToc is a placeholder - full implementation would require integrating with memory-toc segmentation and summarization
  • Stats opens storage directly (not via gRPC) for local admin operations
  • Shellexpand used for tilde expansion in paths

Completed: 2026-01-30

05-RESEARCH

Phase 5 Research: Integration

Overview

Phase 5 connects the memory system to external hook handlers and provides CLI tools for querying and administration.

Requirements

  • HOOK-02: Hook handlers call daemon's IngestEvent RPC
  • HOOK-03: Event types map 1:1 from hook events (SessionStart, UserPromptSubmit, PostToolUse, Stop, etc.)
  • CLI-02: Query CLI for manual TOC navigation and testing
  • CLI-03: Admin commands: rebuild-toc, compact, status

Technical Analysis

Hook Handler Integration (05-01)

gRPC Client Library

Need to expose a client API that hook handlers can use:

// memory-client crate
pub struct MemoryClient {
    inner: memory_service::MemoryServiceClient<Channel>,
}

impl MemoryClient {
    pub async fn connect(endpoint: &str) -> Result<Self, Error>;
    pub async fn ingest(&self, event: Event) -> Result<IngestResponse, Error>;
}

Event Type Mapping

Hook events from code_agent_context_hooks:

  • SessionStart → EventType::SessionStart
  • UserPromptSubmit → EventType::UserMessage
  • PostToolUse → EventType::ToolUse
  • Stop → EventType::SessionEnd

Mapping function:

pub fn map_hook_event(hook_event: HookEvent) -> memory_types::Event {
    // Convert hook event fields to memory event
}

Query CLI (05-02)

Commands to Add

memory-daemon query root                    # List year nodes
memory-daemon query node <node_id>          # Get specific node
memory-daemon query browse <parent_id> [--limit N]  # Browse children
memory-daemon query events --from TS --to TS [--limit N]  # Get events
memory-daemon query expand <grip_id>        # Expand grip context

Implementation

Add Query subcommand to existing CLI with nested subcommands.

Admin Commands (05-03)

Commands to Add

memory-daemon admin rebuild-toc [--from-date DATE]  # Rebuild TOC from events
memory-daemon admin compact                         # Trigger RocksDB compaction
memory-daemon admin stats                           # Show database statistics

Implementation

  • rebuild-toc: Re-run segmentation and summarization from raw events
  • compact: Call db.compact_range() on RocksDB
  • stats: Show event count, TOC node count, grip count, disk usage

Dependencies

  • tonic for gRPC client
  • Existing memory-service proto definitions
  • tokio for async runtime

File Structure

crates/
├── memory-client/          # New crate for client API
│   ├── Cargo.toml
│   └── src/
│       ├── lib.rs
│       └── hook_mapping.rs
├── memory-daemon/
│   └── src/
│       ├── cli.rs          # Update with new commands
│       ├── commands/
│       │   ├── query.rs    # New: query commands
│       │   └── admin.rs    # New: admin commands
└── memory-service/
    └── src/
        └── admin.rs        # Admin RPC implementations

Plan Breakdown

Plan Focus Files
05-01 Client library + hook mapping memory-client crate
05-02 Query CLI commands memory-daemon query subcommand
05-03 Admin CLI commands memory-daemon admin subcommand

Research completed: 2026-01-30

Clone this wiki locally