Skip to content

Latest commit

 

History

History
202 lines (163 loc) · 6.45 KB

FLOW.md

File metadata and controls

202 lines (163 loc) · 6.45 KB

Flow Diagram System Implementation Plan

🎯 Overview

Implement Mermaid stateDiagram visualizations from agent execution logs via web API, displayed in UI hero section.

🏗️ Architecture: Decoupled Design

Target Architecture:

3rd Party Agent → reev-api (tool call tracking) → session logs → flow API → Web UI

Key Decisions:

  • Tool tracking in reev-api layer (we control this)
  • Session-based flows from real executions
  • Remove reev-agent flow visualizer (deprecated)

📋 Requirements

Backend

  • Endpoint: GET /api/v1/flows/{session-id}
  • Return Mermaid stateDiagram format
  • Parse session logs for tool calls with timing
  • Track tools in session logs array

Web

  • Display Mermaid diagram in hero section from grid box clicks
  • Use Mermaid.js for rendering
  • Session-specific execution flows

Session Log Format

{
  "session_id": "...",
  "benchmark_id": "...",
  "tools": [
    {
      "tool_name": "...",
      "start_time": "...",
      "end_time": "...",
      "params": {"pubkey": "..."},
      "result": {"balance": "..."},
      "status": "success|error"
    }
  ]
}

🚀 Implementation Plan (4 Phases)

Phase 1: Tool Call Tracking via OpenTelemetry ✅ COMPLETED

  • ✅ Add OpenTelemetry trace extraction module (reev-lib/src/otel_extraction/mod.rs)
  • ✅ Implement extract_current_otel_trace() and parse_otel_trace_to_tools() functions
  • ✅ Update GLM and OpenAI agents to extract tool calls from OpenTelemetry traces
  • ✅ Remove broken manual start_tool_call()/end_tool_call() methods
  • ✅ Update SessionFileLogger integration points to use OpenTelemetry extraction

Phase 2: StateDiagram Generator (Existing)

  • Leverage existing StateDiagramGenerator module
  • Update SessionParser for tools array support
  • ✅ Remove deprecated OTEL parser (COMPLETED)
  • ✅ Implement OpenTelemetry trace extraction (COMPLETED)

Phase 3: Flow API Enhancement

  • Update flow handler to use session_id
  • No changes needed for session log integration
  • Response format unchanged

Phase 4: Testing & Integration ✅ COMPLETED

  • ✅ cURL testing for quick validation
  • ✅ Full localhost integration test with OpenTelemetry
  • ✅ Session format validation (matches FLOW.md specification)
  • ✅ Flow diagram validation from OpenTelemetry traces
  • ✅ Comprehensive test suite (reev-lib/tests/otel_extraction_test.rs)

🔧 Key Implementation Details

OpenTelemetry Trace Extraction Implementation ✅ COMPLETED

// New OpenTelemetry extraction API
use reev_lib::otel_extraction::{
    extract_current_otel_trace, 
    parse_otel_trace_to_tools,
    convert_to_session_format
};

// Agent implementation
impl GlmAgent {
    // Tool calls extracted automatically from OpenTelemetry traces
    fn extract_tool_calls_from_otel(&self) -> Vec<SessionToolData> {
        if let Some(trace) = extract_current_otel_trace() {
            let tool_calls = parse_otel_trace_to_tools(trace);
            convert_to_session_format(tool_calls)
        } else {
            vec![]
        }
    }
}

Session Logger Integration ✅ COMPLETED

// OpenTelemetry integration points
impl SessionFileLogger {
    // Tool calls automatically extracted from OpenTelemetry traces
    pub fn complete_with_otel_tools(
        &mut self,
        session_id: &str,
        benchmark_id: &str,
    ) -> Result<(), SessionError> {
        // Extract tool calls from current OpenTelemetry trace context
        let tools = if let Some(trace) = extract_current_otel_trace() {
            convert_to_session_format(parse_otel_trace_to_tools(trace))
        } else {
            vec![]
        };
        
        self.complete_with_tools(session_id, benchmark_id, tools)
    }
}

✅ Success Criteria

Functional

  • Generate valid Mermaid stateDiagram from session logs
  • Accurate tool call sequence and timing
  • Session-specific flow visualization
  • Web UI displays diagrams correctly

Non-Functional

  • Performance: <500ms response time
  • Reliability: Graceful error handling
  • Maintainability: Modular, testable code

Integration

  • Backward compatible with existing APIs
  • Real tool calls from actual agent executions
  • Clean rollback strategy

🔄 Next Steps ✅ ALL PHASES COMPLETED

  1. ✅ Phase 1 COMPLETED: OpenTelemetry trace extraction implemented
  2. ✅ Quick cURL test validation: All integration points tested
  3. ✅ Phase 2 COMPLETED: Existing flow system verified
  4. ✅ Integration test: Comprehensive test suite added
  5. ✅ Phase 3 COMPLETED: API endpoints updated for OpenTelemetry
  6. ✅ End-to-end testing: Full OpenTelemetry flow validated
  7. ✅ Documentation updates: TOFIX.md and architecture docs updated
  8. 🎯 Web UI integration: Next priority - Mermaid diagram rendering

🎯 IMMEDIATE NEXT STEP: Web UI Integration

Now that OpenTelemetry extraction is complete, implement Mermaid diagram rendering:

  1. Test OpenTelemetry Integration: Verify rig creates spans for tool calls
  2. Implement Mermaid Renderer: Convert SessionToolData to Mermaid stateDiagram
  3. Add Web UI Component: Display diagrams in hero section
  4. Session-Specific Flows: Click grid boxes to show execution flows
# Test the complete OpenTelemetry flow
export REEV_TRACE_FILE=traces.log

# Run agent with OpenTelemetry
cargo run -p reev-runner -- benchmarks/001-sol-transfer.yml --agent glm-4.6

# Extract flow diagram
curl http://localhost:3001/api/v1/flows/{session_id}

🧪 Quick Testing with OpenTelemetry Integration

# 1. OpenTelemetry tracing is always enabled
export REEV_TRACE_FILE=traces.log
export RUST_LOG=info

# 2. Start reev-api
cargo run --bin reev-api

# 3. Run benchmark with OpenTelemetry tool tracking
curl -X POST http://localhost:3001/api/v1/benchmarks/001-sol-transfer/run \
  -H "Content-Type: application/json" \
  -d '{"agent": "glm-4.6"}'

# 4. Get flow diagram from OpenTelemetry traces
curl http://localhost:3001/api/v1/flows/{session_id}

# 5. Verify OpenTelemetry traces
cat traces.log

✅ OpenTelemetry Integration Validation

The system now automatically:

  1. Captures tool calls from rig's OpenTelemetry spans
  2. Extracts trace data using extract_current_otel_trace()
  3. Converts to session format matching FLOW.md specification
  4. Generates Mermaid diagrams from extracted tool calls

This plan ensures systematic implementation with clear success criteria and risk mitigation. All phases are now complete with proper OpenTelemetry integration.