AI Agent Evaluation Framework for Multi-Participant Coordination

A comprehensive testbed for evaluating AI agents on open-ended coordination tasks Built with LangGraph, Custom MCP Tools, and LLM-as-a-Judge evaluation

🚀 What I Built

This project implements a production-ready AI agent evaluation framework that systematically tests how LLM agents handle complex, multi-step coordination tasks. The system evaluates agent performance across 9 different experimental conditions, processing 540 end-to-end experiments with comprehensive metrics and automated evaluation.

Key Technical Achievements

• 🔧 Custom MCP Server: Built a TypeScript-based Model Context Protocol server extending Google Maps API with custom tools including find_middle_point for geographic calculations
• 🤖 Agent Orchestration: Built LangGraph workflow graphs with conditional routing for both single and multi-agent architectures
• 🧪 Multi-Dimensional Evaluation: Designed comprehensive metrics combining Bayesian-weighted ratings, proximity scores, tool compliance, and LLM-as-a-Judge for holistic agent assessment
• 📊 Experimental Testbed: Designed systematic variations testing agent autonomy (3 planning levels × 3 information structures = 9 conditions)
• 🎯 User Simulation: Built realistic multi-user simulator with adaptive preference changes and decentralized information access

💡 Engineering Highlights

1. Custom MCP Tool Development

Extended an open-source MCP Google Maps server with custom tools:

• find_middle_point: Geographic midpoint calculator for multiple locations
• Enhanced search_nearby with minRating parameter for filtering
• Full TypeScript implementation with proper error handling

2. LangGraph Agent Architecture

Built robust agentic workflows with production monitoring:

• Conditional routing: Agent → Tools → Agent loop based on LLM tool calls
• State management: Conversation history, turn counting, user tracking
• Multi-step reasoning: Iterative planning and execution cycles
• User simulation system: Independent conversation threads for 1-10 participants
• LangSmith integration: Full observability with execution tracing and debugging

3. Comprehensive Evaluation Framework

Built multi-dimensional evaluation system combining hard metrics and LLM assessment:

Reliability Metrics:

• Tool compliance scoring (tracking required API calls)
• Task completion rates
• Trajectory analysis

Performance Metrics (Quantitative):

• Bayesian-weighted ratings: Quality assessment accounting for review count
• Proximity scores: Walking time discretization (5-min intervals)
• Optimal baseline comparison: Ranking against best possible selections

LLM-as-a-Judge (Qualitative):

• Web search integration for dietary restriction validation (259 cached results)
• Structured evaluation with Chain-of-Thought reasoning
• Multi-point scoring rubric

Combined Score:

• Geometric mean preventing one-dimensional optimization
• Balanced trade-offs between ratings, proximity, and restrictions

4. Scalable Experiment Pipeline

• Bash orchestration running 540 experiments across 3 models
• Parallel execution with proper resource management
• Comprehensive logging and result aggregation
• CSV export for analysis

📊 Key Results

Through systematic experimentation, I discovered:

• 40% performance degradation when agents must independently decompose tasks (measuring impact of reduced guidance)
• Multi-agent frameworks don't always help: OctoTools used 3.5× more tokens with no performance gains
• Information structure matters: Centralized information consistently outperforms decentralized
• Agents struggle with autonomy: Error recovery and multi-user coordination remain challenging

🛠️ Tech Stack

Backend & Agents:

• Python 3.8+ with LangGraph for agent orchestration
• LangChain for LLM integration (OpenAI GPT-4o-mini, GPT-4.1-mini)
• LangSmith for production-ready observability and tracing
• Async/await patterns for efficient I/O
• Custom state management and message routing

Infrastructure:

• TypeScript/Node.js for MCP server development
• Model Context Protocol (MCP) for tool integration
• Google Maps API for location services
• Bash scripting for experiment orchestration

Evaluation & Analysis:

• Multi-metric system: Bayesian statistics, proximity calculations, tool compliance
• LLM-as-a-Judge with GPT-4.1-mini for dietary restriction assessment
• Web search integration with caching (259 cached queries)
• Pandas for data analysis and CSV processing
• Matplotlib/Seaborn for visualization

DevOps & Monitoring:

• LangSmith tracing for agent execution monitoring
• Environment-based configuration (.env)
• Modular architecture (src/, infrastructure/, experiments/)
• Comprehensive logging and error tracking
• Cost analysis and token usage monitoring

🏗️ Repository Structure

dissertation/
├── src/                                   # Source code
│   ├── agents/                            # Core experimental agents
│   │   ├── centralized_decentralized_agent.py
│   │   ├── adaptive_agent.py
│   │   ├── octotools_centralized_decentralized_agent.py
│   │   └── octotools_adaptive_agent.py
│   ├── utils/                             # Shared utilities
│   │   ├── prompts.py                     # Experimental prompt manipulations
│   │   ├── message_to_user_n.py          # User simulator framework
│   │   └── middlelocation.py             # Geographical calculations
│   └── users/                             # Test data
│       └── sample_event.py                # Event configurations
│
├── infrastructure/                        # Infrastructure components
│   ├── mcp-google-map/                    # Custom MCP Google Maps server
│   │   ├── src/                           # TypeScript source
│   │   │   ├── index.ts                   # Main MCP server
│   │   │   └── maps-tools/                # Tool implementations
│   │   │       ├── mapsTools.ts           # Tool definitions (find_middle_point)
│   │   │       └── searchPlaces.ts        # Place search logic
│   │   └── dist/                          # Compiled JavaScript
│   ├── local_mcp/                         # MCP utilities
│   │   └── mcp_utils.py                   # MCP client activation
│   └── octotools/                         # OctoTools framework (external)
│       ├── octotools/                     # Core framework
│       │   ├── solver.py                  # Main solver
│       │   ├── models/                    # Planner, executor, memory
│       │   └── tools/                     # Tool implementations
│       └── tasks/                         # Task definitions
│
├── experiments/                           # Experiments and results
│   ├── comprehensive_experiment_pipeline.sh  # Main experiment runner
│   ├── 4o_mini_logs/                      # GPT-4o-mini logs (222 files)
│   ├── 4_1_mini_logs/                     # GPT-4.1-mini logs (214 files)
│   ├── octotools_logs/                    # OctoTools logs (100 files)
│   ├── gpt-4o-mini_analysis_results.csv   # GPT-4o-mini results
│   ├── gpt-4.1-mini_analysis_results.csv  # GPT-4.1-mini results
│   ├── results_octotools.csv              # OctoTools results
│   ├── evaluations_scripts/               # Analysis and visualization
│   └── specialized_analysis/              # Targeted analysis scripts
│
├── LLM_as_a_judge/                        # LLM-as-Judge evaluation
│   ├── llm_as_a_judge.py                  # Main judge implementation
│   ├── judge_process_logs.py              # Log processing
│   ├── cache_manager.py                   # Caching system
│   ├── web_search_cache/                  # Cached searches (259 files)
│   └── octotools_version/                 # OctoTools-specific judge
│
├── demos/                                 # Quick demos
│   ├── demo_centralized.sh
│   ├── demo_adaptive.sh
│   ├── demo_planning_overhead_comparison.sh
│   └── README.md
│
├── docs/                                  # Documentation
│   ├── SETUP.md                           # Installation guide
│   ├── EXPERIMENTS.md                     # Experiments guide
│   └── REPOSITORY_STRUCTURE.md            # Structure documentation
│
├── figures/                               # Visualizations
│   ├── *.jpg                              # Performance plots
│   ├── pdfs/                              # Publication-ready figures
│   └── svgs/                              # Vector graphics
│
├── cost_analysis/                         # Token usage analysis
│   ├── COST_ANALYSIS_SUMMARY.md           # Summary and key findings
│   ├── cost_analysis.py                   # Main analysis script
│   ├── log_cleaner.py                     # Log cleaning utility
│   └── detailed_token_analysis.csv        # Detailed results (508 experiments)
│
├── README.md                              # This file
├── requirements.txt                       # Python dependencies
├── setup.py                               # Package setup
└── LICENSE                                # CC BY-NC 4.0 License

🚀 Quick Start

Prerequisites

• Python 3.8+
• Node.js 16+ (for MCP server)
• Google Maps API key

Installation

1. Clone the repository

git clone <repository-url>
cd dissertation

2. Install the package in development mode

pip install -e .

This installs all Python dependencies and sets up proper import paths.

3. Install MCP server dependencies

cd infrastructure/mcp-google-map
npm install
npm run build
cd ../..

4. Set up environment variables

cp .env.example .env
# Edit .env and add your API keys:
# - GOOGLE_MAPS_API_KEY
# - OPENAI_API_KEY
# - LANGSMITH_API_KEY (optional, for tracing)

Running a Demo

Test a single centralized scenario:

python3 src/agents/centralized_decentralized_agent.py \
    7 \                          # 7 invitees
    0 \                          # preference index 0 (vegetarian)
    "centralized" \              # information complexity
    "medium_overhead" \          # planning overhead
    "gpt-4o-mini"                # model

Running Full Experiments

To reproduce the dissertation results:

cd experiments
./comprehensive_experiment_pipeline.sh

This will:

1. Run 540 experiments across all conditions
2. Analyze results with custom metrics
3. Apply LLM-as-Judge evaluation
4. Generate visualizations

⚠️ Warning: Full pipeline takes ~24 hours and costs ~$150 in API calls.

🔬 Experimental Design

Independent Variables

1. Information Complexity (How information is distributed):

   • Centralized: All participant info provided upfront
   • Decentralized: Agent must query each user separately
   • Adaptive: Users may change preferences mid-conversation

2. Planning Overhead (Amount of guidance given to agent):

   • Low: Detailed step-by-step instructions (max scaffolding)
   • Medium: Goal + general approach (moderate scaffolding)
   • High: Only the goal (tests autonomous planning)

3. Agent Architecture:

   • GPT-4o-mini (single agent)
   • GPT-4.1-mini (single agent)
   • OctoTools (multi-agent framework)

Dependent Variables

Reliability Metrics:

• Tool Compliance Score: Geometric mean of required API call ratios (0 if any tool unused)
• Task Completion Rate: Percentage yielding valid formatted output

Performance Metrics:

• Bayesian Decision Quality: Weighted ratings (prior: μ=3.0, C=50) normalized against optimal baseline
• Proximity Decision Quality: Walking time scores (10pts for 0-5min, 9pts for 5-10min, etc.) vs optimal
• Restriction Fulfillment (LLM-as-a-Judge): 0-2 point rubric per venue, averaged and normalized

Combined Score:

• Geometric mean: ∛(Bayesian × Proximity × Restrictions) - penalizes extreme deficiencies

📈 Key Results

Performance by Planning Overhead

Model	Low	Medium	High
GPT-4o-mini	0.92	0.84	0.53
GPT-4.1-mini	0.95	0.87	0.58
OctoTools	0.87	0.79	0.41

Token Usage by Scenario

Agent	Centralized	Decentralized	Adaptive
GPT-4o-mini	4,034	5,721	9,209
GPT-4.1-mini	6,170	6,173	7,101
OctoTools	14,868	20,882	30,337

Success Rates

• Centralized: 94.7% (GPT-4o-mini best)
• Decentralized: 89.2% (significant drop)
• Adaptive: 73.6% (multi-user failures)

🛠️ Custom Contributions

1. MCP Google Maps Server Extensions

Added to existing @cablate/mcp-google-map server:

• find_middle_point tool: Calculates geographical midpoint from list of locations
• minRating parameter: Quality filtering for venue search
• Enhanced error handling: Improved robustness for agent use

2. Multi-Dimensional Evaluation System

Comprehensive assessment framework combining:

• Hard Metrics: Bayesian-weighted ratings (C=50 prior confidence), proximity scores (5-min discretization), tool compliance (geometric mean)
• LLM-as-a-Judge: Web-search enhanced dietary restriction evaluation with structured rubric
• Optimal Baselines: Systematic comparison against best possible selections from search results
• Cost Optimization: Web search caching reduces evaluation costs by 90%
• Combined Score: Geometric mean preventing single-dimension gaming

3. Planning Overhead Manipulation

Novel experimental design systematically varying agent autonomy:

• Reveals 40% performance drop without scaffolding
• First systematic study of guidance-performance relationship
• Challenges assumptions about autonomous agent capabilities

📊 Analysis Scripts

# Generate performance plots
python3 experiments/evaluations_scripts/new_plots.py

# Analyze unfinished experiments
python3 experiments/evaluations_scripts/unfinished_experiments_analysis.py

# Token cost analysis
python3 cost_analysis/cost_analysis.py

# LLM-as-Judge evaluation
python3 LLM_as_a_judge/judge_process_logs.py <log_dir> <results_csv>

🎓 Research Context

This project was developed as part of my MSc in Speech and Language Processing at the University of Edinburgh (2025). The research investigates how AI agents perform on open-ended coordination tasks and explores the relationship between task scaffolding and agent autonomy. This work was supervised by PhD Alexandra Birch and PhD Patrick Chen.

Key Research Questions:

• How does reducing task guidance affect agent performance?
• When do multi-agent architectures provide advantages over single agents?
• What evaluation methods effectively measure agent capabilities on complex tasks?

Full Abstract: Understanding how AI agents perform on open-ended coordination tasks with minimal human guidance is an important evaluation challenge. This work varies information access and planning autonomy across three levels each (9 conditions total), then evaluates agents using custom reliability and performance metrics. Through 540 experiments, we identify systematic limitations including 40% performance degradation without scaffolding and persistent multi-user communication failures.

🛠️ Potential Applications

This framework serves both research and commercial needs:

Research & Development

• Agent benchmarking: Test any LLM agent on multi-step coordination tasks
• Tool integration testing: Evaluate how agents use custom tools (MCP)
• Multi-agent analysis: Compare single vs multi-agent architectures

Commercial Applications (via quickword.app)

• Coordination Task Automation: Production-ready agent orchestration for multi-participant planning scenarios (event scheduling, resource allocation, group decision-making).

🤝 Built With

• OctoTools Framework: github.com/octotools/octotools
• MCP Google Maps Server: @cablate/mcp-google-map (extended with custom tools)
• LangGraph: Agent orchestration and state machines
• LangChain MCP Adapters: MCP Integration

📄 License

This work is licensed under Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0).

You may:

• ✅ View and review for hiring/recruitment purposes
• ✅ Use for academic research and education
• ✅ Share and adapt with attribution

You may NOT:

• ❌ Use commercially or for profit
• ❌ Sell or monetize this work

See LICENSE file for full details.

📧 Contact

Andrés Cotton MSc Speech and Language Processing, University of Edinburgh

• 💼 Open to opportunities in AI/ML Engineering, Agent Development, and NLP
• 🔗 LinkedIn | GitHub
• 📬 Open an issue for questions about the code

---

This repository showcases a comprehensive AI agent evaluation framework built with modern tools (LangGraph, MCP, LLM-as-a-Judge) and demonstrates full-stack development from infrastructure (TypeScript MCP server) to evaluation (Python pipeline processing 540 experiments).