A comprehensive testbed for evaluating AI agents on open-ended coordination tasks Built with LangGraph, Custom MCP Tools, and LLM-as-a-Judge evaluation
This project implements a production-ready AI agent evaluation framework that systematically tests how LLM agents handle complex, multi-step coordination tasks. The system evaluates agent performance across 9 different experimental conditions, processing 540 end-to-end experiments with comprehensive metrics and automated evaluation.
- 🔧 Custom MCP Server: Built a TypeScript-based Model Context Protocol server extending Google Maps API with custom tools including
find_middle_pointfor geographic calculations - 🤖 Agent Orchestration: Built LangGraph workflow graphs with conditional routing for both single and multi-agent architectures
- 🧪 Multi-Dimensional Evaluation: Designed comprehensive metrics combining Bayesian-weighted ratings, proximity scores, tool compliance, and LLM-as-a-Judge for holistic agent assessment
- 📊 Experimental Testbed: Designed systematic variations testing agent autonomy (3 planning levels × 3 information structures = 9 conditions)
- 🎯 User Simulation: Built realistic multi-user simulator with adaptive preference changes and decentralized information access
Extended an open-source MCP Google Maps server with custom tools:
find_middle_point: Geographic midpoint calculator for multiple locations- Enhanced
search_nearbywithminRatingparameter for filtering - Full TypeScript implementation with proper error handling
Built robust agentic workflows with production monitoring:
- Conditional routing: Agent → Tools → Agent loop based on LLM tool calls
- State management: Conversation history, turn counting, user tracking
- Multi-step reasoning: Iterative planning and execution cycles
- User simulation system: Independent conversation threads for 1-10 participants
- LangSmith integration: Full observability with execution tracing and debugging
Built multi-dimensional evaluation system combining hard metrics and LLM assessment:
Reliability Metrics:
- Tool compliance scoring (tracking required API calls)
- Task completion rates
- Trajectory analysis
Performance Metrics (Quantitative):
- Bayesian-weighted ratings: Quality assessment accounting for review count
- Proximity scores: Walking time discretization (5-min intervals)
- Optimal baseline comparison: Ranking against best possible selections
LLM-as-a-Judge (Qualitative):
- Web search integration for dietary restriction validation (259 cached results)
- Structured evaluation with Chain-of-Thought reasoning
- Multi-point scoring rubric
Combined Score:
- Geometric mean preventing one-dimensional optimization
- Balanced trade-offs between ratings, proximity, and restrictions
- Bash orchestration running 540 experiments across 3 models
- Parallel execution with proper resource management
- Comprehensive logging and result aggregation
- CSV export for analysis
Through systematic experimentation, I discovered:
- 40% performance degradation when agents must independently decompose tasks (measuring impact of reduced guidance)
- Multi-agent frameworks don't always help: OctoTools used 3.5× more tokens with no performance gains
- Information structure matters: Centralized information consistently outperforms decentralized
- Agents struggle with autonomy: Error recovery and multi-user coordination remain challenging
Backend & Agents:
- Python 3.8+ with LangGraph for agent orchestration
- LangChain for LLM integration (OpenAI GPT-4o-mini, GPT-4.1-mini)
- LangSmith for production-ready observability and tracing
- Async/await patterns for efficient I/O
- Custom state management and message routing
Infrastructure:
- TypeScript/Node.js for MCP server development
- Model Context Protocol (MCP) for tool integration
- Google Maps API for location services
- Bash scripting for experiment orchestration
Evaluation & Analysis:
- Multi-metric system: Bayesian statistics, proximity calculations, tool compliance
- LLM-as-a-Judge with GPT-4.1-mini for dietary restriction assessment
- Web search integration with caching (259 cached queries)
- Pandas for data analysis and CSV processing
- Matplotlib/Seaborn for visualization
DevOps & Monitoring:
- LangSmith tracing for agent execution monitoring
- Environment-based configuration (.env)
- Modular architecture (src/, infrastructure/, experiments/)
- Comprehensive logging and error tracking
- Cost analysis and token usage monitoring
dissertation/
├── src/ # Source code
│ ├── agents/ # Core experimental agents
│ │ ├── centralized_decentralized_agent.py
│ │ ├── adaptive_agent.py
│ │ ├── octotools_centralized_decentralized_agent.py
│ │ └── octotools_adaptive_agent.py
│ ├── utils/ # Shared utilities
│ │ ├── prompts.py # Experimental prompt manipulations
│ │ ├── message_to_user_n.py # User simulator framework
│ │ └── middlelocation.py # Geographical calculations
│ └── users/ # Test data
│ └── sample_event.py # Event configurations
│
├── infrastructure/ # Infrastructure components
│ ├── mcp-google-map/ # Custom MCP Google Maps server
│ │ ├── src/ # TypeScript source
│ │ │ ├── index.ts # Main MCP server
│ │ │ └── maps-tools/ # Tool implementations
│ │ │ ├── mapsTools.ts # Tool definitions (find_middle_point)
│ │ │ └── searchPlaces.ts # Place search logic
│ │ └── dist/ # Compiled JavaScript
│ ├── local_mcp/ # MCP utilities
│ │ └── mcp_utils.py # MCP client activation
│ └── octotools/ # OctoTools framework (external)
│ ├── octotools/ # Core framework
│ │ ├── solver.py # Main solver
│ │ ├── models/ # Planner, executor, memory
│ │ └── tools/ # Tool implementations
│ └── tasks/ # Task definitions
│
├── experiments/ # Experiments and results
│ ├── comprehensive_experiment_pipeline.sh # Main experiment runner
│ ├── 4o_mini_logs/ # GPT-4o-mini logs (222 files)
│ ├── 4_1_mini_logs/ # GPT-4.1-mini logs (214 files)
│ ├── octotools_logs/ # OctoTools logs (100 files)
│ ├── gpt-4o-mini_analysis_results.csv # GPT-4o-mini results
│ ├── gpt-4.1-mini_analysis_results.csv # GPT-4.1-mini results
│ ├── results_octotools.csv # OctoTools results
│ ├── evaluations_scripts/ # Analysis and visualization
│ └── specialized_analysis/ # Targeted analysis scripts
│
├── LLM_as_a_judge/ # LLM-as-Judge evaluation
│ ├── llm_as_a_judge.py # Main judge implementation
│ ├── judge_process_logs.py # Log processing
│ ├── cache_manager.py # Caching system
│ ├── web_search_cache/ # Cached searches (259 files)
│ └── octotools_version/ # OctoTools-specific judge
│
├── demos/ # Quick demos
│ ├── demo_centralized.sh
│ ├── demo_adaptive.sh
│ ├── demo_planning_overhead_comparison.sh
│ └── README.md
│
├── docs/ # Documentation
│ ├── SETUP.md # Installation guide
│ ├── EXPERIMENTS.md # Experiments guide
│ └── REPOSITORY_STRUCTURE.md # Structure documentation
│
├── figures/ # Visualizations
│ ├── *.jpg # Performance plots
│ ├── pdfs/ # Publication-ready figures
│ └── svgs/ # Vector graphics
│
├── cost_analysis/ # Token usage analysis
│ ├── COST_ANALYSIS_SUMMARY.md # Summary and key findings
│ ├── cost_analysis.py # Main analysis script
│ ├── log_cleaner.py # Log cleaning utility
│ └── detailed_token_analysis.csv # Detailed results (508 experiments)
│
├── README.md # This file
├── requirements.txt # Python dependencies
├── setup.py # Package setup
└── LICENSE # CC BY-NC 4.0 License
- Python 3.8+
- Node.js 16+ (for MCP server)
- Google Maps API key
- Clone the repository
git clone <repository-url>
cd dissertation- Install the package in development mode
pip install -e .This installs all Python dependencies and sets up proper import paths.
- Install MCP server dependencies
cd infrastructure/mcp-google-map
npm install
npm run build
cd ../..- Set up environment variables
cp .env.example .env
# Edit .env and add your API keys:
# - GOOGLE_MAPS_API_KEY
# - OPENAI_API_KEY
# - LANGSMITH_API_KEY (optional, for tracing)Test a single centralized scenario:
python3 src/agents/centralized_decentralized_agent.py \
7 \ # 7 invitees
0 \ # preference index 0 (vegetarian)
"centralized" \ # information complexity
"medium_overhead" \ # planning overhead
"gpt-4o-mini" # modelTo reproduce the dissertation results:
cd experiments
./comprehensive_experiment_pipeline.shThis will:
- Run 540 experiments across all conditions
- Analyze results with custom metrics
- Apply LLM-as-Judge evaluation
- Generate visualizations
-
Information Complexity (How information is distributed):
- Centralized: All participant info provided upfront
- Decentralized: Agent must query each user separately
- Adaptive: Users may change preferences mid-conversation
-
Planning Overhead (Amount of guidance given to agent):
- Low: Detailed step-by-step instructions (max scaffolding)
- Medium: Goal + general approach (moderate scaffolding)
- High: Only the goal (tests autonomous planning)
-
Agent Architecture:
- GPT-4o-mini (single agent)
- GPT-4.1-mini (single agent)
- OctoTools (multi-agent framework)
Reliability Metrics:
- Tool Compliance Score: Geometric mean of required API call ratios (0 if any tool unused)
- Task Completion Rate: Percentage yielding valid formatted output
Performance Metrics:
- Bayesian Decision Quality: Weighted ratings (prior: μ=3.0, C=50) normalized against optimal baseline
- Proximity Decision Quality: Walking time scores (10pts for 0-5min, 9pts for 5-10min, etc.) vs optimal
- Restriction Fulfillment (LLM-as-a-Judge): 0-2 point rubric per venue, averaged and normalized
Combined Score:
- Geometric mean: ∛(Bayesian × Proximity × Restrictions) - penalizes extreme deficiencies
| Model | Low | Medium | High |
|---|---|---|---|
| GPT-4o-mini | 0.92 | 0.84 | 0.53 |
| GPT-4.1-mini | 0.95 | 0.87 | 0.58 |
| OctoTools | 0.87 | 0.79 | 0.41 |
| Agent | Centralized | Decentralized | Adaptive |
|---|---|---|---|
| GPT-4o-mini | 4,034 | 5,721 | 9,209 |
| GPT-4.1-mini | 6,170 | 6,173 | 7,101 |
| OctoTools | 14,868 | 20,882 | 30,337 |
- Centralized: 94.7% (GPT-4o-mini best)
- Decentralized: 89.2% (significant drop)
- Adaptive: 73.6% (multi-user failures)
Added to existing @cablate/mcp-google-map server:
find_middle_pointtool: Calculates geographical midpoint from list of locationsminRatingparameter: Quality filtering for venue search- Enhanced error handling: Improved robustness for agent use
Comprehensive assessment framework combining:
- Hard Metrics: Bayesian-weighted ratings (C=50 prior confidence), proximity scores (5-min discretization), tool compliance (geometric mean)
- LLM-as-a-Judge: Web-search enhanced dietary restriction evaluation with structured rubric
- Optimal Baselines: Systematic comparison against best possible selections from search results
- Cost Optimization: Web search caching reduces evaluation costs by 90%
- Combined Score: Geometric mean preventing single-dimension gaming
Novel experimental design systematically varying agent autonomy:
- Reveals 40% performance drop without scaffolding
- First systematic study of guidance-performance relationship
- Challenges assumptions about autonomous agent capabilities
# Generate performance plots
python3 experiments/evaluations_scripts/new_plots.py
# Analyze unfinished experiments
python3 experiments/evaluations_scripts/unfinished_experiments_analysis.py
# Token cost analysis
python3 cost_analysis/cost_analysis.py
# LLM-as-Judge evaluation
python3 LLM_as_a_judge/judge_process_logs.py <log_dir> <results_csv>This project was developed as part of my MSc in Speech and Language Processing at the University of Edinburgh (2025). The research investigates how AI agents perform on open-ended coordination tasks and explores the relationship between task scaffolding and agent autonomy. This work was supervised by PhD Alexandra Birch and PhD Patrick Chen.
Key Research Questions:
- How does reducing task guidance affect agent performance?
- When do multi-agent architectures provide advantages over single agents?
- What evaluation methods effectively measure agent capabilities on complex tasks?
Full Abstract: Understanding how AI agents perform on open-ended coordination tasks with minimal human guidance is an important evaluation challenge. This work varies information access and planning autonomy across three levels each (9 conditions total), then evaluates agents using custom reliability and performance metrics. Through 540 experiments, we identify systematic limitations including 40% performance degradation without scaffolding and persistent multi-user communication failures.
This framework serves both research and commercial needs:
- Agent benchmarking: Test any LLM agent on multi-step coordination tasks
- Tool integration testing: Evaluate how agents use custom tools (MCP)
- Multi-agent analysis: Compare single vs multi-agent architectures
Commercial Applications (via quickword.app)
- Coordination Task Automation: Production-ready agent orchestration for multi-participant planning scenarios (event scheduling, resource allocation, group decision-making).
- OctoTools Framework: github.com/octotools/octotools
- MCP Google Maps Server: @cablate/mcp-google-map (extended with custom tools)
- LangGraph: Agent orchestration and state machines
- LangChain MCP Adapters: MCP Integration
This work is licensed under Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0).
You may:
- ✅ View and review for hiring/recruitment purposes
- ✅ Use for academic research and education
- ✅ Share and adapt with attribution
You may NOT:
- ❌ Use commercially or for profit
- ❌ Sell or monetize this work
See LICENSE file for full details.
Andrés Cotton MSc Speech and Language Processing, University of Edinburgh
- 💼 Open to opportunities in AI/ML Engineering, Agent Development, and NLP
- 🔗 LinkedIn | GitHub
- 📬 Open an issue for questions about the code
This repository showcases a comprehensive AI agent evaluation framework built with modern tools (LangGraph, MCP, LLM-as-a-Judge) and demonstrates full-stack development from infrastructure (TypeScript MCP server) to evaluation (Python pipeline processing 540 experiments).