Evaluate any agent on any benchmark in the simplest way possible
Exgentic is a universal evaluation framework that enables standardized testing of AI agents across diverse benchmarks and domains. It provides a consistent interface for evaluating any agent on any benchmark, making it easy to compare performance, reproduce results, and ensure your agent works reliably across different tasks and environments.
- General Audience - Visit www.exgentic.ai to explore the first general agent leaderboard comparing leading agents and frontier models across varied tasks.
- Agent Builders - Evaluate your agents comprehensively across multiple domains and benchmarks.
- Researchers & Component Developers - Test agentic components (memory, context compression, planning) across different agents and domains.
- Benchmark Builders - Evaluate your benchmark across multiple agents to ensure meaningful differentiation.
uv tool install exgenticexport OPENAI_API_KEY=...
# or
export ANTHROPIC_API_KEY=...# List available benchmarks and agents
exgentic list benchmarks
exgentic list agents
# Evaluate an agent on a benchmark
exgentic evaluate --benchmark tau2 --agent tool_calling --subset retail --num-tasks 2 \
--model gpt-4o \
--set benchmark.user_simulator_model="gpt-4o"Benchmarks are automatically installed on first run — no manual installation needed. You can also install them explicitly:
exgentic install --benchmark tau2 # install deps + data (default)
exgentic install --agent tool_calling
exgentic install --benchmark tau2 --docker # build Docker image
exgentic install --benchmark tau2 --local # install into local environment
exgentic uninstall --benchmark tau2 # remove installed environmentNote:
exgentic setupstill works but is deprecated in favor ofinstall/uninstall.
For full container isolation, use the Docker runner (--set benchmark.runner=docker). You only need Docker installed and running:
exgentic evaluate --benchmark tau2 --agent tool_calling --subset retail --num-tasks 2 \
--model gpt-4o \
--set benchmark.runner=docker \
--set benchmark.user_simulator_model="gpt-4o"To use exgentic as a library, install it first:
uv add exgentic # or: pip install exgenticfrom exgentic import evaluate
results = evaluate(
benchmark="tau2",
agent="tool_calling",
subset="retail",
num_tasks=2,
model="gpt-4o",
benchmark_kwargs={"user_simulator_model": "gpt-4o"},
)For more examples, see the examples/ directory.
exgentic list benchmarks| Benchmark | Description |
|---|---|
| tau2 | Simulated customer support tasks across multiple domains (mock, retail, airline, telecom) |
| appworld | Multi-app API environment testing agents' ability to interact with application interfaces |
| browsecompplus | Web search and browsing benchmark for information retrieval and navigation |
| swebench | Software engineering benchmark for resolving real-world GitHub issues |
| hotpotqa | Multi-hop question answering over Wikipedia |
| gsm8k | Grade school math word problems with optional calculator tool |
| bfcl | Berkeley Function Calling Leaderboard for evaluating tool-use capabilities |
| Agent | Description |
|---|---|
| LiteLLM Tool Calling | Generic tool-calling agent via LiteLLM |
| SmolAgents | HuggingFace SmolAgents framework |
| OpenAI MCP | OpenAI Responses API with MCP tools |
| Claude Code | Anthropic Claude Code agent |
| Codex CLI | OpenAI Codex CLI agent |
| Gemini CLI | Google Gemini CLI agent |
exgentic dashboardEach run creates its own directory under outputs/<run_id>/:
outputs/<run_id>/
├── results.json # Overall scores, costs, per-session statistics
├── benchmark_results.json # Benchmark-specific aggregated results
├── run/
│ ├── config.json # Snapshot of benchmark and agent configuration
│ ├── run.log # Main execution log
│ └── warnings.log # Warnings during execution
└── sessions/<session_id>/
├── config.json # Session configuration
├── results.json # Session results
├── trajectory.jsonl # One JSON line per step (action + observation)
├── agent/
│ └── agent.log # Agent execution log
└── benchmark/
├── results.json # Benchmark-specific results
└── session.log # Benchmark session log
# Discover
exgentic list benchmarks
exgentic list subsets --benchmark tau2
exgentic list tasks --benchmark tau2 --subset retail --limit 5
exgentic list agents
exgentic install --benchmark tau2
exgentic install --benchmark tau2 --docker
exgentic install --benchmark tau2 --local
exgentic uninstall --benchmark tau2
# Run
exgentic evaluate --benchmark tau2 --agent tool_calling --subset airline --num-tasks 10
exgentic batch run --benchmark tau2 --agent tool_calling --subset airline --num-tasks 10
# Inspect
exgentic status --benchmark tau2 --agent tool_calling --subset airline --num-tasks 10
exgentic preview --benchmark tau2 --agent tool_calling --subset airline --num-tasks 10
exgentic results --benchmark tau2 --agent tool_calling --subset airline --num-tasks 10
# Analyze
exgentic compare --agents tool_calling openai --benchmark tau2
# Explore
exgentic dashboardexgentic evaluate --benchmark tau2 --agent tool_calling --subset retail --num-tasks 2 \
--set agent.model.temperature=0.2Supported fields: temperature, top_p, max_tokens, reasoning_effort, num_retries, retry_after, retry_strategy
exgentic evaluate --benchmark tau2 --agent tool_calling --subset retail --num-tasks 2 \
--max-steps 100 --max-actions 100Sessions stop at either limit and record limit_reached status. Default: 100 for both.
Use HuggingFace models or run evaluations on HuggingFace Jobs. See docs/huggingface.md.
To learn more about Exgentic's architecture and design, see our arXiv paper.
For local development, editing, and contributing, see DEVELOPMENT.md.
We welcome issues and pull requests! See CONTRIBUTING.md for guidelines.
@misc{bandel2026generalagentevaluation,
title={General Agent Evaluation},
author={Elron Bandel and Asaf Yehudai and Lilach Eden and Yehoshua Sagron and Yotam Perlitz and Elad Venezian and Natalia Razinkov and Natan Ergas and Shlomit Shachor Ifergan and Segev Shlomov and Michal Jacovi and Leshem Choshen and Liat Ein-Dor and Yoav Katz and Michal Shmueli-Scheuer},
year={2026},
url={https://arxiv.org/abs/2602.22953},
}Apache License 2.0 — see LICENSE.
For questions and support, open an issue on GitHub.