An LLM evaluation system where you describe what you want to evaluate in natural language — an expert AI agent handles dataset generation, judge configuration, execution, and analysis end-to-end, and hands you back a PDF report.
- Expert agent interface — The agent knows evaluation best practices, recommends criteria and validates configurations before execution. No config files or CLI expertise needed.
- Jury system — Multiple judges from different model families (e.g. Claude Sonnet, Nova Pro, Nemotron) each evaluate distinct aspects of every response — correctness, reasoning, completeness. Combining diverse judge families reduces self-preference bias, and aggregating weak signals from diverse judges and criteria produces stronger results than any single judge (Lifshitz et al., 2025, Saad-Falcon et al., 2025).
- Adaptable binary scoring — Binary pass/fail per criteria rather than subjective numeric scales, shown to produce more reliable results across judges (Mallinar et al., 2025). Criteria are tailored by the agent to what you're evaluating.
- Document-grounded synthetic data — Upload PDFs, knowledge bases, or product docs and generate QA pairs grounded in your actual content, reflecting real customer scenarios.
- Agentic eval support — Evaluate any agent calling Bedrock (Strands, LangChain, custom boto3) with zero code modification via OpenTelemetry instrumentation.
- AWS credentials with Bedrock model access
uvinstalled- Claude Code, Cursor, Kiro, VS Code, or any MCP-compatible IDE
One command — auto-detects which IDEs are on your machine (Claude Code, Kiro, VS Code, Cursor, Codex), asks which to configure, registers the MCP server in each, and warms the uvx cache:
uvx --from llm-evaluation-system eval-mcp installAdd --yes to skip prompts, --ide claude-code (or comma-separated)
to target a specific subset, --force to overwrite an existing entry.
Then restart your IDE.
Manual install (if you'd rather not use eval-mcp install)
Claude Code:
claude mcp add eval -s user -- uvx --from llm-evaluation-system eval-mcpCursor — one-click deeplink: Install eval-mcp in Cursor
Kiro — ~/.kiro/settings/mcp.json:
{"mcpServers": {"eval": {"command": "uvx", "args": ["--from", "llm-evaluation-system", "eval-mcp"]}}}Codex — ~/.codex/config.toml:
[mcp_servers.eval]
command = "uvx"
args = ["--from", "llm-evaluation-system", "eval-mcp"]VS Code:
code --add-mcp '{"name":"eval","command":"uvx","args":["--from","llm-evaluation-system","eval-mcp"]}'Using a coding agent to install? Point it at INSTALL.md — it just runs eval-mcp install --yes and asks about optional S3 team sharing.
uvx caches the resolved version per package. To pull newer releases, invalidate the cache:
uv cache clean llm-evaluation-systemRestart your IDE after. The next launch resolves and caches the newest published version.
Ask your AI assistant to evaluate agents, models, or prompts — using a dataset you provide or one generated from your documents or context:
- "Evaluate my agent at
./my_agent.py" - "Compare Claude Sonnet vs Nova Pro on this dataset"
- "Test these three prompt templates against my golden QA set"
- "Generate a dataset from this PDF and run an eval"
The agent picks the right mode, auto-generates whatever's missing (dataset, judge, criteria), runs it, opens the results viewer in your browser, and hands you a PDF report.
Share datasets, judges, configs, and eval results across your team via a shared S3 bucket. No servers needed.
uvx --from llm-evaluation-system eval-mcp init my-team-evalsUser identity is auto-detected from your AWS credentials. Projects are auto-discovered from the bucket.
s3://my-team-evals/
users/alice/ ← Alice's evals, datasets, judges, configs (auto-replicated on every write)
users/bob/ ← Bob's
projects/project-alpha/ ← shared team evals
projects/project-beta/ ← shared team evals
- Every write (eval result, dataset, judge, config, PDF report) auto-replicates to
users/{you}/in the background - Every list/read auto-pulls from S3 first (debounced) so your local state mirrors S3
eval-mcp share my-project→ promote your stuff to a shared project prefixeval-mcp sync→ manual reconcile (used after long offline periods or on a fresh laptop)
One person on the team runs this once:
git clone https://github.com/awslabs/llm-evaluation-system.git
cd llm-evaluation-system/infra/modules/eval-logs-bucket
terraform init
terraform apply -var="bucket_name=my-team-evals"For a multi-user web app with Cognito auth, chat UI, and per-user isolation, the repo also ships an EKS deployment. This is the heavyweight path — for most users the MCP above is enough.
./deploy.shThe script auto-installs Terraform, kubectl, and Helm, then deploys the complete platform (Cognito auth, CloudFront, WAF, per-user isolation).
./manage-users.sh create user@example.com
./manage-users.sh list
./manage-users.sh delete user@example.com./destroy.shArchitecture details, OIDC config, Helm values, and manual deployment steps: docs/DEVELOPMENT.md.
See ROADMAP.md for what's planned next and where contributions are most welcome.
See docs/DEVELOPMENT.md for how to clone, run from source, rebuild the viewer frontend, and contribute.
Built on Inspect AI by the UK AI Security Institute.
Sample code, software libraries, command line tools, proofs of concept, templates, or other related technology are provided as AWS Content or Third-Party Content under the AWS Customer Agreement, or the relevant written agreement between you and AWS (whichever applies). You should not use this AWS Content or Third-Party Content in your production accounts, or on production or other critical data. You are responsible for testing, securing, and optimizing the AWS Content or Third-Party Content, such as sample code, as appropriate for production grade use based on your specific quality control practices and standards. Deploying AWS Content or Third-Party Content may incur AWS charges for creating or using AWS chargeable resources, such as running Amazon EC2 instances or using Amazon S3 storage.