Merge pull request #48 from awslabs/eval_mcp

Standalone eval-mcp on PyPI + backend cleanup

Author: DataGomes

.gitignore

@@ -116,3 +116,6 @@ terraform.tfvars
 # Local tool installs (deploy.sh auto-installs terraform/kubectl/helm here)
 .tools/
 .claude/
+
+# Local eval log artifacts (synced via S3)
+logs/

.mcp.json

@@ -1,11 +1,7 @@
 {
   "mcpServers": {
-    "synthetic-eval": {
-      "command": "python",
-      "args": ["-m", "backend.mcp_servers.synthetic.server_http"],
-      "env": {
-        "PYTHONPATH": "."
-      }
+    "eval": {
+      "command": "eval-mcp"
     }
   }
 }

AGENTS.md

@@ -8,17 +8,17 @@ Users interact through a chat interface. The agent uses MCP tools in this order:
 ## Adding a new model
 
 Update both files:
-1. `backend/mcp_servers/providers/server_http.py` — add to `PROMPTFOO_SUPPORTED_MODELS`
-2. `backend/core/bedrock_pricing.json` — add pricing (per 1M tokens)
+1. `eval_mcp/tools/server_http.py` — add to `SUPPORTED_MODELS`
+2. `eval_mcp/provider_pricing.json` — add pricing (per 1M tokens)
 
 ## Key files
 
 | File | Purpose |
 |------|---------|
 | `backend/core/agent.py` | Agent system prompt and loop |
 | `backend/core/bedrock_client.py` | Bedrock client + API key auth helper |
-| `backend/core/bedrock_pricing.json` | Single source of truth for model pricing |
-| `backend/core/judge_config.py` | Default judge models and criteria |
-| `backend/mcp_servers/providers/server_http.py` | Model allowlist and discovery |
-| `backend/mcp_servers/synthetic/server_http.py` | Eval tools (QA gen, judge, config, run) |
+| `eval_mcp/server.py` | Unified MCP server (all eval tools) |
+| `eval_mcp/tools/` | Eval tools (QA gen, judge, config, run, …) |
+| `eval_mcp/provider_pricing.json` | Source of truth for model pricing |
+| `eval_mcp/core/judge_config.py` | Default judge models and criteria |
 | `Makefile` | Local dev commands (`make dev`, `make run`, `make stop`) |

Dockerfile

@@ -0,0 +1,29 @@
+FROM python:3.12-slim
+
+RUN apt-get update \
+    && apt-get install -y --no-install-recommends ca-certificates \
+    && rm -rf /var/lib/apt/lists/*
+
+# uv = fast deterministic resolver (pip's backtracking is too slow on this graph)
+COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv
+
+WORKDIR /app
+COPY pyproject.toml uv.lock ./
+COPY eval_mcp/ ./eval_mcp/
+COPY backend/ ./backend/
+
+# Slim runtime: only the core deps. Add --extra providers if you eval
+# non-Bedrock models, --extra k8s-sandbox for containerized agent evals.
+RUN uv sync --locked --no-dev
+ENV PATH="/app/.venv/bin:$PATH"
+
+# Persistent state (datasets, judges, configs, logs, reports)
+ENV EVAL_MCP_HOME=/data
+RUN mkdir -p /data
+
+EXPOSE 8002
+
+# Bedrock credentials come from the platform (instance role / IRSA / env).
+# S3 replication: set EVAL_MCP_BUCKET, or `eval-mcp config set bucket <name>`
+# inside a writable /data volume, or pre-write /data/config.json before launch.
+CMD ["eval-mcp", "serve", "--host", "0.0.0.0", "--port", "8002"]

INSTALL.md

@@ -0,0 +1,73 @@
+# For coding agents: installing eval-mcp
+
+Follow this when a user asks you to install eval-mcp into their IDE.
+
+## 1. Prerequisite check
+
+User needs `uv`. Check: `command -v uv`. If missing, tell them to run `curl -LsSf https://astral.sh/uv/install.sh | sh` (or `brew install uv` on macOS).
+
+## 2. Pick the IDE
+
+Detect from what's present in the user's home:
+
+| IDE | Check |
+|---|---|
+| Claude Code | `claude` on PATH or `~/.claude/` exists |
+| Cursor | `~/.cursor/` exists |
+| VS Code | `code` on PATH |
+| Kiro | `~/.kiro/` exists |
+| Codex CLI | `~/.codex/` exists |
+
+If multiple, ask the user which.
+
+## 3. Install
+
+Run the one-command install for that IDE:
+
+- **Claude Code**: `claude mcp add-json eval '{"type":"stdio","command":"uvx","args":["--from","llm-evaluation-system","eval-mcp"]}' --scope user`
+- **Cursor**: open `cursor://anysphere.cursor-deeplink/mcp/install?name=eval&config=eyJjb21tYW5kIjoidXZ4IiwiYXJncyI6WyItLWZyb20iLCJsbG0tZXZhbHVhdGlvbi1zeXN0ZW0iLCJldmFsLW1jcCJdfQ==` (or edit `~/.cursor/mcp.json`)
+- **VS Code**: `code --add-mcp '{"name":"eval","command":"uvx","args":["--from","llm-evaluation-system","eval-mcp"]}'`
+- **Kiro**: merge into `~/.kiro/settings/mcp.json` under `mcpServers.eval` — same JSON shape as Cursor. Back up first, atomic write.
+- **Codex**: append to `~/.codex/config.toml`: `[mcp_servers.eval]\ncommand = "uvx"\nargs = ["--from", "llm-evaluation-system", "eval-mcp"]`
+
+For any manual JSON/TOML edit: back up first, merge — never clobber, atomic write.
+
+## 4. Warm the cache
+
+Run this so the user's first IDE launch is instant instead of waiting ~60s for uvx to fetch the package:
+
+```bash
+uvx --from llm-evaluation-system eval-mcp --help >/dev/null
+```
+
+## 5. Ask about S3 team sharing
+
+> Want to share eval results, datasets, judges, and reports with teammates via a shared S3 bucket?
+
+If **yes, team already has a bucket**:
+```bash
+uvx --from llm-evaluation-system eval-mcp init <bucket-name>
+```
+
+If **yes, creating a new bucket** (first on the team):
+```bash
+git clone https://github.com/awslabs/llm-evaluation-system.git /tmp/eval-mcp-infra
+cd /tmp/eval-mcp-infra/infra/modules/eval-logs-bucket
+terraform init
+terraform apply -var="bucket_name=<bucket-name>"
+```
+Then `uvx --from llm-evaluation-system eval-mcp init <bucket-name>`. Tell the user: teammates just run `eval-mcp init <bucket-name>` — no Terraform needed.
+
+Suggested bucket name if they don't have a preference: `eval-mcp-$(aws sts get-caller-identity --query Account --output text)` — globally unique, easy to remember.
+
+If **no**: skip. They can enable it later with `eval-mcp init <bucket>`.
+
+## 6. Tell the user to restart
+
+- Claude Code / Cursor / VS Code: reload/restart the window or app
+- Kiro: save mcp.json — picks up automatically
+- Codex: must restart the CLI
+
+## Uninstall
+
+Reverse step 3 (remove the `eval` entry), restart.

README.md

@@ -1,104 +1,144 @@
+<!-- mcp-name: io.github.awslabs/llm-evaluation-system -->
+
 # Agentic AI-Guided Evaluation Platform
 
-A web-based LLM evaluation platform deployed on AWS. Access it through a browser — each user gets their own isolated workspace. An expert AI agent guides you through the entire evaluation process via natural conversation: upload your documents, describe what you want to evaluate, and the agent handles dataset generation, judge configuration, execution, and analysis.
+An LLM evaluation system where you describe what you want to evaluate in natural language — an expert AI agent handles dataset generation, judge configuration, execution, and analysis end-to-end, and hands you back a PDF report.
 
 ## Features
 
 - **Expert agent interface** — The agent knows evaluation best practices, recommends criteria and validates configurations before execution. No config files or CLI expertise needed.
-- **Jury system** — Multiple judges from different model families (e.g. Claude Sonnet, Nova Pro, Llama) each evaluate distinct aspects of every response — correctness, reasoning, completeness. Combining diverse judge families reduces self-preference bias, and aggregating weak signals from diverse judges and criteria produces stronger results than any single judge ([Verma et al., 2025](https://arxiv.org/abs/2502.20379), [Frick et al., 2025](https://arxiv.org/abs/2506.18203)).
+- **Jury system** — Multiple judges from different model families (e.g. Claude Sonnet, Nova Pro, Nemotron) each evaluate distinct aspects of every response — correctness, reasoning, completeness. Combining diverse judge families reduces self-preference bias, and aggregating weak signals from diverse judges and criteria produces stronger results than any single judge ([Verma et al., 2025](https://arxiv.org/abs/2502.20379), [Frick et al., 2025](https://arxiv.org/abs/2506.18203)).
 - **Adaptable binary scoring** — Binary pass/fail per criteria rather than subjective numeric scales, shown to produce more reliable results across judges ([Chiang et al., 2025](https://arxiv.org/abs/2503.23339v2)). Criteria are tailored by the agent to what you're evaluating.
 - **Document-grounded synthetic data** — Upload PDFs, knowledge bases, or product docs and generate QA pairs grounded in your actual content, reflecting real customer scenarios.
-- **Agentic eval support** — Evaluate agent systems via Inspect AI's `sandbox_agent_bridge` (Strands, LangChain, any OpenAI-compatible framework).
-- **Multi-tenant isolation** — Each user gets isolated data directories. Eval results stored as immutable `.eval` log files.
-- **Integrated viewer** — Inspect AI's evaluation viewer embedded directly in the platform.
+- **Agentic eval support** — Evaluate any agent calling Bedrock (Strands, LangChain, custom boto3) with zero code modification via OpenTelemetry instrumentation.
+
+## Quick Start
 
-## Prerequisites
+### Prerequisites
 
-- AWS CLI configured with credentials
-- An AWS account with Bedrock model access enabled
+- AWS credentials with Bedrock model access
+- [`uv`](https://docs.astral.sh/uv/getting-started/installation/) installed
+- Claude Code, Cursor, Kiro, VS Code, or any MCP-compatible IDE
 
-That's it. The deploy script auto-installs Terraform, kubectl, and Helm locally.
+### Install
 
-## Deploy
+Pick your IDE and paste / click.
 
+**Claude Code** — one CLI command:
 ```bash
-git clone https://github.com/awslabs/llm-evaluation-system.git && cd llm-evaluation-system
-./deploy.sh
+claude mcp add-json eval '{"type":"stdio","command":"uvx","args":["--from","llm-evaluation-system","eval-mcp"]}' --scope user
 ```
 
-The script will:
-1. Install any missing tools to `.tools/` (no sudo required)
-2. Validate AWS credentials and confirm the target account
-3. Deploy infrastructure in two layers (data, then platform)
-4. Build container images via CodeBuild
-5. Deploy the application via Helm
-6. Print the app URL and instructions for creating users
-
-## User Management
+**Cursor** — one-click deeplink: [Install eval-mcp in Cursor](cursor://anysphere.cursor-deeplink/mcp/install?name=eval&config=eyJjb21tYW5kIjoidXZ4IiwiYXJncyI6WyItLWZyb20iLCJsbG0tZXZhbHVhdGlvbi1zeXN0ZW0iLCJldmFsLW1jcCJdfQ==)
+
+**Kiro** — add to `~/.kiro/settings/mcp.json`:
+```json
+{
+  "mcpServers": {
+    "eval": {
+      "command": "uvx",
+      "args": ["--from", "llm-evaluation-system", "eval-mcp"]
+    }
+  }
+}
+```
 
-After deployment, `deploy.sh` prompts you to create an initial admin user with a temporary password shown in the terminal.
+**Codex CLI** — add to `~/.codex/config.toml`, then restart Codex:
+```toml
+[mcp_servers.eval]
+command = "uvx"
+args = ["--from", "llm-evaluation-system", "eval-mcp"]
+```
 
+**VS Code** (with GitHub Copilot MCP) — one CLI command:
 ```bash
-./manage-users.sh create user@example.com   # Create user, show temp password
-./manage-users.sh list                       # List all users
-./manage-users.sh delete user@example.com    # Delete a user
+code --add-mcp '{"name":"eval","command":"uvx","args":["--from","llm-evaluation-system","eval-mcp"]}'
 ```
 
-Users must change the temporary password on first login.
+Using a coding agent to install? Point it at [INSTALL.md](INSTALL.md) — it handles the config edit, warms the uvx cache, and asks about optional S3 team sharing.
 
-## Updating
+### Use
 
-After code changes, re-run the deploy script:
+Ask your AI assistant to evaluate agents, models, or prompts — using a dataset you provide or one generated from your documents or context:
+
+- "Evaluate my agent at `./my_agent.py`"
+- "Compare Claude Sonnet vs Nova Pro on this dataset"
+- "Test these three prompt templates against my golden QA set"
+- "Generate a dataset from this PDF and run an eval"
+
+The agent picks the right mode, auto-generates whatever's missing (dataset, judge, criteria), runs it, opens the results viewer in your browser, and hands you a PDF report.
+
+## Team Sharing (S3)
+
+Share datasets, judges, configs, and eval results across your team via a shared S3 bucket. No servers needed.
+
+### Setup
 
 ```bash
-./deploy.sh
+uvx --from llm-evaluation-system eval-mcp init my-team-evals
 ```
 
-It's fully idempotent — Terraform converges existing infrastructure, CodeBuild rebuilds images, Helm upgrades the release.
+User identity is auto-detected from your AWS credentials. Projects are auto-discovered from the bucket.
 
-## Teardown
+### How it works
 
-```bash
-./destroy.sh
+```
+s3://my-team-evals/
+  users/alice/            ← Alice's evals, datasets, judges, configs (auto-replicated on every write)
+  users/bob/              ← Bob's
+  projects/project-alpha/ ← shared team evals
+  projects/project-beta/  ← shared team evals
 ```
 
-This destroys the platform layer (EKS, CloudFront, ALB, Cognito, etc.) while **preserving data** (VPC, RDS database, S3 buckets, EBS volume). The script prints exact commands to delete preserved resources if you want a full cleanup.
+- Every write (eval result, dataset, judge, config, PDF report) auto-replicates to `users/{you}/` in the background
+- Every list/read auto-pulls from S3 first (debounced) so your local state mirrors S3
+- `eval-mcp share my-project` → promote your stuff to a shared project prefix
+- `eval-mcp sync` → manual reconcile (used after long offline periods or on a fresh laptop)
 
-To destroy the data layer as well (VPC, RDS, S3 buckets):
+### Create the bucket
+
+One person on the team runs this once:
 
 ```bash
-AWS_PROFILE=<your-profile> terraform -chdir=infra/data destroy -auto-approve
+git clone https://github.com/awslabs/llm-evaluation-system.git
+cd llm-evaluation-system/infra/modules/eval-logs-bucket
+terraform init
+terraform apply -var="bucket_name=my-team-evals"
 ```
 
-> **Note:** `destroy.sh` sets `AWS_PROFILE` internally, so it does not persist to your shell session. You must set it explicitly when running Terraform commands manually.
+## Deploy Full Platform on EKS
 
-## Security
+For a multi-user web app with Cognito auth, chat UI, and per-user isolation, the repo also ships an EKS deployment. This is the heavyweight path — for most users the MCP above is enough.
 
-All traffic is served over HTTPS through CloudFront with WAF protection. The application runs inside a private VPC — the load balancer is not exposed to the internet. Authentication is handled via Amazon Cognito. Secrets are managed through AWS Secrets Manager, and database access uses IAM authentication with no static passwords.
+```bash
+./deploy.sh
+```
 
-## Run Locally
+The script auto-installs Terraform, kubectl, and Helm, then deploys the complete platform (Cognito auth, CloudFront, WAF, per-user isolation).
 
-Test it on your machine with only Bedrock API access needed:
+### User Management
 
 ```bash
-# Using AWS SSO/profile
-AWS_PROFILE=my-profile make dev
-
-# Using a Bedrock API key (no IAM credentials needed)
-AWS_BEARER_TOKEN_BEDROCK=your-key make dev
+./manage-users.sh create user@example.com
+./manage-users.sh list
+./manage-users.sh delete user@example.com
 ```
 
-Other commands: `make stop`, `make logs`, `make logs s=backend`, `make restart s=backend`, `make build`.
+### Teardown
+
+```bash
+./destroy.sh
+```
 
-Opens at **http://localhost:4001**. See [local/README.md](local/README.md) for details.
+Architecture details, OIDC config, Helm values, and manual deployment steps: [docs/DEVELOPMENT.md](docs/DEVELOPMENT.md).
 
-## Development
+## Contributing / Local Development
 
-For architecture details, infrastructure reference, OIDC identity provider configuration, Helm chart structure, manual deployment steps, and troubleshooting, see [docs/DEVELOPMENT.md](docs/DEVELOPMENT.md).
+See [docs/DEVELOPMENT.md](docs/DEVELOPMENT.md) for how to clone, run from source, rebuild the viewer frontend, and contribute.
 
 ## Acknowledgments
 
-This platform is built on [Inspect AI](https://github.com/UKGovernmentBEIS/inspect_ai) by the UK AI Security Institute, an open-source framework for large language model evaluations.
+Built on [Inspect AI](https://github.com/UKGovernmentBEIS/inspect_ai) by the UK AI Security Institute.
 
 ## Legal Disclaimer