benchmark-broccoli

A transparent Model Context Protocol (MCP) proxy that sits between your AI client and any MCP server, measuring tokens, estimated cost, and latency for every tool call. Metrics stream to a live dashboard with per-query session tracking and multi-model cost comparison.

Features

• Works with any MCP server — filesystem, database, GraphQL, custom servers
• Works with any AI client — Cursor, VS Code, OpenCode, Claude Desktop, custom agents
• Per-query session tracking — groups tool calls by prompt/query with configurable time gap
• Schema overhead attribution — full listTools() payload token count, booked once per proxy process (first tool call)
• Per-user cost tracking — MCP_USER in repo .env; shown per call on the dashboard
• Multi-model cost comparison — switch models in the dashboard and see costs recalculated live
• Accurate tokenization — uses tiktoken (o200k_base) with char-length fallback
• Live dashboard — auto-updating via Server-Sent Events, dark theme, sessions view, per-tool breakdown
• Export — download call history as CSV for external analysis
• JSONL log — append-only calls.jsonl for downstream analysis

Quick Start

1. Install

git clone https://github.com/Shriya-Chauhan/benchmark-broccoli.git
cd benchmark-broccoli
npm install

Optional: copy .env.example to .env and adjust COST_MODEL, MCP_USER, ports, etc.

2. Configure your AI client

Point your client's MCP config at the proxy, passing the real server command after --.

Cursor (~/.cursor/mcp.json):

{
  "mcpServers": {
    "my-server": {
      "command": "npx",
      "args": [
        "tsx", "/absolute/path/to/benchmark-broccoli/src/index.ts",
        "--",
        "npx", "-y", "mcp-remote", "https://my-mcp-server.example.com/mcp"
      ],
      "env": {
        "COST_MODEL": "claude-sonnet-4-20250514"
      }
    }
  }
}

3. Start the dashboard

npm start
# → [dashboard] http://127.0.0.1:3000

Open http://127.0.0.1:3000 in your browser. Use your AI client normally — every tool call appears on the dashboard in real time, grouped by session.

Set MCP_USER in .env (repo root); restart the MCP server in your client after you change it.

Sessions

The proxy automatically groups tool calls into sessions using a time-gap heuristic. When your AI client processes a prompt, it typically makes several tool calls in quick succession. If no call arrives within SESSION_GAP_MS (default 30 seconds), the next call starts a new session.

Each session tracks:

• Schema overhead — token cost of the full listTools() payload, attributed on the first tool call after the proxy starts only (not repeated when a new session begins after an idle gap)
• Call sequence — numbered tool calls within the session
• Client name — filled when the SDK exposes the host clientInfo (often empty; not reliable)
• User — whatever MCP_USER was when that call was recorded (usually from .env)
• Aggregated cost — total cost across all calls in the session

The dashboard shows sessions as collapsible cards. Click to expand and see individual calls.

Built-in Model Pricing

Model	Input / 1M	Output / 1M
claude-sonnet-4-20250514	$3.00	$15.00
claude-opus-4-20250514	$15.00	$75.00
claude-3-5-sonnet-20241022	$3.00	$15.00
claude-3-5-haiku-20241022	$0.80	$4.00
gpt-4o	$2.50	$10.00
gpt-4o-mini	$0.15	$0.60
gpt-4.1	$2.00	$8.00
gpt-4.1-mini	$0.40	$1.60
gpt-4.1-nano	$0.10	$0.40
gemini-2.5-pro	$1.25	$10.00
gemini-2.5-flash	$0.15	$0.60

Repository layout

Path	Role
src/index.ts	MCP stdio proxy → upstream child; writes calls.jsonl
src/dashboard.ts	Express app + SSE; reads same JSONL
src/store.ts	JSONL path, append/load/clear/delete session
src/cost.ts	Model pricing + estimateCost
src/tokenizer.ts	tiktoken / fallback counting
src/load-repo-env.ts	Optional repo .env (import first from entrypoints)
src/public/index.html	Single-page dashboard (vanilla JS)

Contributing

Contributions are welcome! Please open an issue or pull request.

1. Fork the repo
2. Create a feature branch (git checkout -b feature/my-change)
3. Run npm test and npm run typecheck
4. Push and open a PR

License

GNU General Public License v3.0 only — Copyright 2026 Shriya Chauhan