feat: add latest models, Google API support, and Openfort MCP server evals

[emauja]

Summary

• Updated model list to all latest models across 4 providers:

  • OpenAI: gpt-4.1, gpt-4.1-mini, gpt-4.1-nano, gpt-4o, gpt-5, o3, o3-mini, o4-mini
  • Anthropic: claude-opus-4-6, claude-sonnet-4-5-20250929, claude-haiku-4-5-20251001
  • Google (NEW): gemini-2.5-pro, gemini-2.5-flash
  • Vercel: v0-1.5-md

• Added Google/Gemini provider support:

  • New @ai-sdk/google dependency for Gemini models
  • GOOGLE_API_KEY environment variable
  • Google rate limiting configuration (GOOGLE_RATE_LIMIT_RPM)

• Added --provider flag to filter evals by model provider (-p openai, -p google, etc.)

• Added --mcp flag and MCP runner for tool-assisted evaluations:

  • New src/runners/mcp.ts — MCP runner that connects to Openfort MCP server and provides tools to models during evaluation
  • New src/utils/mcp-client.ts — MCP client utility using @ai-sdk/mcp and Streamable HTTP transport
  • Default MCP server URL: https://mcp.openfort.io/sse (override via MCP_SERVER_URL_OVERRIDE)
  • bun start:mcp shortcut script

• Created Openfort MCP server eval (evals/mcp-server):

  • Tests LLM ability to use Openfort MCP tools for project management, policy creation, contract registration, user/account management, and transaction simulation
  • 13 graders covering all MCP tool categories + 2 LLM-as-judge quality checks

• Infrastructure improvements:

  • Created src/runners/shared.ts with reusable runner utilities (loadPrompt, loadGraders, runGraders, computeScore)
  • Fixed Clerk references in src/graders/catalog.ts and src/scorers/constants.ts — now Openfort-specific
  • Fixed LanguageModel type compatibility with AI SDK v5
  • Updated Framework type to support 'React' | 'Node.js' | 'Next.js' | 'MCP'
  • Updated README with new models table, CLI options, and MCP documentation

Environment Variables Required

To run all models, you need:

OPENAI_API_KEY=
ANTHROPIC_API_KEY=
GOOGLE_API_KEY=      # NEW - for Gemini models
V0_API_KEY=

Test plan

• bun run lint — passes (only pre-existing warnings remain)
• bunx tsc --noEmit — TypeScript compiles cleanly
• bun start --help — shows updated help with all models and new flags
• bun start --provider google --eval evals/basic-setup — runs Google models only
• bun start --mcp --eval evals/mcp-server — runs MCP eval with tool support
• Full suite: bun start with all API keys configured

🤖 Generated with Claude Code

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	@​ai-sdk/​mcp@​0.0.12
	@​ai-sdk/​google@​3.0.34
	@​modelcontextprotocol/​sdk@​1.27.1

View full report