Add Cocoon Stack Gateway to the AI gateway benchmarks

.github/workflows/ai-gateway-benchmarks.yml

@@ -159,7 +159,7 @@ jobs:
           INPUT_ITERATIONS: ${{ github.event.inputs.iterations }}
           FAMILY_ITERATIONS: ${{ matrix.family.iterations }}
         run: |
-          . benchmarks/scripts/load-vault-secrets.sh '^(ANTHROPIC_API_KEY|OPENAI_API_KEY|GEMINI_API_KEY|KIMI_API_KEY|GITHUB_COPILOT_API_KEY|NOVITA_API_KEY|NEON_AI_GATEWAY_BASE_URL|NEON_AI_GATEWAY_TOKEN|NGROK_AI_GATEWAY_API_KEY|LLMAPI_API_KEY|CLOUDFLARE_AI_GATEWAY_ACCOUNT_ID|CLOUDFLARE_AI_GATEWAY_GATEWAY_ID|CLOUDFLARE_AI_GATEWAY_TOKEN|LLM_GATEWAY_API_KEY|OPENROUTER_API_KEY|PYDANTIC_AI_GATEWAY_API_KEY|VERCEL_AI_GATEWAY_API_KEY|CONCENTRATE_AI_GATEWAY_API_KEY|BLAZERAIL_API_KEY|BENCHMARKS_PLATFORM_API_KEY)'
+          . benchmarks/scripts/load-vault-secrets.sh '^(ANTHROPIC_API_KEY|OPENAI_API_KEY|GEMINI_API_KEY|KIMI_API_KEY|GITHUB_COPILOT_API_KEY|NOVITA_API_KEY|NEON_AI_GATEWAY_BASE_URL|NEON_AI_GATEWAY_TOKEN|NGROK_AI_GATEWAY_API_KEY|LLMAPI_API_KEY|CLOUDFLARE_AI_GATEWAY_ACCOUNT_ID|CLOUDFLARE_AI_GATEWAY_GATEWAY_ID|CLOUDFLARE_AI_GATEWAY_TOKEN|LLM_GATEWAY_API_KEY|OPENROUTER_API_KEY|PYDANTIC_AI_GATEWAY_API_KEY|VERCEL_AI_GATEWAY_API_KEY|CONCENTRATE_AI_GATEWAY_API_KEY|BLAZERAIL_API_KEY|COCOONSTACK_GATEWAY_API_KEY|BENCHMARKS_PLATFORM_API_KEY)'
 
           # GitHub Copilot is kept in the daily runs but excluded from the weekly run.
           if [ "${{ github.event_name }}" = "schedule" ]; then

AI_GATEWAYS.md

@@ -2,11 +2,11 @@
 
 This document describes the **AI gateway benchmark** — a phase-by-phase latency, throughput, and reliability comparison of OpenRouter, Vercel AI Gateway, Cloudflare AI Gateway, LLM Gateway, Pydantic AI Gateway, Concentrate AI, Novita, Ramp Router, Neon AI Gateway, ngrok AI Gateway, and LLM API. It's organized as one **family benchmark per target provider**, all built on the same shared task plumbing (`shared-task.ts`, `phase-probe.ts`, scoring) so every family uses an identical prompt, phase methodology, and scoring formula — only the target provider (and therefore the model and each gateway's routing syntax) changes between them:
 
-- **Anthropic family** (`ai-gateway.bench.ts` + `providers.ts`) — ten gateways (OpenRouter, Vercel AI Gateway, Cloudflare AI Gateway, LLM Gateway, Pydantic AI Gateway, Concentrate AI, Ramp Router, Neon AI Gateway, ngrok AI Gateway, and LLM API, plus anthropic-direct as a direct baseline; Novita excluded — see below) routed to Claude Haiku 4.5, measured against a direct-to-Anthropic baseline. This is the original benchmark and the one with the deepest "confirmed live" verification (see [Every gateway is hit directly](#every-gateway-is-hit-directly--no-gateway-is-proxied-through-another) below).
+- **Anthropic family** (`ai-gateway.bench.ts` + `providers.ts`) — eleven gateways (Cocoon Stack Gateway, OpenRouter, Vercel AI Gateway, Cloudflare AI Gateway, LLM Gateway, Pydantic AI Gateway, Concentrate AI, Ramp Router, Neon AI Gateway, ngrok AI Gateway, and LLM API, plus anthropic-direct as a direct baseline; Novita excluded — see below) routed to Claude Haiku 4.5, measured against a direct-to-Anthropic baseline. This is the original benchmark and the one with the deepest "confirmed live" verification (see [Every gateway is hit directly](#every-gateway-is-hit-directly--no-gateway-is-proxied-through-another) below).
 
-- **OpenAI family** (`ai-gateway-openai.bench.ts` + `providers-openai.ts`) — ten gateways (OpenRouter, Vercel AI Gateway, Cloudflare AI Gateway, LLM Gateway, Concentrate AI, Pydantic AI Gateway, Ramp Router, Neon AI Gateway, ngrok AI Gateway, and LLM API; Novita excluded — see below) routed to `gpt-5.4-mini` instead, measured against a direct-to-OpenAI baseline. See [OpenAI family benchmark](#openai-family-benchmark).
-- **Gemini family** (`ai-gateway-gemini.bench.ts` + `providers-gemini.ts`) — eight gateways (OpenRouter, Vercel AI Gateway, Cloudflare AI Gateway, LLM Gateway, Concentrate AI, Neon AI Gateway, ngrok AI Gateway, and LLM API; Pydantic, Novita, and Ramp Router excluded — see below) routed to `gemini-3.6-flash`, measured against a direct-to-Gemini baseline. See [Gemini family benchmark](#gemini-family-benchmark).
-- **Kimi family** (`ai-gateway-kimi.bench.ts` + `providers-kimi.ts`) — ten gateways (OpenRouter, Vercel AI Gateway, Cloudflare AI Gateway, LLM Gateway, Concentrate AI, Novita, Ramp Router, Neon AI Gateway, ngrok AI Gateway, and LLM API; Pydantic excluded — see below) routed to `kimi-k3`, measured against a direct-to-Moonshot baseline. See [Kimi family benchmark](#kimi-family-benchmark).
+- **OpenAI family** (`ai-gateway-openai.bench.ts` + `providers-openai.ts`) — eleven gateways (Cocoon Stack Gateway, OpenRouter, Vercel AI Gateway, Cloudflare AI Gateway, LLM Gateway, Concentrate AI, Pydantic AI Gateway, Ramp Router, Neon AI Gateway, ngrok AI Gateway, and LLM API; Novita excluded — see below) routed to `gpt-5.4-mini` instead, measured against a direct-to-OpenAI baseline. See [OpenAI family benchmark](#openai-family-benchmark).
+- **Gemini family** (`ai-gateway-gemini.bench.ts` + `providers-gemini.ts`) — nine gateways (Cocoon Stack Gateway, OpenRouter, Vercel AI Gateway, Cloudflare AI Gateway, LLM Gateway, Concentrate AI, Neon AI Gateway, ngrok AI Gateway, and LLM API; Pydantic, Novita, and Ramp Router excluded — see below) routed to `gemini-3.6-flash`, measured against a direct-to-Gemini baseline. See [Gemini family benchmark](#gemini-family-benchmark).
+- **Kimi family** (`ai-gateway-kimi.bench.ts` + `providers-kimi.ts`) — eleven gateways (Cocoon Stack Gateway, OpenRouter, Vercel AI Gateway, Cloudflare AI Gateway, LLM Gateway, Concentrate AI, Novita, Ramp Router, Neon AI Gateway, ngrok AI Gateway, and LLM API; Pydantic excluded — see below) routed to `kimi-k3`, measured against a direct-to-Moonshot baseline. See [Kimi family benchmark](#kimi-family-benchmark).
 
 A result from one family is **not** directly comparable to the same gateway's result in another family — different target provider means a different underlying model and (for some gateways) a different routing path, so a difference in numbers can't be attributed to the gateway alone the way it can within a single family.
 
@@ -277,7 +277,7 @@ npx tsx benchmarks/ai-gateway/ai-gateway.bench.ts --ai-gateway-iterations-cold 2
 npx tsx benchmarks/ai-gateway/ai-gateway-openai.bench.ts --ai-gateway-iterations-cold 20 --ai-gateway-iterations-warm 0
 ```
 
-Required environment variables (`benchmarks/.env.example`): `OPENROUTER_API_KEY`, `VERCEL_AI_GATEWAY_API_KEY`, `LLM_GATEWAY_API_KEY`, `CONCENTRATE_AI_GATEWAY_API_KEY` are shared across all four families (same gateway accounts, just routed to a different target model each time). `CLOUDFLARE_AI_GATEWAY_ACCOUNT_ID`, `CLOUDFLARE_AI_GATEWAY_GATEWAY_ID`, and `CLOUDFLARE_AI_GATEWAY_TOKEN` are all **required** by every family's Cloudflare entry (the Kimi family's entry uses a different Cloudflare endpoint than the other three, and is the only one where the token is the *sole* auth — no separate per-family key is forwarded there — but all four now require Authenticated Gateway mode, not just Kimi). `NEON_AI_GATEWAY_BASE_URL` and `NEON_AI_GATEWAY_TOKEN` are used across all four families. `NGROK_AI_GATEWAY_API_KEY` is used by all four families (Anthropic, OpenAI, Gemini, and Kimi). `LLMAPI_API_KEY` is used by all four families (Anthropic, OpenAI, Gemini, and Kimi). `PYDANTIC_AI_GATEWAY_API_KEY` is used by the Anthropic and OpenAI families — the Gemini and Kimi families exclude Pydantic. `NOVITA_API_KEY` is used by the Anthropic and Kimi families — OpenAI and Gemini exclude Novita. `RAMP_ROUTER_API_KEY` is used by the Anthropic, OpenAI, and Kimi families — Gemini excludes Ramp Router. `ANTHROPIC_API_KEY` is used by the Anthropic family (Cloudflare's passthrough + `anthropic-direct`); `OPENAI_API_KEY` by the OpenAI family (Cloudflare's passthrough + `openai-direct`); `GEMINI_API_KEY` by the Gemini family (Cloudflare's passthrough + `gemini-direct`); `KIMI_API_KEY` by the Kimi family (`kimi-direct` only — Cloudflare's Kimi entry doesn't need it). Missing credentials cause that participant to be reported as `SKIPPED` rather than failing the run.
+Required environment variables (`benchmarks/.env.example`): `OPENROUTER_API_KEY`, `VERCEL_AI_GATEWAY_API_KEY`, `LLM_GATEWAY_API_KEY`, `CONCENTRATE_AI_GATEWAY_API_KEY` are shared across all four families (same gateway accounts, just routed to a different target model each time). `CLOUDFLARE_AI_GATEWAY_ACCOUNT_ID`, `CLOUDFLARE_AI_GATEWAY_GATEWAY_ID`, and `CLOUDFLARE_AI_GATEWAY_TOKEN` are all **required** by every family's Cloudflare entry (the Kimi family's entry uses a different Cloudflare endpoint than the other three, and is the only one where the token is the *sole* auth — no separate per-family key is forwarded there — but all four now require Authenticated Gateway mode, not just Kimi). `NEON_AI_GATEWAY_BASE_URL` and `NEON_AI_GATEWAY_TOKEN` are used across all four families. `NGROK_AI_GATEWAY_API_KEY` is used by all four families (Anthropic, OpenAI, Gemini, and Kimi). `COCOONSTACK_GATEWAY_API_KEY` is used by all four families (Cocoon Stack Gateway speaks the Anthropic Messages and OpenAI Responses wires natively and the OpenAI chat wire for the Gemini and Kimi families). `LLMAPI_API_KEY` is used by all four families (Anthropic, OpenAI, Gemini, and Kimi). `PYDANTIC_AI_GATEWAY_API_KEY` is used by the Anthropic and OpenAI families — the Gemini and Kimi families exclude Pydantic. `NOVITA_API_KEY` is used by the Anthropic and Kimi families — OpenAI and Gemini exclude Novita. `RAMP_ROUTER_API_KEY` is used by the Anthropic, OpenAI, and Kimi families — Gemini excludes Ramp Router. `ANTHROPIC_API_KEY` is used by the Anthropic family (Cloudflare's passthrough + `anthropic-direct`); `OPENAI_API_KEY` by the OpenAI family (Cloudflare's passthrough + `openai-direct`); `GEMINI_API_KEY` by the Gemini family (Cloudflare's passthrough + `gemini-direct`); `KIMI_API_KEY` by the Kimi family (`kimi-direct` only — Cloudflare's Kimi entry doesn't need it). Missing credentials cause that participant to be reported as `SKIPPED` rather than failing the run.
 
 ## Output
 

benchmarks/.env.example

@@ -177,3 +177,4 @@ NEON_AI_GATEWAY_TOKEN=your_neon_ai_gateway_token
 NGROK_AI_GATEWAY_API_KEY=your_ngrok_ai_gateway_api_key
 LLMAPI_API_KEY=your_llmapi_api_key
 BLAZERAIL_API_KEY=your_blazerail_api_key
+COCOONSTACK_GATEWAY_API_KEY=your_cocoonstack_gateway_api_key

benchmarks/ai-gateway/providers-gemini.ts

@@ -36,6 +36,15 @@ import { resolveNeonHost } from './neon-host.js';
  * family's entries in `providers.ts` were held to.
  */
 export const providers: AIGatewayProviderConfig[] = [
+  {
+    name: 'cocoonstack-gateway',
+    requiredEnvVars: ['COCOONSTACK_GATEWAY_API_KEY'],
+    wireFormat: 'openai',
+    model: 'gemini-3.6-flash',
+    host: 'gw-bench.simular.cloud',
+    path: '/v1/chat/completions',
+    buildHeaders: () => ({ Authorization: `Bearer ${process.env.COCOONSTACK_GATEWAY_API_KEY}` }),
+  },
   {
     // No native Gemini passthrough documented for OpenRouter — its whole
     // platform is built around one normalized, OpenAI-Chat-Completions-

benchmarks/ai-gateway/providers-kimi.ts

@@ -84,6 +84,17 @@ import { resolveNeonHost } from './neon-host.js';
  * OpenRouter, and Vercel here are the exceptions — all confirmed live).
  */
 export const providers: AIGatewayProviderConfig[] = [
+  {
+    name: 'cocoonstack-gateway',
+    requiredEnvVars: ['COCOONSTACK_GATEWAY_API_KEY'],
+    wireFormat: 'openai',
+    model: 'kimi-k3',
+    host: 'gw-bench.simular.cloud',
+    path: '/v1/chat/completions',
+    buildHeaders: () => ({ Authorization: `Bearer ${process.env.COCOONSTACK_GATEWAY_API_KEY}` }),
+    extraBody: { temperature: undefined },
+    reasoningCountsAsFirstToken: true,
+  },
   {
     // `moonshotai/kimi-k3` is OpenRouter's confirmed catalog convention
     // (openrouter.ai/moonshotai). Provider-order pinning to `moonshotai`

benchmarks/ai-gateway/providers-openai.ts

@@ -30,6 +30,15 @@ import { resolveNeonHost } from './neon-host.js';
  * family's entries in `providers.ts` were held to.
  */
 export const providers: AIGatewayProviderConfig[] = [
+  {
+    name: 'cocoonstack-gateway',
+    requiredEnvVars: ['COCOONSTACK_GATEWAY_API_KEY'],
+    wireFormat: 'responses',
+    model: 'gpt-5.4-mini',
+    host: 'gw-bench.simular.cloud',
+    path: '/v1/responses',
+    buildHeaders: () => ({ Authorization: `Bearer ${process.env.COCOONSTACK_GATEWAY_API_KEY}` }),
+  },
   {
     // Confirmed: OpenRouter's Responses API passthrough exists at
     // `/api/v1/responses` (openrouter.ai/docs/api_reference/responses/overview),

benchmarks/ai-gateway/providers.ts

@@ -11,6 +11,18 @@ import { resolveNeonHost } from './neon-host.js';
  * latency each gateway adds on top of the underlying provider.
  */
 export const providers: AIGatewayProviderConfig[] = [
+  {
+    name: 'cocoonstack-gateway',
+    requiredEnvVars: ['COCOONSTACK_GATEWAY_API_KEY'],
+    wireFormat: 'anthropic',
+    model: 'claude-haiku-4-5-20251001',
+    host: 'gw-bench.simular.cloud',
+    path: '/v1/messages',
+    buildHeaders: () => ({
+      'x-api-key': process.env.COCOONSTACK_GATEWAY_API_KEY || '',
+      'anthropic-version': '2023-06-01',
+    }),
+  },
   {
     // `anthropic/claude-haiku-4.5` is a catalog alias that OpenRouter can
     // serve from more than one upstream (Anthropic direct, Bedrock, Vertex,

benchmarks/scripts/provider-vars.json

@@ -287,6 +287,9 @@
     ],
     "anthropic-direct": [
       "ANTHROPIC_API_KEY"
+    ],
+    "cocoonstack-gateway": [
+      "COCOONSTACK_GATEWAY_API_KEY"
     ]
   },
   "ai-gateway-openai": {
@@ -329,6 +332,9 @@
     ],
     "openai-direct": [
       "OPENAI_API_KEY"
+    ],
+    "cocoonstack-gateway": [
+      "COCOONSTACK_GATEWAY_API_KEY"
     ]
   },
   "ai-gateway-gemini": {
@@ -365,6 +371,9 @@
     ],
     "gemini-direct": [
       "GEMINI_API_KEY"
+    ],
+    "cocoonstack-gateway": [
+      "COCOONSTACK_GATEWAY_API_KEY"
     ]
   },
   "ai-gateway-kimi": {
@@ -409,6 +418,9 @@
     ],
     "kimi-direct": [
       "KIMI_API_KEY"
+    ],
+    "cocoonstack-gateway": [
+      "COCOONSTACK_GATEWAY_API_KEY"
     ]
   },
   "snapshot-fork": {

package.json

@@ -83,6 +83,7 @@
     "generate-browser-throughput-svg": "tsx benchmarks/browser/generate-throughput-svg.ts",
     "generate-pricing-svg": "tsx benchmarks/sandbox/generate-pricing-svg.ts",
     "bench:ai-gateway": "tsx packages/benchsdk-runner/dist/bin.js run benchmarks/ai-gateway/ai-gateway.bench.ts",
+    "bench:ai-gateway:cocoonstack-gateway": "tsx packages/benchsdk-runner/dist/bin.js run benchmarks/ai-gateway/ai-gateway.bench.ts --provider cocoonstack-gateway",
     "bench:ai-gateway:github-copilot": "tsx packages/benchsdk-runner/dist/bin.js run benchmarks/ai-gateway/ai-gateway.bench.ts --provider github-copilot",
     "bench:ai-gateway:openrouter": "tsx packages/benchsdk-runner/dist/bin.js run benchmarks/ai-gateway/ai-gateway.bench.ts --provider openrouter",
     "bench:ai-gateway:vercel": "tsx packages/benchsdk-runner/dist/bin.js run benchmarks/ai-gateway/ai-gateway.bench.ts --provider vercel-ai-gateway",
@@ -97,6 +98,7 @@
     "bench:ai-gateway:ramp": "tsx packages/benchsdk-runner/dist/bin.js run benchmarks/ai-gateway/ai-gateway.bench.ts --provider ramp",
     "generate-ai-gateway-svg": "tsx benchmarks/ai-gateway/generate-svg.ts",
     "bench:ai-gateway-openai": "tsx packages/benchsdk-runner/dist/bin.js run benchmarks/ai-gateway/ai-gateway-openai.bench.ts",
+    "bench:ai-gateway-openai:cocoonstack-gateway": "tsx packages/benchsdk-runner/dist/bin.js run benchmarks/ai-gateway/ai-gateway-openai.bench.ts --provider cocoonstack-gateway",
     "bench:ai-gateway-openai:github-copilot": "tsx packages/benchsdk-runner/dist/bin.js run benchmarks/ai-gateway/ai-gateway-openai.bench.ts --provider github-copilot",
     "bench:ai-gateway-openai:openrouter": "tsx packages/benchsdk-runner/dist/bin.js run benchmarks/ai-gateway/ai-gateway-openai.bench.ts --provider openrouter",
     "bench:ai-gateway-openai:vercel": "tsx packages/benchsdk-runner/dist/bin.js run benchmarks/ai-gateway/ai-gateway-openai.bench.ts --provider vercel-ai-gateway",
@@ -111,6 +113,7 @@
     "bench:ai-gateway-openai:llmapi": "tsx packages/benchsdk-runner/dist/bin.js run benchmarks/ai-gateway/ai-gateway-openai.bench.ts --provider llmapi",
     "generate-ai-gateway-openai-svg": "tsx benchmarks/ai-gateway/generate-svg.ts --dir ai-gateway-latency/openai --title \"AI Gateway Benchmarks — OpenAI\" --out ai-gateway-openai.svg",
     "bench:ai-gateway-gemini": "tsx packages/benchsdk-runner/dist/bin.js run benchmarks/ai-gateway/ai-gateway-gemini.bench.ts",
+    "bench:ai-gateway-gemini:cocoonstack-gateway": "tsx packages/benchsdk-runner/dist/bin.js run benchmarks/ai-gateway/ai-gateway-gemini.bench.ts --provider cocoonstack-gateway",
     "bench:ai-gateway-gemini:openrouter": "tsx packages/benchsdk-runner/dist/bin.js run benchmarks/ai-gateway/ai-gateway-gemini.bench.ts --provider openrouter",
     "bench:ai-gateway-gemini:vercel": "tsx packages/benchsdk-runner/dist/bin.js run benchmarks/ai-gateway/ai-gateway-gemini.bench.ts --provider vercel-ai-gateway",
     "bench:ai-gateway-gemini:cloudflare": "tsx packages/benchsdk-runner/dist/bin.js run benchmarks/ai-gateway/ai-gateway-gemini.bench.ts --provider cloudflare-ai-gateway",
@@ -122,6 +125,7 @@
     "bench:ai-gateway-gemini:llmapi": "tsx packages/benchsdk-runner/dist/bin.js run benchmarks/ai-gateway/ai-gateway-gemini.bench.ts --provider llmapi",
     "generate-ai-gateway-gemini-svg": "tsx benchmarks/ai-gateway/generate-svg.ts --dir ai-gateway-latency/gemini --title \"AI Gateway Benchmarks — Gemini\" --out ai-gateway-gemini.svg",
     "bench:ai-gateway-kimi": "tsx packages/benchsdk-runner/dist/bin.js run benchmarks/ai-gateway/ai-gateway-kimi.bench.ts",
+    "bench:ai-gateway-kimi:cocoonstack-gateway": "tsx packages/benchsdk-runner/dist/bin.js run benchmarks/ai-gateway/ai-gateway-kimi.bench.ts --provider cocoonstack-gateway",
     "bench:ai-gateway-kimi:github-copilot": "tsx packages/benchsdk-runner/dist/bin.js run benchmarks/ai-gateway/ai-gateway-kimi.bench.ts --provider github-copilot",
     "bench:ai-gateway-kimi:openrouter": "tsx packages/benchsdk-runner/dist/bin.js run benchmarks/ai-gateway/ai-gateway-kimi.bench.ts --provider openrouter",
     "bench:ai-gateway-kimi:vercel": "tsx packages/benchsdk-runner/dist/bin.js run benchmarks/ai-gateway/ai-gateway-kimi.bench.ts --provider vercel-ai-gateway",