From 748d33336a45fe8e62f52803b0d35bca7a953dd2 Mon Sep 17 00:00:00 2001 From: CMGS Date: Thu, 10 Sep 2026 19:43:08 +0900 Subject: [PATCH] Add Cocoon Stack Gateway to the AI gateway benchmarks Adds `cocoonstack-gateway` (github.com/cocoonstack/gateway, served from gw-bench.simular.cloud in us-east4) to all four families: - Anthropic: native `/v1/messages`, claude-haiku-4-5-20251001 - OpenAI: `/v1/responses`, gpt-5.4-mini - Gemini: OpenAI-compatible `/v1/chat/completions`, gemini-3.6-flash (the gateway translates to Gemini's native API; there is no native Gemini inbound) - Kimi: `/v1/chat/completions`, kimi-k3 on Moonshot's international endpoint; `temperature` omitted (kimi-k3 accepts only 1) and `reasoningCountsAsFirstToken: true` (the stream carries `reasoning_content`) One credential, COCOONSTACK_GATEWAY_API_KEY, wired through .env.example, provider-vars.json and the workflow's vault list; a missing key reports SKIPPED as usual. Confirmed live with this repo's runner from us-east4: `--iterations 50` on all four families, 800/800 probes succeeded, and every wire ends with a usage frame so tokens/sec resolves. --- .github/workflows/ai-gateway-benchmarks.yml | 2 +- AI_GATEWAYS.md | 10 +++++----- benchmarks/.env.example | 1 + benchmarks/ai-gateway/providers-gemini.ts | 9 +++++++++ benchmarks/ai-gateway/providers-kimi.ts | 11 +++++++++++ benchmarks/ai-gateway/providers-openai.ts | 9 +++++++++ benchmarks/ai-gateway/providers.ts | 12 ++++++++++++ benchmarks/scripts/provider-vars.json | 12 ++++++++++++ package.json | 4 ++++ 9 files changed, 64 insertions(+), 6 deletions(-) diff --git a/.github/workflows/ai-gateway-benchmarks.yml b/.github/workflows/ai-gateway-benchmarks.yml index 9cbea5a1..e7b6322e 100644 --- a/.github/workflows/ai-gateway-benchmarks.yml +++ b/.github/workflows/ai-gateway-benchmarks.yml @@ -159,7 +159,7 @@ jobs: INPUT_ITERATIONS: ${{ github.event.inputs.iterations }} FAMILY_ITERATIONS: ${{ matrix.family.iterations }} run: | - . benchmarks/scripts/load-vault-secrets.sh '^(ANTHROPIC_API_KEY|OPENAI_API_KEY|GEMINI_API_KEY|KIMI_API_KEY|GITHUB_COPILOT_API_KEY|NOVITA_API_KEY|NEON_AI_GATEWAY_BASE_URL|NEON_AI_GATEWAY_TOKEN|NGROK_AI_GATEWAY_API_KEY|LLMAPI_API_KEY|CLOUDFLARE_AI_GATEWAY_ACCOUNT_ID|CLOUDFLARE_AI_GATEWAY_GATEWAY_ID|CLOUDFLARE_AI_GATEWAY_TOKEN|LLM_GATEWAY_API_KEY|OPENROUTER_API_KEY|PYDANTIC_AI_GATEWAY_API_KEY|VERCEL_AI_GATEWAY_API_KEY|CONCENTRATE_AI_GATEWAY_API_KEY|BLAZERAIL_API_KEY|BENCHMARKS_PLATFORM_API_KEY)' + . benchmarks/scripts/load-vault-secrets.sh '^(ANTHROPIC_API_KEY|OPENAI_API_KEY|GEMINI_API_KEY|KIMI_API_KEY|GITHUB_COPILOT_API_KEY|NOVITA_API_KEY|NEON_AI_GATEWAY_BASE_URL|NEON_AI_GATEWAY_TOKEN|NGROK_AI_GATEWAY_API_KEY|LLMAPI_API_KEY|CLOUDFLARE_AI_GATEWAY_ACCOUNT_ID|CLOUDFLARE_AI_GATEWAY_GATEWAY_ID|CLOUDFLARE_AI_GATEWAY_TOKEN|LLM_GATEWAY_API_KEY|OPENROUTER_API_KEY|PYDANTIC_AI_GATEWAY_API_KEY|VERCEL_AI_GATEWAY_API_KEY|CONCENTRATE_AI_GATEWAY_API_KEY|BLAZERAIL_API_KEY|COCOONSTACK_GATEWAY_API_KEY|BENCHMARKS_PLATFORM_API_KEY)' # GitHub Copilot is kept in the daily runs but excluded from the weekly run. if [ "${{ github.event_name }}" = "schedule" ]; then diff --git a/AI_GATEWAYS.md b/AI_GATEWAYS.md index 7fe51f43..abba5a6d 100644 --- a/AI_GATEWAYS.md +++ b/AI_GATEWAYS.md @@ -2,11 +2,11 @@ This document describes the **AI gateway benchmark** — a phase-by-phase latency, throughput, and reliability comparison of OpenRouter, Vercel AI Gateway, Cloudflare AI Gateway, LLM Gateway, Pydantic AI Gateway, Concentrate AI, Novita, Ramp Router, Neon AI Gateway, ngrok AI Gateway, and LLM API. It's organized as one **family benchmark per target provider**, all built on the same shared task plumbing (`shared-task.ts`, `phase-probe.ts`, scoring) so every family uses an identical prompt, phase methodology, and scoring formula — only the target provider (and therefore the model and each gateway's routing syntax) changes between them: -- **Anthropic family** (`ai-gateway.bench.ts` + `providers.ts`) — ten gateways (OpenRouter, Vercel AI Gateway, Cloudflare AI Gateway, LLM Gateway, Pydantic AI Gateway, Concentrate AI, Ramp Router, Neon AI Gateway, ngrok AI Gateway, and LLM API, plus anthropic-direct as a direct baseline; Novita excluded — see below) routed to Claude Haiku 4.5, measured against a direct-to-Anthropic baseline. This is the original benchmark and the one with the deepest "confirmed live" verification (see [Every gateway is hit directly](#every-gateway-is-hit-directly--no-gateway-is-proxied-through-another) below). +- **Anthropic family** (`ai-gateway.bench.ts` + `providers.ts`) — eleven gateways (Cocoon Stack Gateway, OpenRouter, Vercel AI Gateway, Cloudflare AI Gateway, LLM Gateway, Pydantic AI Gateway, Concentrate AI, Ramp Router, Neon AI Gateway, ngrok AI Gateway, and LLM API, plus anthropic-direct as a direct baseline; Novita excluded — see below) routed to Claude Haiku 4.5, measured against a direct-to-Anthropic baseline. This is the original benchmark and the one with the deepest "confirmed live" verification (see [Every gateway is hit directly](#every-gateway-is-hit-directly--no-gateway-is-proxied-through-another) below). -- **OpenAI family** (`ai-gateway-openai.bench.ts` + `providers-openai.ts`) — ten gateways (OpenRouter, Vercel AI Gateway, Cloudflare AI Gateway, LLM Gateway, Concentrate AI, Pydantic AI Gateway, Ramp Router, Neon AI Gateway, ngrok AI Gateway, and LLM API; Novita excluded — see below) routed to `gpt-5.4-mini` instead, measured against a direct-to-OpenAI baseline. See [OpenAI family benchmark](#openai-family-benchmark). -- **Gemini family** (`ai-gateway-gemini.bench.ts` + `providers-gemini.ts`) — eight gateways (OpenRouter, Vercel AI Gateway, Cloudflare AI Gateway, LLM Gateway, Concentrate AI, Neon AI Gateway, ngrok AI Gateway, and LLM API; Pydantic, Novita, and Ramp Router excluded — see below) routed to `gemini-3.6-flash`, measured against a direct-to-Gemini baseline. See [Gemini family benchmark](#gemini-family-benchmark). -- **Kimi family** (`ai-gateway-kimi.bench.ts` + `providers-kimi.ts`) — ten gateways (OpenRouter, Vercel AI Gateway, Cloudflare AI Gateway, LLM Gateway, Concentrate AI, Novita, Ramp Router, Neon AI Gateway, ngrok AI Gateway, and LLM API; Pydantic excluded — see below) routed to `kimi-k3`, measured against a direct-to-Moonshot baseline. See [Kimi family benchmark](#kimi-family-benchmark). +- **OpenAI family** (`ai-gateway-openai.bench.ts` + `providers-openai.ts`) — eleven gateways (Cocoon Stack Gateway, OpenRouter, Vercel AI Gateway, Cloudflare AI Gateway, LLM Gateway, Concentrate AI, Pydantic AI Gateway, Ramp Router, Neon AI Gateway, ngrok AI Gateway, and LLM API; Novita excluded — see below) routed to `gpt-5.4-mini` instead, measured against a direct-to-OpenAI baseline. See [OpenAI family benchmark](#openai-family-benchmark). +- **Gemini family** (`ai-gateway-gemini.bench.ts` + `providers-gemini.ts`) — nine gateways (Cocoon Stack Gateway, OpenRouter, Vercel AI Gateway, Cloudflare AI Gateway, LLM Gateway, Concentrate AI, Neon AI Gateway, ngrok AI Gateway, and LLM API; Pydantic, Novita, and Ramp Router excluded — see below) routed to `gemini-3.6-flash`, measured against a direct-to-Gemini baseline. See [Gemini family benchmark](#gemini-family-benchmark). +- **Kimi family** (`ai-gateway-kimi.bench.ts` + `providers-kimi.ts`) — eleven gateways (Cocoon Stack Gateway, OpenRouter, Vercel AI Gateway, Cloudflare AI Gateway, LLM Gateway, Concentrate AI, Novita, Ramp Router, Neon AI Gateway, ngrok AI Gateway, and LLM API; Pydantic excluded — see below) routed to `kimi-k3`, measured against a direct-to-Moonshot baseline. See [Kimi family benchmark](#kimi-family-benchmark). A result from one family is **not** directly comparable to the same gateway's result in another family — different target provider means a different underlying model and (for some gateways) a different routing path, so a difference in numbers can't be attributed to the gateway alone the way it can within a single family. @@ -277,7 +277,7 @@ npx tsx benchmarks/ai-gateway/ai-gateway.bench.ts --ai-gateway-iterations-cold 2 npx tsx benchmarks/ai-gateway/ai-gateway-openai.bench.ts --ai-gateway-iterations-cold 20 --ai-gateway-iterations-warm 0 ``` -Required environment variables (`benchmarks/.env.example`): `OPENROUTER_API_KEY`, `VERCEL_AI_GATEWAY_API_KEY`, `LLM_GATEWAY_API_KEY`, `CONCENTRATE_AI_GATEWAY_API_KEY` are shared across all four families (same gateway accounts, just routed to a different target model each time). `CLOUDFLARE_AI_GATEWAY_ACCOUNT_ID`, `CLOUDFLARE_AI_GATEWAY_GATEWAY_ID`, and `CLOUDFLARE_AI_GATEWAY_TOKEN` are all **required** by every family's Cloudflare entry (the Kimi family's entry uses a different Cloudflare endpoint than the other three, and is the only one where the token is the *sole* auth — no separate per-family key is forwarded there — but all four now require Authenticated Gateway mode, not just Kimi). `NEON_AI_GATEWAY_BASE_URL` and `NEON_AI_GATEWAY_TOKEN` are used across all four families. `NGROK_AI_GATEWAY_API_KEY` is used by all four families (Anthropic, OpenAI, Gemini, and Kimi). `LLMAPI_API_KEY` is used by all four families (Anthropic, OpenAI, Gemini, and Kimi). `PYDANTIC_AI_GATEWAY_API_KEY` is used by the Anthropic and OpenAI families — the Gemini and Kimi families exclude Pydantic. `NOVITA_API_KEY` is used by the Anthropic and Kimi families — OpenAI and Gemini exclude Novita. `RAMP_ROUTER_API_KEY` is used by the Anthropic, OpenAI, and Kimi families — Gemini excludes Ramp Router. `ANTHROPIC_API_KEY` is used by the Anthropic family (Cloudflare's passthrough + `anthropic-direct`); `OPENAI_API_KEY` by the OpenAI family (Cloudflare's passthrough + `openai-direct`); `GEMINI_API_KEY` by the Gemini family (Cloudflare's passthrough + `gemini-direct`); `KIMI_API_KEY` by the Kimi family (`kimi-direct` only — Cloudflare's Kimi entry doesn't need it). Missing credentials cause that participant to be reported as `SKIPPED` rather than failing the run. +Required environment variables (`benchmarks/.env.example`): `OPENROUTER_API_KEY`, `VERCEL_AI_GATEWAY_API_KEY`, `LLM_GATEWAY_API_KEY`, `CONCENTRATE_AI_GATEWAY_API_KEY` are shared across all four families (same gateway accounts, just routed to a different target model each time). `CLOUDFLARE_AI_GATEWAY_ACCOUNT_ID`, `CLOUDFLARE_AI_GATEWAY_GATEWAY_ID`, and `CLOUDFLARE_AI_GATEWAY_TOKEN` are all **required** by every family's Cloudflare entry (the Kimi family's entry uses a different Cloudflare endpoint than the other three, and is the only one where the token is the *sole* auth — no separate per-family key is forwarded there — but all four now require Authenticated Gateway mode, not just Kimi). `NEON_AI_GATEWAY_BASE_URL` and `NEON_AI_GATEWAY_TOKEN` are used across all four families. `NGROK_AI_GATEWAY_API_KEY` is used by all four families (Anthropic, OpenAI, Gemini, and Kimi). `COCOONSTACK_GATEWAY_API_KEY` is used by all four families (Cocoon Stack Gateway speaks the Anthropic Messages and OpenAI Responses wires natively and the OpenAI chat wire for the Gemini and Kimi families). `LLMAPI_API_KEY` is used by all four families (Anthropic, OpenAI, Gemini, and Kimi). `PYDANTIC_AI_GATEWAY_API_KEY` is used by the Anthropic and OpenAI families — the Gemini and Kimi families exclude Pydantic. `NOVITA_API_KEY` is used by the Anthropic and Kimi families — OpenAI and Gemini exclude Novita. `RAMP_ROUTER_API_KEY` is used by the Anthropic, OpenAI, and Kimi families — Gemini excludes Ramp Router. `ANTHROPIC_API_KEY` is used by the Anthropic family (Cloudflare's passthrough + `anthropic-direct`); `OPENAI_API_KEY` by the OpenAI family (Cloudflare's passthrough + `openai-direct`); `GEMINI_API_KEY` by the Gemini family (Cloudflare's passthrough + `gemini-direct`); `KIMI_API_KEY` by the Kimi family (`kimi-direct` only — Cloudflare's Kimi entry doesn't need it). Missing credentials cause that participant to be reported as `SKIPPED` rather than failing the run. ## Output diff --git a/benchmarks/.env.example b/benchmarks/.env.example index ee75fb65..28d75a66 100644 --- a/benchmarks/.env.example +++ b/benchmarks/.env.example @@ -177,3 +177,4 @@ NEON_AI_GATEWAY_TOKEN=your_neon_ai_gateway_token NGROK_AI_GATEWAY_API_KEY=your_ngrok_ai_gateway_api_key LLMAPI_API_KEY=your_llmapi_api_key BLAZERAIL_API_KEY=your_blazerail_api_key +COCOONSTACK_GATEWAY_API_KEY=your_cocoonstack_gateway_api_key diff --git a/benchmarks/ai-gateway/providers-gemini.ts b/benchmarks/ai-gateway/providers-gemini.ts index 9b114005..b9aa9624 100644 --- a/benchmarks/ai-gateway/providers-gemini.ts +++ b/benchmarks/ai-gateway/providers-gemini.ts @@ -36,6 +36,15 @@ import { resolveNeonHost } from './neon-host.js'; * family's entries in `providers.ts` were held to. */ export const providers: AIGatewayProviderConfig[] = [ + { + name: 'cocoonstack-gateway', + requiredEnvVars: ['COCOONSTACK_GATEWAY_API_KEY'], + wireFormat: 'openai', + model: 'gemini-3.6-flash', + host: 'gw-bench.simular.cloud', + path: '/v1/chat/completions', + buildHeaders: () => ({ Authorization: `Bearer ${process.env.COCOONSTACK_GATEWAY_API_KEY}` }), + }, { // No native Gemini passthrough documented for OpenRouter — its whole // platform is built around one normalized, OpenAI-Chat-Completions- diff --git a/benchmarks/ai-gateway/providers-kimi.ts b/benchmarks/ai-gateway/providers-kimi.ts index 4021f6df..b7a47d33 100644 --- a/benchmarks/ai-gateway/providers-kimi.ts +++ b/benchmarks/ai-gateway/providers-kimi.ts @@ -84,6 +84,17 @@ import { resolveNeonHost } from './neon-host.js'; * OpenRouter, and Vercel here are the exceptions — all confirmed live). */ export const providers: AIGatewayProviderConfig[] = [ + { + name: 'cocoonstack-gateway', + requiredEnvVars: ['COCOONSTACK_GATEWAY_API_KEY'], + wireFormat: 'openai', + model: 'kimi-k3', + host: 'gw-bench.simular.cloud', + path: '/v1/chat/completions', + buildHeaders: () => ({ Authorization: `Bearer ${process.env.COCOONSTACK_GATEWAY_API_KEY}` }), + extraBody: { temperature: undefined }, + reasoningCountsAsFirstToken: true, + }, { // `moonshotai/kimi-k3` is OpenRouter's confirmed catalog convention // (openrouter.ai/moonshotai). Provider-order pinning to `moonshotai` diff --git a/benchmarks/ai-gateway/providers-openai.ts b/benchmarks/ai-gateway/providers-openai.ts index 33bda460..10a78ea3 100644 --- a/benchmarks/ai-gateway/providers-openai.ts +++ b/benchmarks/ai-gateway/providers-openai.ts @@ -30,6 +30,15 @@ import { resolveNeonHost } from './neon-host.js'; * family's entries in `providers.ts` were held to. */ export const providers: AIGatewayProviderConfig[] = [ + { + name: 'cocoonstack-gateway', + requiredEnvVars: ['COCOONSTACK_GATEWAY_API_KEY'], + wireFormat: 'responses', + model: 'gpt-5.4-mini', + host: 'gw-bench.simular.cloud', + path: '/v1/responses', + buildHeaders: () => ({ Authorization: `Bearer ${process.env.COCOONSTACK_GATEWAY_API_KEY}` }), + }, { // Confirmed: OpenRouter's Responses API passthrough exists at // `/api/v1/responses` (openrouter.ai/docs/api_reference/responses/overview), diff --git a/benchmarks/ai-gateway/providers.ts b/benchmarks/ai-gateway/providers.ts index 156a89d5..0af54f44 100644 --- a/benchmarks/ai-gateway/providers.ts +++ b/benchmarks/ai-gateway/providers.ts @@ -11,6 +11,18 @@ import { resolveNeonHost } from './neon-host.js'; * latency each gateway adds on top of the underlying provider. */ export const providers: AIGatewayProviderConfig[] = [ + { + name: 'cocoonstack-gateway', + requiredEnvVars: ['COCOONSTACK_GATEWAY_API_KEY'], + wireFormat: 'anthropic', + model: 'claude-haiku-4-5-20251001', + host: 'gw-bench.simular.cloud', + path: '/v1/messages', + buildHeaders: () => ({ + 'x-api-key': process.env.COCOONSTACK_GATEWAY_API_KEY || '', + 'anthropic-version': '2023-06-01', + }), + }, { // `anthropic/claude-haiku-4.5` is a catalog alias that OpenRouter can // serve from more than one upstream (Anthropic direct, Bedrock, Vertex, diff --git a/benchmarks/scripts/provider-vars.json b/benchmarks/scripts/provider-vars.json index ce64ab1b..40ffaf1d 100644 --- a/benchmarks/scripts/provider-vars.json +++ b/benchmarks/scripts/provider-vars.json @@ -287,6 +287,9 @@ ], "anthropic-direct": [ "ANTHROPIC_API_KEY" + ], + "cocoonstack-gateway": [ + "COCOONSTACK_GATEWAY_API_KEY" ] }, "ai-gateway-openai": { @@ -329,6 +332,9 @@ ], "openai-direct": [ "OPENAI_API_KEY" + ], + "cocoonstack-gateway": [ + "COCOONSTACK_GATEWAY_API_KEY" ] }, "ai-gateway-gemini": { @@ -365,6 +371,9 @@ ], "gemini-direct": [ "GEMINI_API_KEY" + ], + "cocoonstack-gateway": [ + "COCOONSTACK_GATEWAY_API_KEY" ] }, "ai-gateway-kimi": { @@ -409,6 +418,9 @@ ], "kimi-direct": [ "KIMI_API_KEY" + ], + "cocoonstack-gateway": [ + "COCOONSTACK_GATEWAY_API_KEY" ] }, "snapshot-fork": { diff --git a/package.json b/package.json index 2e3edf92..6c526b97 100644 --- a/package.json +++ b/package.json @@ -83,6 +83,7 @@ "generate-browser-throughput-svg": "tsx benchmarks/browser/generate-throughput-svg.ts", "generate-pricing-svg": "tsx benchmarks/sandbox/generate-pricing-svg.ts", "bench:ai-gateway": "tsx packages/benchsdk-runner/dist/bin.js run benchmarks/ai-gateway/ai-gateway.bench.ts", + "bench:ai-gateway:cocoonstack-gateway": "tsx packages/benchsdk-runner/dist/bin.js run benchmarks/ai-gateway/ai-gateway.bench.ts --provider cocoonstack-gateway", "bench:ai-gateway:github-copilot": "tsx packages/benchsdk-runner/dist/bin.js run benchmarks/ai-gateway/ai-gateway.bench.ts --provider github-copilot", "bench:ai-gateway:openrouter": "tsx packages/benchsdk-runner/dist/bin.js run benchmarks/ai-gateway/ai-gateway.bench.ts --provider openrouter", "bench:ai-gateway:vercel": "tsx packages/benchsdk-runner/dist/bin.js run benchmarks/ai-gateway/ai-gateway.bench.ts --provider vercel-ai-gateway", @@ -97,6 +98,7 @@ "bench:ai-gateway:ramp": "tsx packages/benchsdk-runner/dist/bin.js run benchmarks/ai-gateway/ai-gateway.bench.ts --provider ramp", "generate-ai-gateway-svg": "tsx benchmarks/ai-gateway/generate-svg.ts", "bench:ai-gateway-openai": "tsx packages/benchsdk-runner/dist/bin.js run benchmarks/ai-gateway/ai-gateway-openai.bench.ts", + "bench:ai-gateway-openai:cocoonstack-gateway": "tsx packages/benchsdk-runner/dist/bin.js run benchmarks/ai-gateway/ai-gateway-openai.bench.ts --provider cocoonstack-gateway", "bench:ai-gateway-openai:github-copilot": "tsx packages/benchsdk-runner/dist/bin.js run benchmarks/ai-gateway/ai-gateway-openai.bench.ts --provider github-copilot", "bench:ai-gateway-openai:openrouter": "tsx packages/benchsdk-runner/dist/bin.js run benchmarks/ai-gateway/ai-gateway-openai.bench.ts --provider openrouter", "bench:ai-gateway-openai:vercel": "tsx packages/benchsdk-runner/dist/bin.js run benchmarks/ai-gateway/ai-gateway-openai.bench.ts --provider vercel-ai-gateway", @@ -111,6 +113,7 @@ "bench:ai-gateway-openai:llmapi": "tsx packages/benchsdk-runner/dist/bin.js run benchmarks/ai-gateway/ai-gateway-openai.bench.ts --provider llmapi", "generate-ai-gateway-openai-svg": "tsx benchmarks/ai-gateway/generate-svg.ts --dir ai-gateway-latency/openai --title \"AI Gateway Benchmarks — OpenAI\" --out ai-gateway-openai.svg", "bench:ai-gateway-gemini": "tsx packages/benchsdk-runner/dist/bin.js run benchmarks/ai-gateway/ai-gateway-gemini.bench.ts", + "bench:ai-gateway-gemini:cocoonstack-gateway": "tsx packages/benchsdk-runner/dist/bin.js run benchmarks/ai-gateway/ai-gateway-gemini.bench.ts --provider cocoonstack-gateway", "bench:ai-gateway-gemini:openrouter": "tsx packages/benchsdk-runner/dist/bin.js run benchmarks/ai-gateway/ai-gateway-gemini.bench.ts --provider openrouter", "bench:ai-gateway-gemini:vercel": "tsx packages/benchsdk-runner/dist/bin.js run benchmarks/ai-gateway/ai-gateway-gemini.bench.ts --provider vercel-ai-gateway", "bench:ai-gateway-gemini:cloudflare": "tsx packages/benchsdk-runner/dist/bin.js run benchmarks/ai-gateway/ai-gateway-gemini.bench.ts --provider cloudflare-ai-gateway", @@ -122,6 +125,7 @@ "bench:ai-gateway-gemini:llmapi": "tsx packages/benchsdk-runner/dist/bin.js run benchmarks/ai-gateway/ai-gateway-gemini.bench.ts --provider llmapi", "generate-ai-gateway-gemini-svg": "tsx benchmarks/ai-gateway/generate-svg.ts --dir ai-gateway-latency/gemini --title \"AI Gateway Benchmarks — Gemini\" --out ai-gateway-gemini.svg", "bench:ai-gateway-kimi": "tsx packages/benchsdk-runner/dist/bin.js run benchmarks/ai-gateway/ai-gateway-kimi.bench.ts", + "bench:ai-gateway-kimi:cocoonstack-gateway": "tsx packages/benchsdk-runner/dist/bin.js run benchmarks/ai-gateway/ai-gateway-kimi.bench.ts --provider cocoonstack-gateway", "bench:ai-gateway-kimi:github-copilot": "tsx packages/benchsdk-runner/dist/bin.js run benchmarks/ai-gateway/ai-gateway-kimi.bench.ts --provider github-copilot", "bench:ai-gateway-kimi:openrouter": "tsx packages/benchsdk-runner/dist/bin.js run benchmarks/ai-gateway/ai-gateway-kimi.bench.ts --provider openrouter", "bench:ai-gateway-kimi:vercel": "tsx packages/benchsdk-runner/dist/bin.js run benchmarks/ai-gateway/ai-gateway-kimi.bench.ts --provider vercel-ai-gateway",