Skip to content

Add API benchmark suite#375

Open
maikeru199 wants to merge 2 commits into
SecureBananaLabs:mainfrom
maikeru199:issue-30-api-benchmark-suite
Open

Add API benchmark suite#375
maikeru199 wants to merge 2 commits into
SecureBananaLabs:mainfrom
maikeru199:issue-30-api-benchmark-suite

Conversation

@maikeru199
Copy link
Copy Markdown

Summary

  • Adds a reproducible API benchmark suite under benchmarks/ covering every /api/ route.
  • Captures p50/p95/p99 latency, sustained and peak RPS, error rate, and TTFB per endpoint.
  • Adds a reviewable threshold file plus a pull-request smoke workflow that runs the low-concurrency benchmark gate.
  • Documents .env.benchmark setup and writes JSON/Markdown reports to benchmarks/results/.

Fixes #30

Validation

  • node --check benchmarks/run-api-benchmark.mjs
  • node .tools/npm/bin/npm-cli.js run test -w apps/api
  • BENCHMARK_BASE_URL=http://127.0.0.1:4000 BENCHMARK_AUTH_SECRET=development-secret node benchmarks/run-api-benchmark.mjs --smoke --gate
  • BENCHMARK_BASE_URL=http://127.0.0.1:4000 BENCHMARK_AUTH_SECRET=development-secret node .tools/npm/bin/npm-cli.js run benchmark:smoke
  • BENCHMARK_BASE_URL=http://127.0.0.1:4000 BENCHMARK_AUTH_SECRET=development-secret node .tools/npm/bin/npm-cli.js run benchmark

Benchmark Environment

Hardware

  • CPU model & core count: Apple M4, 10 cores
  • RAM (total & available during benchmark): 24GB total; generated report recorded 227MB free at run start
  • Storage type (SSD / NVMe / HDD): internal Apple SSD/NVMe-class storage
  • Network interface (Ethernet / WiFi / loopback): loopback to 127.0.0.1:4000
  • Machine type (local workstation / cloud VM / CI runner - include instance type if cloud): local workstation
  • OS & version: macOS 26.5 (Darwin 25.5.0, arm64)

Runtime

  • Node.js version (or relevant runtime): Node.js v24.14.0 locally; CI workflow pins Node 20
  • Any resource limits applied (Docker memory cap, cgroup limits, etc.): none
  • Other significant processes running during benchmark (yes / no - if yes, describe): yes, normal local desktop/Codex workload

If submitted by or with an AI agent

  • Agent or tool name (e.g. Claude Code, Devin, Copilot Workspace, AutoGPT): Codex
  • Underlying model and version (e.g. claude-sonnet-4-5, gpt-4o - if known): OpenAI GPT-5-class Codex model
  • Inference provider (e.g. Anthropic, OpenAI, Azure, self-hosted): OpenAI
  • Orchestration framework if any (e.g. LangChain, AutoGen, custom): Codex desktop tool orchestration
  • Execution mode (fully autonomous / human-supervised / human-initiated per step): human-initiated, agent-executed with progress updates
  • Did the agent have shell/tool access during execution (yes / no): yes
  • Did the agent have internet access during execution (yes / no): yes
  • Were benchmark commands run by the agent directly or handed off to the human to run: run directly by the agent
  • Any known agent constraints or sandboxing that may have affected execution: local npm was not on PATH, so npm CLI was temporarily downloaded under the workspace for validation; generated benchmark results are intentionally gitignored

API Benchmark Report

  • Run ID: 2026-05-20T01-56-24-090Z
  • Target: http://127.0.0.1:4000
  • Started: 2026-05-20T01:56:24.095Z
  • Duration: 430ms
  • Concurrency: 2
  • Requests per endpoint: 5
  • Runtime: Node.js v24.14.0
  • OS: darwin 25.5.0 arm64
  • CPU: Apple M4 (10 cores)
  • Memory: 24576MB total, 227MB free at start
Endpoint Method Path Requests p50 ms p95 ms p99 ms TTFB p95 ms Sustained RPS Peak RPS Error %
auth.register POST /api/auth/register 5 5.07 17.44 17.44 17.37 152.79 5 0
auth.login POST /api/auth/login 5 7.31 17.05 17.05 10.59 157.67 5 0
auth.oauthCallback GET /api/auth/oauth/github/callback 5 3.8 6.5 6.5 6.45 396.19 5 0
auth.refresh POST /api/auth/refresh 5 4.02 8.99 8.99 8.92 359.51 5 0
users.list GET /api/users 5 2.3 3.82 3.82 3.76 574.99 5 0
users.create POST /api/users 5 2.76 4.65 4.65 4.62 524.7 5 0
jobs.list GET /api/jobs 5 37.4 93.16 93.16 93.09 41.71 5 0
jobs.create POST /api/jobs 5 2.24 5 5 4.92 542.01 5 0
proposals.list GET /api/proposals 5 2.97 5.58 5.58 5.55 490.23 5 0
proposals.create POST /api/proposals 5 3.15 4.44 4.44 4.42 499.3 5 0
payments.create POST /api/payments 5 18.24 62.28 62.28 62.24 70.39 5 0
reviews.list GET /api/reviews 5 10.65 27.33 27.33 27.3 128.72 5 0
reviews.create POST /api/reviews 5 1.98 6 6 5.96 489.32 5 0
messages.list GET /api/messages 5 1.48 2.03 2.03 1.88 1055.04 5 0
messages.create POST /api/messages 5 1.26 1.7 1.7 1.68 1156.57 5 0
notifications.list GET /api/notifications 5 0.94 1.75 1.75 1.72 1476.25 5 0
notifications.create POST /api/notifications 5 1.93 2.77 2.77 2.73 799.9 5 0
uploads.create POST /api/uploads 5 4.94 9.47 9.47 9.44 307.21 5 0
search.global GET /api/search?q=react%20marketplace%20developer 5 3.45 3.68 3.68 3.63 520.55 5 0
admin.metrics GET /api/admin/metrics 5 1.28 2.41 2.41 2.39 968.7 5 0

Threshold Gate

No threshold failures.

github-actions Bot added a commit that referenced this pull request May 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Benchmark APIs with p50, p95, p99 latency, RPS, error rate and TTFB

1 participant