feat(orchestrator): per-client token-bucket rate limiter for LLM routes

[hoangsonww]

Summary

Adds a token-bucket rate limiter to the orchestrator — a new reliability primitive that sits alongside the existing circuit breaker, cost tracker, and dead-letter queue. It guards the expensive LLM-backed routes from runaway usage / abuse (which directly maps to API spend), per client.

What's included

core/rate-limiter.js

• RateLimiter — per-key token buckets with configurable capacity and refillPerSec. The clock is injectable (now) so refill behavior is fully deterministic in tests (no real timers/sleeps). API: tryConsume(key, cost), getState(key), reset(key), resetAll(), getStats(); validates its config.
• rateLimitMiddleware(limiter, { keyFn, cost }) — Express factory that sets X-RateLimit-Limit / X-RateLimit-Remaining on every response and, when a bucket is empty, returns 429 Too Many Requests with a Retry-After header and { error, retryAfterMs }.

index.js wiring

• Guards the three expensive routes: /api/supervisor/process, /api/agent/run, /api/batch/process (keyed by client IP).
• New GET /api/rate-limit stats route; rate-limit stats also surfaced in /health.
• Tunable via RATE_LIMIT_CAPACITY (default 60) and RATE_LIMIT_REFILL_PER_SEC (default 1).

Docs: README.md (new endpoint section + env) and .env.example.

Testing (TDD)

Built test-first (red → green). 14 new unit tests cover capacity limits, time-based refill + cap, per-key isolation, cost > 1, reset/resetAll, stats, config validation, and the middleware (headers, 429 + Retry-After, custom keyFn).

Test Suites: 2 passed, 2 total
Tests:       116 passed, 116 total   (102 existing + 14 new)

Notes

• In-memory by design (single-instance orchestrator); a Redis-backed store could be swapped in later behind the same interface if the orchestrator is horizontally scaled.
• Scope limited to orchestrator/ per the project's module conventions.

🤖 Generated with Claude Code

[netlify]

✅ Deploy Preview for docuthinker-ai-app ready!

Name	Link
🔨 Latest commit	6fe4bdc
🔍 Latest deploy log	https://app.netlify.com/projects/docuthinker-ai-app/deploys/6a2b6206237121000855849e
😎 Deploy Preview	https://deploy-preview-65--docuthinker-ai-app.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.
Lighthouse	1 paths audited Performance: 18 Accessibility: 93 Best Practices: 92 SEO: 99 PWA: 80 View the detailed breakdown and full score reports

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
docuthinker-fullstack-app	Ignored		Jun 12, 2026 1:34am

[gemini-code-assist]

Code Review

This pull request introduces an in-memory token-bucket rate limiter and middleware to protect expensive LLM-backed routes from runaway usage, along with corresponding configuration, endpoints, and unit tests. While the implementation is clean, several critical issues were identified: an unbounded Map for client buckets poses a memory leak and DoS risk; deploying behind a reverse proxy without configuring Express's trust proxy will cause all clients to share a single bucket; static costs do not reflect the varying loads of different routes; and requests with a cost exceeding the bucket's capacity will be permanently blocked.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

Using an unbounded Map to store client buckets in-memory introduces a potential memory leak and Denial of Service (DoS) vulnerability. Every unique client IP/key will create a new entry in this.buckets that is never cleaned up. Over time, or under a distributed traffic/DoS attack with rotating IPs, this map will grow indefinitely and eventually crash the process due to an Out-Of-Memory (OOM) error.

To mitigate this, you can implement a simple cleanup mechanism. Since any bucket that hasn't been active for capacity / refillPerSec seconds is guaranteed to be fully refilled, it can be safely deleted from the map. Alternatively, you can use a limited-size LRU cache or run a periodic sweep to delete idle buckets.

[gemini-code-assist]

When using req.ip as the rate-limiting key, Express requires app.set('trust proxy', true) (or a specific proxy configuration) to be enabled if the application is deployed behind a reverse proxy (e.g., Nginx, Cloudflare, AWS ALB, Kubernetes Ingress). Without this setting, req.ip will resolve to the upstream proxy's IP address for all requests, causing all clients to share a single rate-limiting bucket and leading to accidental global rate limiting.

Consider adding app.set('trust proxy', true) (or your specific proxy trust configuration) right after initializing app in index.js.

[gemini-code-assist]

The current middleware implementation only supports a static cost value. However, the guarded routes have vastly different resource consumption profiles: /api/supervisor/process runs a single pipeline, /api/agent/run can run up to 10 iterations, and /api/batch/process processes an array of documents. Applying a static cost of 1 to all of them does not accurately reflect their actual load.

Allowing cost to be a function (req) => number enables dynamic cost calculation (e.g., scaling with the number of documents in a batch request).

Suggested change

	function rateLimitMiddleware(limiter, { keyFn, cost = 1 } = {}) {
	return (req, res, next) => {
	const key = (keyFn ? keyFn(req) : req.ip) || "global";
	const result = limiter.tryConsume(key, cost);
	function rateLimitMiddleware(limiter, { keyFn, cost = 1 } = {}) {
	return (req, res, next) => {
	const key = (keyFn ? keyFn(req) : req.ip) || "global";
	const computedCost = typeof cost === "function" ? cost(req) : cost;
	const result = limiter.tryConsume(key, computedCost);

[gemini-code-assist]

If a request is made with a cost greater than the rate limiter's capacity, it will be permanently blocked. This is because the bucket's token count is capped at capacity, so bucket.tokens >= cost will always be false. Furthermore, the returned retryAfterMs will be calculated based on a deficit that can never be resolved, returning a misleading retry time to the client.

Consider validating that cost <= this.capacity at the beginning of tryConsume and throwing an error to fail fast.

  tryConsume(key, cost = 1) {
    if (cost > this.capacity) {
      throw new Error("RateLimiter: cost " + cost + " exceeds capacity " + this.capacity);
    }
    const bucket = this._refill(key);