Skip to content

feat(matches): backoff and circuit breaker for the external result feed - #1790

Open
bilhokista wants to merge 4 commits into
Arena1X:mainfrom
bilhokista:feat/1620-feed-breaker
Open

feat(matches): backoff and circuit breaker for the external result feed#1790
bilhokista wants to merge 4 commits into
Arena1X:mainfrom
bilhokista:feat/1620-feed-breaker

Conversation

@bilhokista

Copy link
Copy Markdown

Closes #1620.

What this changes

HttpExternalResultFeedClient.fetchResults() was a bare firstValueFrom(httpService.get(...)) with no error handling: a single upstream blip failed a poll, and a sustained outage meant every scheduled poll kept hammering a service that was already down.

Backoff reuses the existing withRetry / computeBackoffDelay in backend/src/common/retry.util.ts rather than introducing a second backoff implementation, so the feed behaves like soroban.service.ts and notifications/email.service.ts already do. The issue describes the client as having "no backoff", which is accurate — but the helper it needs was already in the codebase and simply not wired in.

Circuit breaker is new: backend/src/common/circuit-breaker.ts, placed alongside retry.util.ts. Retry handles one bad call; the breaker handles a bad upstream. It is deliberately small — one upstream, one instance, constructed by the client that owns the call — rather than a decorator or a module.

  • closed → calls pass through, consecutive failures counted
  • open → calls rejected locally with CircuitOpenError for openMs
  • half-openone probe admitted; success closes, failure re-opens for a full window

Retry sits inside the breaker, so one poll gets one retry budget and the breaker counts polls rather than individual attempts.

Error classification (isTransientFeedError): 408/425/429/5xx and requests that never got a response are retried; other 4xx are not, because a wrong URL or a rejected credential will be rejected identically on every attempt.

Health signal is exposed as client.getHealth(): BreakerHealth{ state, consecutiveFailures, retryAfterMs, lastError }.

One scope decision worth flagging

The issue lists backend/src/health in its file list. I did not wire the feed into HealthService: that service composes injected Terminus indicators, and adding the feed changes which dependencies /health reports on — an architecture call for a maintainer, not a side effect of this PR. The signal is exposed and ready to consume; I'm happy to add the indicator in this PR if you'd like it here, and there is a comment in the client saying so.

Tests

backend/src/common/circuit-breaker.spec.ts (new, 6 cases) drives the breaker through an injected clock, so no test sleeps through a real openMs:

  • passes results through while closed
  • opens after the failure threshold
  • rejects without calling upstream while open
  • half-open probe recovers and resets the failure count
  • failed probe re-opens for a full window
  • only one probe is admitted while half-open

external-result-feed.client.spec.ts adds status-classification cases, "retries a transient failure", "does not retry a rejected credential", "opens after repeated failures and stops calling the feed", and "closes again when the half-open probe succeeds". The existing URL/credential test is unchanged and still passes.

Honest note on verification: I could not run the backend Jest suite locally (no full workspace install). I did two things instead: every changed file was parsed with the TypeScript compiler API, and the breaker's state machine was transpiled and executed standalone against the same 14 assertions the spec makes — all passed. CI is the real check on the Nest-level tests.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CrfEY1tvXrbeMDAUzxfuk7

@vercel

vercel Bot commented Sep 10, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
insight-arena-4rll Ready Ready Preview Sep 10, 2026 1:26pm UTC

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Backend] — External Result Feed Client Backoff and Circuit Breaker

1 participant