Skip to content

[RESILIENCE]: Add timeout, retry, and circuit-breaker policies at external boundaries #220

Description

@barry01-hash

Context

GrantFox depends on Supabase, PostgreSQL, Resend, PostHog, wallet providers, and Trustless Work/Stellar. Unbounded calls or indiscriminate retries can exhaust serverless execution time, duplicate side effects, and produce confusing UI states.

Scope

  • Inventory external calls and classify them as read, idempotent write, or non-idempotent write.
  • Add explicit deadlines/abort signals and typed error translation at adapter boundaries.
  • Implement bounded exponential backoff with jitter only where safe.
  • Add circuit breaking/bulkheads for degraded dependencies and define fallback behavior.
  • Propagate correlation IDs through API handlers, jobs, and dependency calls.
  • Surface recoverable versus terminal failures to clients.

Acceptance criteria

  • Every outbound request has a documented timeout and retry policy.
  • Non-idempotent writes are never blindly retried.
  • Dependency degradation does not consume unbounded connections or function duration.
  • Fault-injection tests cover latency, timeout, connection reset, 429, and 5xx.
  • Metrics identify the failing dependency and circuit state.
  • User-facing errors give a safe retry/recovery action.

Relevant areas

Escrow hooks/adapters, email service, Supabase/Prisma clients, PostHog, API route error handling.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions