Implement circuit breaker pattern for downstream API protection

[mheadd]

Problem

While the project has good timeout and retry mechanisms, there's no protection against repeatedly calling downstream APIs that are experiencing prolonged outages. This can lead to:

• Wasted resources retrying dead APIs
• Cascading failures
• Poor user experience with repeated timeouts
• Unnecessary load on failing services

Proposed Solution

Implement a circuit breaker pattern for each downstream API (NPS, Recreation.gov, OpenWeather) with three states:

Circuit States

• Closed: Normal operation, requests pass through
• Open: API is failing, immediately return error without calling API
• Half-Open: Testing if API has recovered, allow limited requests

Configuration

type CircuitBreaker struct {
    maxFailures   int           // Failures before opening (e.g., 5)
    resetTimeout  time.Duration // Time before trying half-open (e.g., 30s)
    halfOpenLimit int           // Test requests in half-open state (e.g., 3)
}

Implementation Approach

1. Create internal/api/circuit_breaker.go
2. Wrap each API client's HTTP calls with circuit breaker
3. Add metrics/logging for circuit state changes
4. Make thresholds configurable via config.yaml

Benefits

• ✅ Fail fast when APIs are down (no wasted retries)
• ✅ Automatic recovery detection
• ✅ Better resource utilization
• ✅ Improved user feedback (immediate error vs timeout)
• ✅ Reduced load on failing downstream services

Current State

The project already has:

• Context-aware requests with timeouts (30s NPS/RecGov, 15s Weather)
• Retry logic with exponential backoff
• Rate limit handling (429 responses)

Circuit breaker complements these by preventing repeated attempts during outages.

Implementation Checklist

• Create circuit breaker implementation
• Add configuration options
• Integrate with existing API clients
• Add circuit state logging
• Update documentation
• Add tests for state transitions
• Add metrics/monitoring hooks

References

• Current timeout config: internal/config/config.go
• HTTP client with retry: pkg/util/http_client.go
• API clients: internal/api/nps.go, recreation.go, weather.go