Observability, Alerting, and Circuit Breaking [LOW]

[HayimShaul]

Description
Instrument all wallet service entry points with metrics (request rate, latency percentiles, error rate, cache level, registry size) and expose them through the existing metrics.Provider infrastructure already present in the codebase. Define alert thresholds that trigger when request rates or resource utilisation approach saturation. Pair this with a circuit-breaker pattern on the wallet service: when error rates or latency exceed configured thresholds, the service temporarily rejects new requests and returns a back-pressure signal to callers. This limits blast radius during an active DoS attempt and gives operators visibility to detect and respond before full exhaustion occurs.

[SurbhiAgarwal1]

I’d like to propose a structured implementation for observability, alerting, and circuit breaking around wallet service entry points.

---

1. Metrics Instrumentation (via metrics.Provider)

Instrument all wallet service entry points (e.g., Select, Transfer, Issue, Redeem) using the existing metrics.Provider.

Metrics to expose

• Request rate (counter)
• Latency (histogram with percentiles)
• Error rate (counter)
• In-flight requests (gauge)
• Cache stats (hit/miss if applicable)
• Registry / store size (gauge where available)

Example (Go)

type WalletMetrics struct {
    RequestCount   metrics.Counter
    ErrorCount     metrics.Counter
    Latency        metrics.Histogram
    InFlight       metrics.Gauge
}

func NewWalletMetrics(p metrics.Provider) *WalletMetrics {
    return &WalletMetrics{
        RequestCount: p.NewCounter("wallet_requests_total"),
        ErrorCount:   p.NewCounter("wallet_errors_total"),
        Latency:      p.NewHistogram("wallet_request_latency_ms"),
        InFlight:     p.NewGauge("wallet_inflight_requests"),
    }
}

[adecaro]

Hi @SurbhiAgarwal1 , thanks for offering 🙏

I think we target here the identity service token/services/identity.

Please, go ahead 😄