Observability, Alerting, and Automated Throttle Escalation [LOW]

[HayimShaul]

Description
Instrument every Signer and Verifier service operation with metrics (call rate, latency, error rate, cache hit/miss ratio) and expose them via the existing token/core/common/metrics infrastructure. Define alert thresholds that trigger automated responses -- such as temporarily blocking a caller or reducing its quota -- when anomalous call volumes are detected. Comprehensive audit logging of caller identity, operation type, and outcome provides the forensic trail needed to detect abuse early, attribute it to a specific principal, and tune throttling policies over time.

[SurbhiAgarwal1]

Hi @HayimShaul, I would like to work on this issue.

After reviewing the current metrics infrastructure and the Security By Design epic, my understanding is that the goal here is to introduce lightweight but extensible observability and automated defensive throttling around Signer and Verifier operations.

Proposed approach:

• Instrument Signer and Verifier service operations using the existing token/core/common/metrics infrastructure.

• Capture metrics such as:

  • request/call rate
  • operation latency
  • success/error counts
  • cache hit/miss ratio
  • throttle escalation events

• Introduce middleware/interceptor-style wrappers so instrumentation remains decoupled from business logic.

• Add structured audit logging for:

  • caller identity/principal
  • operation type
  • timestamp
  • outcome/error
  • escalation/throttle action applied

• Define configurable anomaly thresholds (burst rate, sustained error rate, excessive retries, etc.) through policy/configuration instead of hardcoded values.

• Implement an initial automated escalation mechanism that can:

  • temporarily reduce quotas
  • apply cooldown windows
  • temporarily block abusive callers
  • emit alerting metrics/events for operators

• Keep the escalation logic extensible so future integrations with external monitoring/alerting systems can be added without changing core service logic.

I’ll first review the existing metrics abstractions and identify the best integration points before proposing concrete implementation changes.