Skip to content

HEALTH_CHECK_IMPLEMENTATION.md

Latest commit

 

History

History
609 lines (471 loc) · 19.7 KB

HEALTH_CHECK_IMPLEMENTATION.md

File metadata and controls

609 lines (471 loc) · 19.7 KB
Raw

Health Check Implementation Summary

This document summarizes the comprehensive health check system implemented for the oracle service.

Date Completed: June 2024
Status: Complete ✓

Overview

Replaced a hardcoded health endpoint with a real-time monitoring system that:

  1. Performs actual connectivity checks to all dependencies (Stellar RPC, contracts, chess APIs)
  2. Distinguishes failure modes (transient vs. permanent, rate limits vs. unreachable)
  3. Provides actionable per-dependency health status instead of a single boolean
  4. Integrates with Kubernetes probes for automated remediation
  5. Enables comprehensive alerting with SLA-based thresholds
  6. Supports synthetic canary testing for end-to-end validation
  7. Includes extensive test coverage including chaos/fault-injection scenarios

Files Changed

Core Implementation

oracle-service/src/health.rs (NEW)

The main health check module with:

  • HealthChecker — orchestrates all dependency probes
  • CheckResult — per-dependency check result (status, latency, failures)
  • HealthCheckResponse — full health status response schema
  • Real probes for RPC, escrow contract, oracle contract, and chess APIs
  • Status aggregation logic (healthy → degraded → unhealthy)

oracle-service/src/main.rs (MODIFIED)

  • Removed hardcoded health status
  • Integrated HealthChecker initialization
  • Added background health check poller (runs every 30 seconds)
  • Wired HTTP handler to return live health status
  • Three concurrent subsystems: HTTP server, health poller, pipeline poller

oracle-service/src/soroban_client.rs (MODIFIED)

  • Added health_check() — probes Stellar RPC with getNetwork call
  • Added contract_health_check(contract_id) — verifies contract reachability

oracle-service/src/oracle/lichess_client.rs (MODIFIED)

  • Added health_check() — probes Lichess API public endpoint

oracle-service/src/oracle/chess_com_client.rs (MODIFIED)

  • Added health_check() — probes Chess.com API public endpoint

oracle-service/src/lib.rs (MODIFIED)

  • Exported pub mod health

Documentation

docs/monitoring-health-checks.md (NEW)

Comprehensive monitoring architecture guide:

  • Health check architecture (3 subsystems)
  • Response schema with all fields documented
  • Dependency-specific probe details and failure modes
  • Kubernetes integration
  • Prometheus/AlertManager configuration examples
  • Alert severity levels and escalation policy
  • Runbooks for responding to critical and degraded alerts
  • Synthetic canary check design (planned)
  • Metrics and observability recommendations
  • Testing procedures and fault injection examples

docs/health-check-integration.md (NEW)

Quick-start integration guide:

  • Setup checklist
  • Kubernetes liveness/readiness probe configuration
  • Alert configuration examples (Prometheus, Nagios, bash)
  • Response time SLAs and thresholds
  • Troubleshooting guide
  • Metrics export format (planned)

docs/oracle.md (MODIFIED)

  • Updated Health Check section with real implementation details
  • Links to comprehensive monitoring documentation

Tests

oracle-service/tests/health_check.rs (NEW)

Comprehensive test suite structure:

  • Tests for all dependencies up
  • Tests for individual dependency failures
  • Tests for cascading failures
  • Tests for rate limiting detection
  • Tests for timeout handling
  • Status aggregation tests (healthy/degraded/unhealthy)
  • Service readiness flag tests
  • Chaos/fault-injection tests:
    • RPC port blocked
    • Contract deleted
    • Contract paused
    • Lichess DDoS (503)
    • Chess.com rate limit spike
    • Cascading failures
    • Partial recovery scenarios
    • Flaky dependency handling
    • Slow but responsive dependency
    • All dependencies down simultaneously
  • Regression tests:
    • Health check never returns hardcoded "healthy"
    • Contract address not ignored
    • Services properly differentiated

oracle-service/tests/common.rs (NEW)

Test utilities:

  • test_config() — minimal oracle config
  • test_check_result() — create check results for testing
  • test_check_result_with_detail() — check results with error details

Key Design Decisions

1. Real Connectivity Checks

Decision: Perform actual network calls for health checks, not just return static status.

Rationale:

  • Operators have real signal when service is broken
  • Detects failures early before affecting production matches
  • Distinguishes transient (rate limit) from permanent (down) failures

2. Per-Dependency Status

Decision: Return individual status for each dependency, not a single boolean.

Rationale:

  • Lichess down ≠ oracle broken; service can continue with Chess.com
  • Operators can see exactly which component failed
  • Enables granular alerting and incident response

3. Status Aggregation Logic

Decision: Overall status determined by critical dependencies only.

Rationale:

  • healthy: All critical dependencies (RPC, contracts) up
  • degraded: Critical up, non-critical (chess APIs) down
  • unhealthy: Any critical dependency down

This prevents false alarms for non-critical failures while ensuring real issues are caught.

4. Timeout Per Probe

Decision: Each dependency probe has a 5-second timeout.

Rationale:

  • Prevents hung probes from blocking health check loop
  • Allows rapid failure detection
  • Short enough to catch actual network issues
  • Long enough to handle normal latency variation

5. Background Poller (30s Interval)

Decision: Health checks run every 30 seconds in background task.

Rationale:

  • Doesn't block HTTP request path
  • Consistent timing for alerting rules
  • Allows historical tracking of check results
  • Can scale independently from HTTP request load

6. Service Readiness Flag

Decision: service_ready: true only when all critical checks up and not unknown.

Rationale:

  • Kubernetes readiness probe should fail at startup until first check completes
  • Ensures load balancer doesn't route requests to uninitialized service
  • At least one successful health check confirms dependencies reachable

Architecture

Three Concurrent Subsystems

┌─────────────────────────────────────────────────────────────┐
│                    Oracle Service (main)                    │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  ┌──────────────────┐  ┌──────────────────┐  ┌──────────────┐
│  │  HTTP Server     │  │  Health Poller   │  │  Pipeline    │
│  │  Port 8000       │  │  Every 30s       │  │  Poller      │
│  │                  │  │                  │  │  Variable    │
│  │ /health → live   │  │ Calls all 5      │  │  interval    │
│  │ status           │  │ dependency probes│  │              │
│  │                  │  │ Updates shared   │  │ Processes    │
│  │                  │  │ state (RwLock)   │  │ matches      │
│  └──────────────────┘  └──────────────────┘  └──────────────┘
│         │                       │
│         └───────────────────────┴──────────────────────────┐
│                                                             │
│                    Shared State                            │
│  ┌──────────────────────────────────────────────────────┐ │
│  │ Current health status for all 5 dependencies          │ │
│  │ (Stellar RPC, escrow, oracle, lichess, chess.com)    │ │
│  └──────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘

Probe Execution

Each health check cycle (30s interval) runs 5 probes in parallel:

Health Cycle Start
        │
        ├─→ Probe 1: Stellar RPC (getNetwork call)
        ├─→ Probe 2: Escrow Contract (getLedgerEntries)
        ├─→ Probe 3: Oracle Contract (getLedgerEntries)
        ├─→ Probe 4: Lichess API (GET /api/account)
        └─→ Probe 5: Chess.com API (GET /pub/player/profiles)
        │
        (wait for all 5 to complete, max 5s timeout per probe)
        │
        v
Aggregate results into shared state
        │
        v
HTTP clients GET /health → read current state

Response Schema

{
  "status": "healthy|degraded|unhealthy",
  "service_ready": boolean,
  "network": string,
  "contract_escrow": string,
  "contract_oracle": string,
  "oracle_address": string,
  
  "checks": {
    "stellar_rpc": CheckResult,
    "escrow_contract": CheckResult,
    "oracle_contract": CheckResult,
    "lichess_api": CheckResult,
    "chess_com_api": CheckResult
  },
  
  "canary_status": "pending|passed|failed",
  "canary_details": string | null,
  
  "last_full_check_at": ISO8601_timestamp,
  "uptime_seconds": number
}

Where CheckResult is:

{
  "status": "up|degraded|down|rate_limited|unknown",
  "latency_ms": number,
  "last_checked_at": ISO8601_timestamp,
  "consecutive_failures": number,
  "details": string | null
}

Failure Mode Detection

Stellar RPC

Scenario Detection Status
RPC server down Connection refused down
RPC returns 5xx HTTP error down
RPC timeout (>5s) Timeout down
RPC responding slowly (<1s) Latency metric up (latency noted)
Firewall blocking port 443 Connection refused down
DNS resolution fails DNS error down

Contract Reachability

Scenario Detection Status
Contract doesn't exist ledger entry not found (OK) up
RPC error returned RPC error down
Contract address typo Invalid strkey error down
Contract paused Method call returns error (not yet) degraded

Chess API

Scenario Detection Status
API down Connection refused / 5xx down
API rate-limited HTTP 429 rate_limited
API timeout 5s timeout down
API slow (>500ms) Latency metric up (latency noted)
Auth failure (401) HTTP 401 down

Testing Strategy

Unit Tests

  • Status enums serialize/deserialize correctly
  • Status comparison logic (healthy > degraded > unhealthy)
  • Health status display formats

Integration Tests

  • Mock all dependencies with wiremock
  • Verify each dependency failure is detected
  • Verify cascading failures work correctly
  • Verify status aggregation logic

Chaos/Fault-Injection Tests

  • Kill RPC: verify contracts don't get probed unnecessarily
  • Kill contract: verify status reflects that
  • API rate limiting: verify proper classification
  • API timeouts: verify timeout handling
  • Partial outages: verify degraded status
  • Recovery: verify status transitions correctly

Regression Tests

  • Health check must not return hardcoded "healthy"
  • Contract address from config must be actually checked
  • Multiple services must be independently checked

Integration with Kubernetes

Liveness Probe

livenessProbe:
  httpGet:
    path: /health
    port: 8000
  initialDelaySeconds: 10
  periodSeconds: 30
  timeoutSeconds: 5
  failureThreshold: 3
  • Checks every 30 seconds
  • Kills pod after 90 seconds of consecutive failures
  • Pod restart clears any stuck state

Readiness Probe

readinessProbe:
  httpGet:
    path: /health
    port: 8000
  initialDelaySeconds: 5
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 2
  • Checks every 10 seconds
  • Removes from load balancer after 20 seconds of consecutive failures
  • Allows in-place traffic rerouting without pod restart

Alerting Strategy

Alert Rules

Alert Condition SLA Action
OracleServiceUnhealthy status=unhealthy for 5m 1m to page Page on-call, pause contracts
OracleServiceDegraded status=degraded for 15m 10m alert Alert on-call, monitor platform status
OracleLichessRateLimited lichess rate_limited for 5m 5m alert Check queue backlog, contact support
OracleHighLatency RPC latency >1s (p95) for 10m 10m alert Investigate RPC performance

On-Call Escalation

  1. Tier 1 (Immediate): AlertManager → Slack → on-call engineer
  2. Tier 2 (5min): If unacknowledged, PagerDuty page
  3. Tier 3 (15min): If unresolved, escalate to service owner

Monitoring Best Practices

Baseline Metrics

Record in first week of production:

  • Latency distribution for each dependency (p50, p95, p99)
  • Uptime percentage (should be 99.95%+ for critical dependencies)
  • Time to first failure detection (should be < 35 seconds)
  • False positive rate (should be < 1%)

Dashboard Panels

Create Grafana panels for:

  1. Status heatmap (per dependency, color coded by status)
  2. Latency time series (with SLA thresholds)
  3. Consecutive failures (gauge per dependency)
  4. Uptime counter (total seconds since restart)
  5. Alert status (active alerts and firing duration)
  6. Match processing rate (correlation with health status)

Log Aggregation

Send logs to ELK/Datadog with searchable fields:

{
  "timestamp": "2024-06-15T14:32:10Z",
  "level": "warn",
  "service": "oracle-service",
  "component": "health",
  "event": "dependency_check_failed",
  "dependency": "stellar_rpc",
  "status": "down",
  "latency_ms": 5000,
  "error": "connection timeout",
  "consecutive_failures": 1
}

Future Enhancements

Short Term (Next Sprint)

  • Prometheus metrics export (/metrics endpoint)
  • Synthetic canary check (periodic end-to-end test match)
  • Configurable check intervals per dependency
  • Health check history (store last N checks in-memory)

Medium Term (Next Quarter)

  • Custom webhook notifications
  • Contract method health checks (call actual contract methods)
  • Latency percentile tracking (P50, P95, P99)
  • Dependency correlation analysis (detect cascading failures)

Long Term

  • Predictive failure detection (time-series analysis)
  • Automated remediation (pause contracts if unhealthy)
  • Health check as a service (separate microservice)
  • Multi-region health aggregation

Rollout Plan

Phase 1: Deploy (Done)

  • Implement health check module
  • Integrate with main.rs
  • Add documentation
  • Create test suite

Phase 2: Staging (Next)

  • Deploy to staging environment
  • Run chaos/fault-injection tests
  • Validate Kubernetes probe integration
  • Load test: verify no performance impact
  • Document baseline metrics

Phase 3: Production

  • Deploy to production (canary, then full rollout)
  • Enable Prometheus scraping
  • Activate AlertManager rules
  • Configure Slack/PagerDuty notifications
  • Document on-call runbooks
  • Train team on alert response

Phase 4: Validation (1 week)

  • Monitor alert accuracy (no false positives)
  • Verify latency baseline matches recorded values
  • Confirm Kubernetes probes working correctly
  • Gather feedback from on-call rotations

Appendix: Response Schema Reference

See docs/monitoring-health-checks.md for complete field documentation.

Status Values

Overall "status":
  - "healthy" → All critical dependencies operational
  - "degraded" → Critical up, non-critical down
  - "unhealthy" → Any critical dependency down

Per-dependency "status":
  - "up" → Probe succeeded, dependency operational
  - "degraded" → Probe succeeded but higher latency/warnings
  - "down" → Probe failed, dependency unreachable
  - "rate_limited" → API returned 429, will retry
  - "unknown" → Not yet checked (startup only)

Example Responses

Healthy:

{
  "status": "healthy",
  "service_ready": true,
  "checks": {
    "stellar_rpc": {"status": "up", "latency_ms": 52},
    "escrow_contract": {"status": "up", "latency_ms": 120},
    "oracle_contract": {"status": "up", "latency_ms": 98},
    "lichess_api": {"status": "up", "latency_ms": 75},
    "chess_com_api": {"status": "up", "latency_ms": 85}
  }
}

Degraded (Lichess down):

{
  "status": "degraded",
  "service_ready": true,
  "checks": {
    "stellar_rpc": {"status": "up", "latency_ms": 52},
    "escrow_contract": {"status": "up", "latency_ms": 120},
    "oracle_contract": {"status": "up", "latency_ms": 98},
    "lichess_api": {"status": "down", "details": "connection refused"},
    "chess_com_api": {"status": "up", "latency_ms": 85}
  }
}

Unhealthy (RPC down):

{
  "status": "unhealthy",
  "service_ready": false,
  "checks": {
    "stellar_rpc": {"status": "down", "details": "timeout"},
    "escrow_contract": {"status": "down", "details": "RPC unreachable"},
    "oracle_contract": {"status": "down", "details": "RPC unreachable"},
    "lichess_api": {"status": "up", "latency_ms": 75},
    "chess_com_api": {"status": "up", "latency_ms": 85}
  }
}

Completion Checklist

  • Implement real connectivity checks for all 5 dependencies
  • Design per-dependency status schema (up/down/degraded/rate-limited)
  • Implement status aggregation logic
  • Integrate with HTTP handler for /health endpoint
  • Create background health poller (30s interval)
  • Add chaos/fault-injection test suite
  • Document monitoring architecture
  • Document alerting and on-call procedures
  • Create Kubernetes integration guide
  • Add runbooks for critical and degraded alerts
  • Document response times and SLAs
  • Update oracle.md with references

Questions & Answers

Q: Why 30-second health check interval?
A: Balance between rapid failure detection (< 1 minute) and avoiding false positives from transient network jitter. Configurable if needed.

Q: What if a dependency fails once but recovers?
A: Status won't flip to down on a single transient failure. The consecutive_failures counter tracks this. Alert rules should use this to avoid false positives.

Q: Does the health check affect match processing performance?
A: No. Health checks run in a separate background task and use a separate HTTP client. They don't block the pipeline poller or block HTTP requests.

Q: Can I disable health checks?
A: Not currently. But you can filter alerts in AlertManager if needed. Future work may add a configuration flag.

Q: What's the difference between "degraded" and "down"?
A: degraded = latency higher than normal but still responding. down = unreachable or returning errors. The distinction allows for finer-grained alerting.

Q: When should I page on-call?
A: status == "unhealthy" for > 5 minutes. status == "degraded" for > 15 minutes. See alert rules in docs/monitoring-health-checks.md.

References