Skip to content

Commit 9ce5a63

Browse files
onahiOMOTIonahiOMOTI
committed
feat: implement multi-region replication and disaster recovery (#121)
Add system-wide multi-region replication and DR testing infrastructure targeting 99.99% availability with RPO ≤ 60s and RTO ≤ 5 minutes. Architecture: - Active-passive configuration across three regions: us-east-1 (primary) → eu-west-1 (secondary) → ap-southeast-1 (tertiary) - Synchronous PostgreSQL streaming replication to secondary - Async PostgreSQL streaming to tertiary - Kafka MirrorMaker 2 for topic replication - Redis primary-replica with Sentinel failover - Stellar RPC node per region (stateless switch) Core logic (meter-simulator): - multi-region-replication.js: ReplicationManager with health tracking, lag monitoring, failover orchestration, RPO/RTO enforcement - dr-health-checker.js: Cross-region health probes and failover readiness reports with consecutive-critical detection - dr-canary-analyzer.js: Canary promotion decisions (PROMOTE/HOLD/ROLLBACK) comparing P99 latency, availability, error rate, and replication lag Tests: 89 new tests across 3 suites, all passing Scripts: - scripts/dr-failover.sh: Controlled failover with dry-run, rollback, Prometheus textfile metrics, and operator confirmation prompts - scripts/dr-test.sh: DR validation runner for 5 scenarios (connectivity, replication-lag, failover-simulation, rto-validation, rpo-validation) with JSON and Prometheus output - scripts/dr-canary-promote.sh: Stage-based canary promotion (5% → 25% → 50% → 100%) with SLO validation at each gate Monitoring and alerting: - monitoring/multi-region-dr-alerts.yml: 11 Prometheus alert rules (ReplicationLagHigh, RegionHealthCritical, FailoverRPO/RTOViolation, CrossRegionLatencyHigh, MultiRegionAvailabilityLow, DRTestStale, etc.) - monitoring/multi-region-dr-dashboard.json: Grafana dashboard with 12 panels covering region health, replication lag, failover events, RPO/RTO compliance, canary stage, and DR test history - deploy/service-mesh/multi-region-dr.yaml: PrometheusRule CRD + Istio VirtualService/DestinationRule for cross-region routing Blue-green deployment: - deploy/service-mesh/dr-blue-green.yaml: Istio VirtualService with blue/green/dr-primary/dr-secondary/dr-tertiary subsets, header-based canary routing (x-dr-canary, x-dr-failover), 100ms timeout Dashboard component: - usage-dashboard/src/components/MultiRegionDRPanel.tsx: React component showing region health indicators, replication lag, RPO compliance, last DR test result, and failover event history Documentation: - docs/MULTI_REGION_DR_ARCHITECTURE.md: Full architecture document covering topology, replication strategy, RPO/RTO targets, failover decision matrix, security controls, and test scenarios - docs/runbooks/DR_FAILOVER_RUNBOOK.md: Operator runbook with pre-failover checklist, manual/automatic failover procedure, post-failover validation, failback, and canary promotion guide - README.md: Added Multi-Region DR feature entry and architecture section Closes #121
1 parent f87e341 commit 9ce5a63

17 files changed

Lines changed: 5024 additions & 0 deletions

‎README.md‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ Soroban smart contracts for a decentralized utility metering and streaming proto
1818
- **Grant Stream** — Conservation goals trigger automatic grant matching
1919
- **Scheduled Backup Verification** — Restore-tested database backups with metrics, alerts, and canary rollout guidance
2020
- **Oracle Aggregation Framework** — Multi-provider oracle aggregation with a Chainlink `AggregatorV3Interface` adapter, median consensus, deviation/staleness validation, graceful fallback, and per-provider health monitoring (`contracts/oracle-aggregator`)
21+
- **Multi-Region Replication and Disaster Recovery** — Active-passive cross-region replication (us-east-1 → eu-west-1 → ap-southeast-1) with RPO ≤ 60s, RTO ≤ 5 min, automated health monitoring, blue-green canary promotion, and scheduled DR validation tests (`docs/MULTI_REGION_DR_ARCHITECTURE.md`)
2122

2223
## Project Structure
2324

@@ -102,6 +103,34 @@ Verified via 15 property tests with 100+ randomized cases each, covering pause/r
102103

103104
Staging resilience exercises are governed by the [Chaos Engineering Testing Blueprint](docs/runbooks/chaos-engineering-staging.md). The blueprint defines approved fault scenarios, security guardrails, P99 and availability SLOs, monitoring requirements, and blue-green/canary rollout steps for chaos-enabled staging deployments.
104105

106+
### Multi-Region Replication and Disaster Recovery
107+
108+
The Utility Protocol stack operates across three regions in active-passive configuration to meet its 99.99% availability and < 100 ms P99 targets:
109+
110+
| Region | Role | Replication |
111+
|---|---|---|
112+
| `us-east-1` | Primary (active) | — |
113+
| `eu-west-1` | Secondary (hot standby) | Synchronous PostgreSQL streaming, Kafka MirrorMaker 2 |
114+
| `ap-southeast-1` | Tertiary (warm standby) | Async PostgreSQL streaming, Kafka MirrorMaker 2 |
115+
116+
**Recovery targets:**
117+
- **RPO:** ≤ 60 seconds (maximum data loss on failover)
118+
- **RTO:** ≤ 5 minutes (time to restore service after region failure)
119+
120+
**Key components:**
121+
- `meter-simulator/src/multi-region-replication.js` — Replication state tracking, health monitoring, failover orchestration
122+
- `meter-simulator/src/dr-health-checker.js` — Cross-region health probes and failover readiness reports
123+
- `meter-simulator/src/dr-canary-analyzer.js` — Canary promotion decisions (PROMOTE / HOLD / ROLLBACK)
124+
- `scripts/dr-failover.sh` — Controlled DR failover with dry-run mode and Prometheus metrics
125+
- `scripts/dr-test.sh` — DR validation test runner (connectivity, replication-lag, rto-validation, rpo-validation)
126+
- `scripts/dr-canary-promote.sh` — Canary stage promotion (5% → 25% → 50% → 100%) with SLO gates
127+
- `deploy/service-mesh/dr-blue-green.yaml` — Istio VirtualService/DestinationRule for DR-aware blue-green routing
128+
- `monitoring/multi-region-dr-alerts.yml` — Prometheus alert rules for replication lag, RPO/RTO, and region health
129+
- `monitoring/multi-region-dr-dashboard.json` — Grafana dashboard for DR observability
130+
- `usage-dashboard/src/components/MultiRegionDRPanel.tsx` — React component for DR status in the operator dashboard
131+
132+
See [Multi-Region DR Architecture](docs/MULTI_REGION_DR_ARCHITECTURE.md) and [DR Failover Runbook](docs/runbooks/DR_FAILOVER_RUNBOOK.md) for full details.
133+
105134
### Security Properties
106135

107136
- **Nonce sync** prevents replay attacks on IoT heartbeats

‎deploy/service-mesh/dr-blue-green.yaml‎

Lines changed: 177 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,177 @@
1+
---
2+
# VirtualService: DR-Aware Blue-Green Deployment
3+
#
4+
# Combines the standard blue/green deployment slot routing with DR-region
5+
# failover routing. Traffic flows:
6+
# - x-dr-canary: "true" → green DR slice (canary testing)
7+
# - x-dr-failover: "eu" → eu-west-1 (force secondary, for DR drills)
8+
# - x-dr-failover: "ap" → ap-southeast-1 (force tertiary)
9+
# - default → blue (production) at configurable weights
10+
#
11+
# During normal operation: blue=100%, green=0%
12+
# During canary-5: blue=95%, green=5%
13+
# During canary-25: blue=75%, green=25%
14+
# During canary-50: blue=50%, green=50%
15+
# During production: blue=0%, green=100%
16+
# During DR failover: primary route updated to dr-secondary or dr-tertiary
17+
#
18+
# Weights are managed by scripts/dr-canary-promote.sh via kubectl patch.
19+
apiVersion: networking.istio.io/v1beta1
20+
kind: VirtualService
21+
metadata:
22+
name: utility-contracts-dr-blue-green
23+
namespace: utility-contracts
24+
labels:
25+
app.kubernetes.io/part-of: utility-contracts
26+
app.kubernetes.io/component: dr-traffic-management
27+
spec:
28+
hosts:
29+
- api.utility-contracts.example.com
30+
- utility-api.utility-contracts.svc.cluster.local
31+
gateways:
32+
- utility-contracts-gateway
33+
- mesh
34+
http:
35+
# DR canary: header routes to the green DR slice for canary testing.
36+
- name: dr-canary
37+
match:
38+
- headers:
39+
x-dr-canary:
40+
exact: "true"
41+
route:
42+
- destination:
43+
host: utility-api.utility-contracts.svc.cluster.local
44+
subset: green
45+
weight: 100
46+
timeout: 100ms
47+
retries:
48+
attempts: 2
49+
perTryTimeout: 40ms
50+
retryOn: connect-failure,refused-stream,unavailable,cancelled,5xx
51+
52+
# DR force-secondary: header routes directly to eu-west-1 for DR drills.
53+
- name: dr-force-eu
54+
match:
55+
- headers:
56+
x-dr-failover:
57+
exact: "eu"
58+
route:
59+
- destination:
60+
host: utility-api.utility-contracts.svc.cluster.local
61+
subset: dr-secondary
62+
weight: 100
63+
timeout: 100ms
64+
retries:
65+
attempts: 2
66+
perTryTimeout: 40ms
67+
retryOn: connect-failure,refused-stream,unavailable,cancelled,5xx
68+
69+
# DR force-tertiary: header routes directly to ap-southeast-1.
70+
- name: dr-force-ap
71+
match:
72+
- headers:
73+
x-dr-failover:
74+
exact: "ap"
75+
route:
76+
- destination:
77+
host: utility-api.utility-contracts.svc.cluster.local
78+
subset: dr-tertiary
79+
weight: 100
80+
timeout: 100ms
81+
retries:
82+
attempts: 2
83+
perTryTimeout: 40ms
84+
retryOn: connect-failure,refused-stream,unavailable,cancelled,5xx
85+
86+
# Primary route: blue/green weights. Update weights via dr-canary-promote.sh.
87+
- name: primary
88+
route:
89+
- destination:
90+
host: utility-api.utility-contracts.svc.cluster.local
91+
subset: blue
92+
weight: 100
93+
- destination:
94+
host: utility-api.utility-contracts.svc.cluster.local
95+
subset: green
96+
weight: 0
97+
timeout: 100ms
98+
retries:
99+
attempts: 2
100+
perTryTimeout: 40ms
101+
retryOn: connect-failure,refused-stream,unavailable,cancelled,5xx
102+
103+
---
104+
# DestinationRule: Blue/Green/DR Regional Subsets
105+
#
106+
# Defines five subsets:
107+
# blue — current production deployment slot (label: deployment-slot=blue)
108+
# green — canary/new deployment slot (label: deployment-slot=green)
109+
# dr-primary — us-east-1 (normal active region)
110+
# dr-secondary — eu-west-1 (hot standby)
111+
# dr-tertiary — ap-southeast-1 (warm standby)
112+
apiVersion: networking.istio.io/v1beta1
113+
kind: DestinationRule
114+
metadata:
115+
name: utility-api-dr-blue-green-subsets
116+
namespace: utility-contracts
117+
labels:
118+
app.kubernetes.io/part-of: utility-contracts
119+
app.kubernetes.io/component: dr-traffic-management
120+
spec:
121+
host: utility-api.utility-contracts.svc.cluster.local
122+
trafficPolicy:
123+
tls:
124+
mode: ISTIO_MUTUAL
125+
connectionPool:
126+
tcp:
127+
connectTimeout: 50ms
128+
maxConnections: 1024
129+
http:
130+
http1MaxPendingRequests: 1024
131+
http2MaxRequests: 1024
132+
maxRetries: 3
133+
outlierDetection:
134+
consecutive5xxErrors: 5
135+
interval: 30s
136+
baseEjectionTime: 30s
137+
maxEjectionPercent: 50
138+
subsets:
139+
# Blue deployment slot (current production).
140+
- name: blue
141+
labels:
142+
deployment-slot: blue
143+
trafficPolicy:
144+
tls:
145+
mode: ISTIO_MUTUAL
146+
147+
# Green deployment slot (canary / next version).
148+
- name: green
149+
labels:
150+
deployment-slot: green
151+
trafficPolicy:
152+
tls:
153+
mode: ISTIO_MUTUAL
154+
155+
# Primary region (us-east-1 active).
156+
- name: dr-primary
157+
labels:
158+
dr-region: us-east-1
159+
trafficPolicy:
160+
tls:
161+
mode: ISTIO_MUTUAL
162+
163+
# Secondary region (eu-west-1 hot standby).
164+
- name: dr-secondary
165+
labels:
166+
dr-region: eu-west-1
167+
trafficPolicy:
168+
tls:
169+
mode: ISTIO_MUTUAL
170+
171+
# Tertiary region (ap-southeast-1 warm standby).
172+
- name: dr-tertiary
173+
labels:
174+
dr-region: ap-southeast-1
175+
trafficPolicy:
176+
tls:
177+
mode: ISTIO_MUTUAL

0 commit comments

Comments
 (0)