|
| 1 | +# Runbook: Incident Response |
| 2 | + |
| 3 | +Procedures for detecting, classifying, escalating, and resolving incidents in SubTrackr. |
| 4 | + |
| 5 | +--- |
| 6 | + |
| 7 | +## Severity Classification |
| 8 | + |
| 9 | +| Severity | Criteria | Initial Response | Escalation | |
| 10 | +| -------- | -------- | ---------------- | ---------- | |
| 11 | +| P1 — Critical | Service down, mass payment failures, data loss, security breach | 15 min | Immediate — wake on-call lead | |
| 12 | +| P2 — High | Failure rate >10%, notification outage, contract unreachable | 1 hour | After 30 min without resolution | |
| 13 | +| P3 — Medium | Single-user billing issue, degraded performance | 4 hours | Next business day if unresolved | |
| 14 | +| P4 — Low | UI glitch, minor doc gap, non-critical warning | Next business day | N/A | |
| 15 | + |
| 16 | +--- |
| 17 | + |
| 18 | +## Incident Response Lifecycle |
| 19 | + |
| 20 | +``` |
| 21 | +Detect → Triage → Contain → Investigate → Resolve → Post-mortem |
| 22 | +``` |
| 23 | + |
| 24 | +### 1. Detect |
| 25 | + |
| 26 | +Alerts fire from: |
| 27 | +- `MonitoringService` — transaction failure rate >30% triggers `high-failure-rate` alert |
| 28 | +- `MonitoringService` — avg gas >500,000 triggers `gas-spike` alert |
| 29 | +- `AlertingService` — dispatches to Slack / PagerDuty / console |
| 30 | +- Manual report from user or merchant |
| 31 | + |
| 32 | +### 2. Triage |
| 33 | + |
| 34 | +Acknowledge the alert in PagerDuty within the SLA window. Determine: |
| 35 | + |
| 36 | +- Is the contract reachable on Soroban RPC? |
| 37 | +- Is the failure isolated (single user/plan) or widespread? |
| 38 | +- Is there a security component (unauthorized access, data exposure)? |
| 39 | + |
| 40 | +```bash |
| 41 | +# Quick health check — get total subscription count |
| 42 | +soroban contract invoke \ |
| 43 | + --id $CONTRACT_ID \ |
| 44 | + --network $NETWORK \ |
| 45 | + -- get_subscription_count |
| 46 | +``` |
| 47 | + |
| 48 | +### 3. Contain |
| 49 | + |
| 50 | +Stop the bleeding before investigating root cause. |
| 51 | + |
| 52 | +| Scenario | Containment Action | |
| 53 | +| -------- | ------------------ | |
| 54 | +| Mass payment failures | Pause affected plans via `deactivate_plan` | |
| 55 | +| Compromised admin key | Rotate key; redeploy contract if necessary | |
| 56 | +| Runaway charge loop | Identify caller; block at RPC level if possible | |
| 57 | +| Corrupted local state | Trigger DR failover (see [DISASTER_RECOVERY_RUNBOOK.md](../DISASTER_RECOVERY_RUNBOOK.md)) | |
| 58 | + |
| 59 | +### 4. Investigate |
| 60 | + |
| 61 | +Check monitoring dashboard: |
| 62 | + |
| 63 | +```ts |
| 64 | +const dashboard = monitoringService.getDashboard(); |
| 65 | +// { |
| 66 | +// totalTransactions, successRate, failureCount, |
| 67 | +// avgGasUsed, activeAlerts, recentMetrics |
| 68 | +// } |
| 69 | +``` |
| 70 | + |
| 71 | +Query audit log for suspicious activity: |
| 72 | + |
| 73 | +```ts |
| 74 | +const events = auditService.query({ |
| 75 | + from: Date.now() - 3_600_000, // last hour |
| 76 | + action: 'SUBSCRIPTION_CANCELLED', |
| 77 | +}); |
| 78 | +``` |
| 79 | + |
| 80 | +Check active alerts: |
| 81 | + |
| 82 | +```ts |
| 83 | +const alerts = monitoringService.getActiveAlerts(); |
| 84 | +``` |
| 85 | + |
| 86 | +### 5. Resolve |
| 87 | + |
| 88 | +Apply fix. Resolve the alert once confirmed stable: |
| 89 | + |
| 90 | +```ts |
| 91 | +monitoringService.resolveAlert(alertId); |
| 92 | +``` |
| 93 | + |
| 94 | +Notify affected users via notification service if billing was impacted: |
| 95 | + |
| 96 | +```ts |
| 97 | +await notificationService.presentChargeFailedNotification(sub, 'Service disruption — no charge applied'); |
| 98 | +``` |
| 99 | + |
| 100 | +### 6. Post-mortem |
| 101 | + |
| 102 | +For P1/P2 incidents, complete a post-mortem within 48 hours covering: |
| 103 | + |
| 104 | +- Timeline of events |
| 105 | +- Root cause |
| 106 | +- Impact (users affected, revenue impact) |
| 107 | +- Corrective actions with owners and due dates |
| 108 | + |
| 109 | +--- |
| 110 | + |
| 111 | +## Common Incident Scenarios |
| 112 | + |
| 113 | +### Scenario A — High Transaction Failure Rate |
| 114 | + |
| 115 | +**Alert:** `high-failure-rate` (failure rate >30%) |
| 116 | + |
| 117 | +**Steps:** |
| 118 | +1. Check `dashboard.failureCount` and `dashboard.recentMetrics` |
| 119 | +2. Identify if failures are concentrated on a specific plan or token |
| 120 | +3. Verify token contract is operational on the relevant chain |
| 121 | +4. If token contract is down, deactivate affected plans temporarily |
| 122 | +5. Resolve alert once failure rate drops below threshold |
| 123 | + |
| 124 | +--- |
| 125 | + |
| 126 | +### Scenario B — Contract Unreachable |
| 127 | + |
| 128 | +**Symptoms:** All `soroban contract invoke` calls time out or return RPC errors. |
| 129 | + |
| 130 | +**Steps:** |
| 131 | +1. Check Stellar network status: https://status.stellar.org |
| 132 | +2. Try alternate RPC endpoint (testnet: `https://soroban-testnet.stellar.org`, mainnet: `https://soroban.stellar.org`) |
| 133 | +3. If network-wide outage, communicate status to users; no action on contract needed |
| 134 | +4. If isolated RPC issue, switch `SOROBAN_RPC_URL` env var and redeploy app config |
| 135 | + |
| 136 | +--- |
| 137 | + |
| 138 | +### Scenario C — Unauthorized Refund Approvals |
| 139 | + |
| 140 | +**Symptoms:** Unexpected `refund_approved` events in contract event stream. |
| 141 | + |
| 142 | +**Steps:** |
| 143 | +1. Immediately rotate the admin key |
| 144 | +2. Query all recent `approve_refund` calls via Soroban event stream |
| 145 | +3. Assess financial impact |
| 146 | +4. If contract admin key is compromised, redeploy contract with new admin |
| 147 | +5. File security advisory (see [security.md](../security.md)) |
| 148 | + |
| 149 | +--- |
| 150 | + |
| 151 | +### Scenario D — Notification Delivery Failure |
| 152 | + |
| 153 | +**Symptoms:** Users not receiving billing reminders or charge notifications. |
| 154 | + |
| 155 | +**Steps:** |
| 156 | +1. Check Expo push notification service status |
| 157 | +2. Verify `notificationService.getPermissionStatus()` returns `GRANTED` for affected users |
| 158 | +3. Confirm `syncRenewalReminders` is being called after subscription mutations |
| 159 | +4. Check Android notification channel is configured: `ensureAndroidNotificationChannel()` |
| 160 | +5. Re-sync reminders manually if needed |
| 161 | + |
| 162 | +--- |
| 163 | + |
| 164 | +## Escalation Contacts |
| 165 | + |
| 166 | +| Condition | Escalate To | |
| 167 | +| --------- | ----------- | |
| 168 | +| P1 security incident | Security team + on-call lead immediately | |
| 169 | +| Contract redeployment needed | Contract admin key holder | |
| 170 | +| Stellar network outage | Monitor https://status.stellar.org; no internal escalation | |
| 171 | +| Persistent P2 >2 hours | Engineering lead | |
0 commit comments