Skip to content

Commit 66dcf27

Browse files
mannomannXmannomannX
committed
stuff
1 parent 2513e7c commit 66dcf27

14 files changed

Lines changed: 7919 additions & 1161 deletions

‎auth_and_identity_export.txt‎

Lines changed: 3395 additions & 0 deletions
Large diffs are not rendered by default.

‎docs/runbooks/OPS-081-autonomy-verification-production.md‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
# Runbook: Production Control-Plane Autonomy Verification (OPS-081)
2+
3+
## Overview
4+
This runbook details the procedure for executing a single manual control-plane shutdown verification test in Production during a low-traffic maintenance window.
5+
6+
---
7+
8+
## Prerequisites & Safety
9+
- **Timing:** Off-peak maintenance window (e.g., 03:00 AM UTC).
10+
- **Communication:** Notify operations on-call team prior to execution.
11+
- **External Host:** Execute probes from an independent external workstation (outside `10.9.0.0/16` and Hetzner VPC).
12+
13+
---
14+
15+
## Step-by-Step Procedure
16+
17+
1. **Initiate Maintenance Window Mode:**
18+
- Log into `ops-hub` (`10.9.1.2`).
19+
2. **Temporarily Suspend Control-Plane Tunnel:**
20+
```bash
21+
sudo systemctl stop wg-quick@wg0-hub
22+
```
23+
3. **Execute External Autonomy Probe (from external workstation):**
24+
```bash
25+
./deploy/scripts/verify-autonomy.sh production
26+
```
27+
4. **Verify Core Workloads:**
28+
- [ ] `registry.the-toob.com/health` returns `HTTP 200`.
29+
- [ ] `id.the-toob.com/health` returns `HTTP 200`.
30+
- [ ] `ota.the-toob.com/health` returns `HTTP 200`.
31+
- [ ] `fw.the-toob.com/firmware.bin` range request returns `HTTP 416`.
32+
- [ ] MCU device check-in completes successfully.
33+
5. **Restore Control-Plane Tunnel:**
34+
```bash
35+
sudo systemctl start wg-quick@wg0-hub
36+
```
37+
6. **Confirm Telemetry Recovery:**
38+
- Verify Grafana `toob-ops` dashboards resume telemetry scraping within 2 minutes.

‎docs/runbooks/break-glass.md‎

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
# Runbook: Emergency Break-Glass & Out-of-Band Incident Recovery (OPS-022)
2+
3+
## Overview
4+
This runbook defines the emergency out-of-band access procedures when the central control plane (`ops-hub`) or central WireGuard VPN tunnel is completely unreachable, down, or compromised.
5+
6+
> [!CAUTION]
7+
> **Break-Glass Principles:**
8+
> 1. Under normal operation, zero public TCP ports (including SSH port 22) are open to the internet.
9+
> 2. Break-glass procedures must bypass standard WireGuard hub routing **without** permanently relaxing firewall security boundaries.
10+
> 3. All break-glass activations MUST be logged, timed, and audited. Target SLA for regaining administrative SSH access is **< 15 minutes**.
11+
12+
---
13+
14+
## Emergency Access Paths
15+
16+
```
17+
┌────────────────────────────────────────┐
18+
│ ops-hub / WireGuard VPN Down │
19+
└───────────────────┬────────────────────┘
20+
│
21+
┌────────────────────────┴────────────────────────┐
22+
│ │
23+
▼ ▼
24+
┌───────────────────────────────┐ ┌───────────────────────────────┐
25+
│ PATH 1: Out-of-Band Console │ │ PATH 2: Standby Emergency │
26+
│ (Hetzner VNC / Rescue Mode) │ │ WireGuard Interface │
27+
└───────────────┬───────────────┘ └───────────────┬───────────────┘
28+
│ │
29+
▼ ▼
30+
┌───────────────────────────────┐ ┌───────────────────────────────┐
31+
│ Serial VNC or Emergency Key │ │ Activate dormant │
32+
│ Injection via Hetzner API │ │ /etc/wireguard/wg-emergency │
33+
└───────────────────────────────┘ └───────────────────────────────┘
34+
```
35+
36+
---
37+
38+
## Procedure 1: Hetzner Cloud VNC Serial Console Access (Primary Path)
39+
40+
The primary break-glass path uses Hetzner's hardware-isolated serial VNC console. It requires no network services or public ports on the target machine.
41+
42+
### Step 1.1 — Access Hetzner Console
43+
1. Log into the [Hetzner Cloud Console](https://console.hetzner.cloud).
44+
2. Select the target project (e.g., `toob-ops`, `toob-registry`, `toob-identity`, `toob-update`).
45+
3. Click on the affected server node (e.g., `reg-api-fsn1`, `ops-hub`).
46+
4. Click **Console** in the top-right menu to open the VNC web console.
47+
48+
### Step 1.2 — Login via VNC
49+
1. Authenticate with root credentials or emergency operator credentials.
50+
2. If root password login is disabled, use **Hetzner Rescue System**:
51+
- Go to **Rescue** -> **Enable Rescue & Power Cycle**.
52+
- Reboot node into Hetzner Linux Rescue Environment.
53+
- Mount host disk:
54+
```bash
55+
mount /dev/sda1 /mnt
56+
# Copy emergency SSH public key to authorized_keys
57+
echo "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAI..." >> /mnt/root/.ssh/authorized_keys
58+
umount /mnt
59+
reboot
60+
```
61+
62+
---
63+
64+
## Procedure 2: Standby Emergency WireGuard Profile Activation (Secondary Path)
65+
66+
Each spoke node maintains a dormant emergency WireGuard configuration (`/etc/wireguard/wg-emergency.conf.disabled`).
67+
68+
### Step 2.1 — Enable Emergency WireGuard Interface via API or Script
69+
If `ops-hub` is down, run the break-glass helper script from an administrative workstation with Hetzner API access:
70+
71+
```bash
72+
# Check status of ops-hub and spoke nodes
73+
./deploy/scripts/break-glass.sh status
74+
75+
# Enable dormant emergency WireGuard interface on target spoke node
76+
./deploy/scripts/break-glass.sh enable-emergency --server reg-api-fsn1
77+
```
78+
79+
### Step 2.2 — Direct Emergency Dial-Out
80+
Once `wg-emergency` is enabled on the target node, dial out directly to the node's public IP using your emergency WireGuard profile:
81+
82+
```bash
83+
wg-quick up ./wg-admin-keys/toob-emergency.conf
84+
ssh -i ~/.ssh/id_ed25519 root@10.253.0.1
85+
```
86+
87+
---
88+
89+
## Procedure 3: Recovery & Re-Hardening
90+
91+
Once the incident is resolved and `ops-hub` is restored to healthy operation:
92+
93+
1. **Disable Emergency WireGuard Interface:**
94+
```bash
95+
./deploy/scripts/break-glass.sh disable-emergency --server reg-api-fsn1
96+
```
97+
2. **Verify Firewall Integrity:**
98+
Verify that zero public TCP ports respond on the node's public IP:
99+
```bash
100+
nmap -Pn -p 22,80,443,8200 <node-public-ip>
101+
```
102+
3. **Log Incident Metrics:**
103+
Document the break-glass drill or incident metrics in the table below.
104+
105+
---
106+
107+
## Break-Glass Drill Protocol & Log Matrix
108+
109+
| Date | Trigger / Scenario | Executing Operator | Path Used | Time to Access (SLA < 15m) | Status | Notes |
110+
|---|---|---|---|---|---|---|
111+
| 2026-07-31 | Initial OPS-022 Break-Glass Drill | DevOps Lead | Path 1 (VNC) & Path 2 (Emergency WG) | 4 min 12 sec | SUCCESS | Verified out-of-band access and immediate re-hardening |

0 commit comments

Comments
 (0)