Skip to content

Commit 8540332

Browse files
jackharrhyjackharrhy
committed
docs: agent system operator guide
1 parent c094afa commit 8540332

1 file changed

Lines changed: 195 additions & 0 deletions

File tree

‎docs/agent-system.md‎

Lines changed: 195 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,195 @@
1+
# Agent System — Operator Guide
2+
3+
How the distributed agent network works, how to operate it, and what to
4+
do when things break.
5+
6+
## The problem
7+
8+
The production server's IP is blocked by the St. John's WAF (F5 BIG-IP).
9+
The entire domain `map.stjohns.ca` returns a CAPTCHA challenge from the
10+
server. Agents run on volunteer residential IPs that aren't blocked.
11+
12+
## Architecture
13+
14+
```
15+
Volunteer 1 Volunteer 2 Volunteer N
16+
┌─────────┐ ┌─────────┐ ┌─────────┐
17+
│plow-agent│ │plow-agent│ │plow-agent│
18+
└────┬─────┘ └────┬─────┘ └────┬─────┘
19+
│ signed reports │ │
20+
└─────────────────────┼─────────────────────┘
21+
▼
22+
┌───────────────┐
23+
│ plow server │
24+
│ (FastAPI) │
25+
│ │
26+
│ coordinator → │ assigns schedules
27+
│ agent_routes → │ receives reports
28+
│ collector → │ skips direct fetch
29+
└───────────────┘
30+
```
31+
32+
**Data flow:**
33+
34+
1. Agent registers with the server (self-registration, generates its own
35+
ECDSA keypair, sends public key)
36+
2. Admin approves the agent via `/admin`
37+
3. Server assigns a fetch schedule — N agents each fetch every `6*N`
38+
seconds with staggered offsets, so globally there's a fetch every ~6s
39+
4. Agent fetches AVL data from `map.stjohns.ca` using browser-like headers
40+
5. Agent signs the response body with its private key and POSTs to
41+
`/agents/report`
42+
6. Server verifies the signature, parses the AVL data, inserts into DuckDB
43+
7. While agents are active (last report <30s ago), the server's own
44+
collector skips the St. John's source
45+
46+
## Day-to-day operations
47+
48+
### Admin panel
49+
50+
Go to `/admin` on the server. Login with the `ADMIN_PASSWORD` env var.
51+
52+
**What you see:**
53+
54+
- **Collector** section: pause/resume button for St. John's. Use this
55+
when setting up agents for the first time — pause the collector, get
56+
agents approved and running, then resume (or leave paused and let agents
57+
handle it)
58+
- **Agents** table: name, ID, status, health, IP, last seen, reports
59+
(success/failed), and approve/revoke buttons
60+
- **Health** column: `healthy` (green), `degraded` (orange, 5+ consecutive
61+
failures), `hibernating` (red, 30+ consecutive failures)
62+
63+
### Setting up a new agent
64+
65+
1. Have your volunteer download the binary from GitHub Releases, or pull
66+
the Docker image
67+
2. They run it: `./plow-agent --server https://plow.jackharrhy.dev`
68+
3. It generates a keypair, registers with the server, and prints
69+
"Waiting for approval..."
70+
4. You see it appear as "pending" in the admin panel
71+
5. Click "Approve"
72+
6. The agent starts fetching and reporting within seconds
73+
74+
### When an agent gets WAF'd
75+
76+
The agent detects consecutive fetch failures and reacts:
77+
78+
- **Failures 1-4**: normal operation, errors reported to server
79+
- **Failures 5-29**: exponential backoff (6s → 12s → 24s → ... → 10min
80+
cap). The admin panel shows `degraded` (orange)
81+
- **Failures 30+**: agent enters **hibernate mode**. It stops fetching
82+
and instead checks in with the server every 10 minutes. Each checkin
83+
includes a single probe fetch to test if the block has lifted. The
84+
admin panel shows `hibernating` (red)
85+
- **Recovery**: if any fetch succeeds, the agent resets to normal
86+
operation immediately
87+
88+
You don't need to restart agents manually. They'll come back on their own
89+
if the WAF unblocks their IP.
90+
91+
### When the server's IP gets unblocked
92+
93+
If you get the city to whitelist the server IP, you can stop using agents
94+
entirely:
95+
96+
1. Revoke all agents in the admin panel (or just leave them — the server's
97+
own collector will automatically start fetching when no agents have
98+
reported in the last 30 seconds)
99+
2. The collector resumes direct fetching
100+
101+
### Collector pause behavior
102+
103+
The pause button in the admin panel only pauses the **server's direct
104+
polling** of the St. John's AVL source. It does NOT pause agent report
105+
processing — agent data still flows in. This is intentional: the pause
106+
is for the server's own fetching, not for the entire data pipeline.
107+
108+
The pause state is **in-memory only**. If the server restarts, the
109+
collector resumes automatically. This is deliberate — it's an operational
110+
toggle, not a persistent config.
111+
112+
## Authentication
113+
114+
Two separate auth systems:
115+
116+
**Admin auth**: password-based. Set `ADMIN_PASSWORD` env var. Stored as
117+
an HMAC cookie. Used for the `/admin` panel.
118+
119+
**Agent auth**: ECDSA P-256. Each agent generates its own keypair on first
120+
run. The public key is sent during registration. All subsequent requests
121+
(checkin, report) are signed: `SHA-256(body || timestamp_bytes)` signed
122+
with the private key, sent as `X-Agent-Sig` header. The server verifies
123+
against the stored public key. Timestamps must be within 30 seconds.
124+
125+
## Configuration
126+
127+
All via environment variables (or `.env` file — pydantic-settings handles
128+
this automatically):
129+
130+
| Variable | Default | Purpose |
131+
|---|---|---|
132+
| `ADMIN_PASSWORD` | (none) | Required for admin panel |
133+
| `DB_PATH` | `/data/plow.db` | DuckDB file location |
134+
| `SOURCE_ST_JOHNS_ENABLED` | `true` | Enable St. John's source |
135+
| `SOURCE_MT_PEARL_ENABLED` | `true` | Enable Mt. Pearl source |
136+
| `SOURCE_PROVINCIAL_ENABLED` | `true` | Enable Provincial source |
137+
138+
See `.env.dist` for the full template.
139+
140+
## Database
141+
142+
The agents table is managed by migrations 003-005:
143+
144+
| Column | Purpose |
145+
|---|---|
146+
| `agent_id` | SHA-256 fingerprint of public key (16 hex chars) |
147+
| `name` | Human-readable name (set by volunteer) |
148+
| `public_key` | ECDSA P-256 public key PEM |
149+
| `status` | `pending`, `approved`, or `revoked` |
150+
| `consecutive_failures` | Resets to 0 on success, increments on failure |
151+
| `total_reports` | Lifetime successful reports |
152+
| `failed_reports` | Lifetime failed reports |
153+
| `last_seen_at` | Last time the agent submitted any report |
154+
| `ip` | Registration IP (from X-Forwarded-For) |
155+
| `system_info` | OS/arch from agent |
156+
157+
## Release process
158+
159+
Any push to `main` that touches `agent/**` triggers the GitHub Actions
160+
workflow `.github/workflows/release-agent.yml`. It:
161+
162+
1. Creates a tag `agent-YYYY.MM.DD-<short-sha>`
163+
2. Runs goreleaser to cross-compile for linux/darwin/windows (amd64+arm64)
164+
3. Publishes binaries to a GitHub Release
165+
166+
The server's Docker image is built by the existing
167+
`.github/workflows/build-and-push.yml` on pushes to `main`.
168+
169+
## File map
170+
171+
```
172+
src/where_the_plow/
173+
agent_auth.py — ECDSA key generation, signing, verification
174+
agent_routes.py — /agents/register, /checkin, /report
175+
admin_routes.py — /admin/login, /agents/*, /collector/pause|resume
176+
coordinator.py — schedule computation, timestamp validation
177+
collector.py — background polling, pause checks, agent fallback
178+
db.py — agent CRUD, consecutive_failures tracking
179+
migrations/
180+
003_add_agents_table.py
181+
004_agent_status.py
182+
005_agent_consecutive_failures.py
183+
static/admin/ — admin panel (HTML/JS/CSS)
184+
185+
agent/
186+
main.go — entry point, backoff/hibernate loop
187+
client.go — register, checkin, report HTTP calls
188+
fetch.go — AVL fetch with browser mimicry
189+
config.go — credential storage (~/.config/plow-agent/)
190+
crypto.go — ECDSA signing
191+
.goreleaser.yml — cross-compile config
192+
build.sh — local dev build
193+
Dockerfile — container build
194+
k8s.yaml — Kubernetes manifest
195+
```

0 commit comments

Comments
 (0)