Skip to content

Commit dca6426

Browse files
thebenternclaude
andcommitted
ROLLBACK.md: record that the flip has happened
The preconditions block added earlier today said the record was grey-cloud and the flip had not happened yet. That was true when written and is now false: the record is proxied and api.meshtastic.org is served by the meshtastic-api Worker. Leaving it would have been the same hazard it was written to fix, just inverted -- someone reading it mid-incident would believe the Worker was not carrying traffic and that route deletion was a no-op, when it is in fact the rollback. Replaces it with the verified current state, keeps the A-records-vs-CNAME test as the fastest way to tell which state you are in, and records the cutover verification (self-check 12/12 against production, 39/40 on the full Railway diff, the one FAIL being the known iconUrl host-rewrite artifact). Adds one thing that is easy to get wrong during a rollback: the production route was added in the dashboard, not via wrangler.jsonc, where the routes block under env.production is still commented out. So a production deploy does NOT re-create the route, which makes the "disable the Deploy workflow first" step precautionary rather than load-bearing -- and load-bearing again the moment that block is uncommented. Also flags that production routing therefore exists only as dashboard state and is not described by the repo. Finally, `wrangler rollback --env production` now has a version to reach for, which it did not when the earlier note was written. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 parent 52fdd24 commit dca6426

1 file changed

Lines changed: 31 additions & 42 deletions

File tree

ROLLBACK.md

Lines changed: 31 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -1,53 +1,43 @@
11
# Rollback
22

3-
Read this **before** the flip, not during one. Rehearse the clicks.
3+
Read this before you need it, not during an incident. Rehearse the clicks.
44

5-
## Preconditions -- READ FIRST, this is not yet true
5+
## Current state -- the flip is DONE (2026-09-08)
66

7-
> **`api.meshtastic.org` is currently a DNS-only (grey-cloud) CNAME straight to Railway.**
8-
>
9-
> Everything below describes the system *after* the flip, and the flip itself cannot happen until
10-
> that record is set to **Proxied (orange)** in Cloudflare. A Worker Route only sees traffic that
11-
> reaches Cloudflare's edge; against a grey-cloud record, uncommenting the route in
12-
> `wrangler.jsonc` and deploying is a **silent no-op** -- no error, no traffic moved.
13-
> `tools/setup-cloudflare.sh` states the requirement: "It must be PROXIED (orange) or the route
14-
> never sees traffic."
15-
>
16-
> Verified 2026-09-08, four independent ways:
7+
> `api.meshtastic.org` is **proxied** and served by the **`meshtastic-api`** Worker.
178
>
189
> ```
19-
> dig +noall +answer @gene.ns.cloudflare.com api.meshtastic.org
20-
> api.meshtastic.org. 60 IN CNAME api-production-871d.up.railway.app.
21-
> api-production-871d.up.railway.app. 60 IN A 69.46.46.101 # Railway's IP, not CF anycast
22-
> curl -sI https://api.meshtastic.org/ | grep cf-ray # (nothing)
23-
> curl -s -o /dev/null -w '%{http_code}' https://api.meshtastic.org/cdn-cgi/trace # 404
24-
> openssl s_client -connect api.meshtastic.org:443 ... -issuer # Let's Encrypt (Railway's), not
25-
> # Google Trust Services (CF's)
10+
> dig +short api.meshtastic.org -> 104.21.46.221, 172.67.142.226 (CF anycast A, no CNAME)
11+
> curl -sI https://api.meshtastic.org/ -> server: cloudflare, cf-ray: ...
12+
> TLS issuer -> Google Trust Services (Cloudflare)
2613
> ```
2714
>
28-
> For contrast, `apiv2.meshtastic.org` answers on 104.21.46.221 / 172.67.142.226, sends a `cf-ray`,
29-
> serves `/cdn-cgi/trace`, and presents a Google Trust Services cert.
15+
> A proxied record answers with Cloudflare anycast **A records**; a grey-cloud one answers with the
16+
> origin **CNAME**. That difference is the fastest way to tell which state you are in.
17+
>
18+
> Verified at cutover: `parity.mjs --self-check` 12/12 against production, and 39/40 on the full
19+
> diff against Railway. The single FAIL is a harness artifact -- the old server derives `iconUrl`
20+
> from the request Host, so the railway.app baseline emits railway.app URLs; the payload is
21+
> byte-identical.
22+
23+
**Railway is still running and still the rollback target.** Nothing about it changed; the DNS
24+
record underneath still points at it, which is what makes route deletion an instant rollback.
3025
31-
### Order of operations for the flip
26+
### One thing to know before you roll back
3227
33-
1. **Set the record to Proxied** in Cloudflare, and confirm SSL/TLS mode is **Full** for this
34-
hostname before doing so -- proxying moves TLS termination to Cloudflare, and a Flexible or
35-
Full (strict) mismatch against Railway's origin cert breaks every request. With SSL correct
36-
this step is transparent: traffic flows eyeball -> Cloudflare -> Railway, still Railway-served.
37-
Verify with `curl -sI https://api.meshtastic.org/ | grep -i 'cf-ray\|server'` -- expect a
38-
`cf-ray` AND `server: railway-hikari`. This step is independently reversible (toggle back to
39-
grey; the record carries a 60s TTL).
40-
2. **Only then** uncomment the `routes` block under `env.production` in `wrangler.jsonc` and run
41-
the Deploy workflow with `environment: production`. That is the actual cutover.
42-
3. Verify `server: cloudflare` on `api.meshtastic.org`, then soak before touching Railway.
28+
The production route was added **in the dashboard**, not via `wrangler.jsonc` -- the `routes` block
29+
under `env.production` there is still commented out. Two consequences:
4330
44-
Doing 2 before 1 is harmless but accomplishes nothing. Doing 1 alone is a safe, reversible
45-
half-step that proves the SSL path before any traffic changes hands.
31+
- A `wrangler deploy --env production` does **not** re-create the route, so the "disable the Deploy
32+
workflow first" step below is belt-and-braces rather than load-bearing. It is still the right
33+
habit, and it becomes load-bearing the moment that block is uncommented.
34+
- Production routing currently lives only as dashboard state and is not described by the repo.
35+
Uncommenting the block to make config match reality is worth doing deliberately, not by accident.
4636
47-
## The one-line summary (post-flip)
37+
## The one-line summary
4838
49-
Once the record is proxied, `api.meshtastic.org` is served by a Cloudflare **Worker Route**, not a
50-
Custom Domain, and the DNS record still points at Railway underneath. Removing the route hands
39+
`api.meshtastic.org` is served by a Cloudflare **Worker Route**, not a Custom Domain, and the DNS
40+
record still points at Railway underneath. Removing the route hands
5141
traffic straight back to Railway with no DNS change and no TTL to wait out.
5242
5343
## If the Worker is serving something wrong
@@ -80,11 +70,10 @@ curl -sI https://api.meshtastic.org/resource/deviceHardware | grep -iE 'server|c
8070
wrangler rollback --env production
8171
```
8272

83-
Reverts to the previous Worker version. **Check there is one first:** every Deploy run before
84-
2026-09-08 was `--env staging`, so the production Worker script did not exist at all. It has since
85-
been deployed once (run 34267700592, sha 87c7325, no route attached), which means the script now
86-
exists but has a single version -- `rollback` needs a *previous* one, so it only becomes a real
87-
option after the flip deploy makes a second. Until then the route deletion above is the rollback. Because the five consumer-facing JSON documents are
73+
Reverts to the previous Worker version. This is available now: every Deploy run before 2026-09-08
74+
was `--env staging` so the production script did not exist, but `meshtastic-api` has since been
75+
deployed more than once and therefore has a version to fall back to. Route deletion above remains
76+
the faster and broader rollback -- it does not depend on the Worker being healthy at all. Because the five consumer-facing JSON documents are
8877
compiled **into the bundle**, this reverts data and routing together, atomically — there is no
8978
window where a new Worker reads an old object.
9079

0 commit comments

Comments
 (0)