|
1 | 1 | # Rollback |
2 | 2 |
|
3 | | -Read this **before** the flip, not during one. Rehearse the clicks. |
| 3 | +Read this before you need it, not during an incident. Rehearse the clicks. |
4 | 4 |
|
5 | | -## Preconditions -- READ FIRST, this is not yet true |
| 5 | +## Current state -- the flip is DONE (2026-09-08) |
6 | 6 |
|
7 | | -> **`api.meshtastic.org` is currently a DNS-only (grey-cloud) CNAME straight to Railway.** |
8 | | -> |
9 | | -> Everything below describes the system *after* the flip, and the flip itself cannot happen until |
10 | | -> that record is set to **Proxied (orange)** in Cloudflare. A Worker Route only sees traffic that |
11 | | -> reaches Cloudflare's edge; against a grey-cloud record, uncommenting the route in |
12 | | -> `wrangler.jsonc` and deploying is a **silent no-op** -- no error, no traffic moved. |
13 | | -> `tools/setup-cloudflare.sh` states the requirement: "It must be PROXIED (orange) or the route |
14 | | -> never sees traffic." |
15 | | -> |
16 | | -> Verified 2026-09-08, four independent ways: |
| 7 | +> `api.meshtastic.org` is **proxied** and served by the **`meshtastic-api`** Worker. |
17 | 8 | > |
18 | 9 | > ``` |
19 | | -> dig +noall +answer @gene.ns.cloudflare.com api.meshtastic.org |
20 | | -> api.meshtastic.org. 60 IN CNAME api-production-871d.up.railway.app. |
21 | | -> api-production-871d.up.railway.app. 60 IN A 69.46.46.101 # Railway's IP, not CF anycast |
22 | | -> curl -sI https://api.meshtastic.org/ | grep cf-ray # (nothing) |
23 | | -> curl -s -o /dev/null -w '%{http_code}' https://api.meshtastic.org/cdn-cgi/trace # 404 |
24 | | -> openssl s_client -connect api.meshtastic.org:443 ... -issuer # Let's Encrypt (Railway's), not |
25 | | -> # Google Trust Services (CF's) |
| 10 | +> dig +short api.meshtastic.org -> 104.21.46.221, 172.67.142.226 (CF anycast A, no CNAME) |
| 11 | +> curl -sI https://api.meshtastic.org/ -> server: cloudflare, cf-ray: ... |
| 12 | +> TLS issuer -> Google Trust Services (Cloudflare) |
26 | 13 | > ``` |
27 | 14 | > |
28 | | -> For contrast, `apiv2.meshtastic.org` answers on 104.21.46.221 / 172.67.142.226, sends a `cf-ray`, |
29 | | -> serves `/cdn-cgi/trace`, and presents a Google Trust Services cert. |
| 15 | +> A proxied record answers with Cloudflare anycast **A records**; a grey-cloud one answers with the |
| 16 | +> origin **CNAME**. That difference is the fastest way to tell which state you are in. |
| 17 | +> |
| 18 | +> Verified at cutover: `parity.mjs --self-check` 12/12 against production, and 39/40 on the full |
| 19 | +> diff against Railway. The single FAIL is a harness artifact -- the old server derives `iconUrl` |
| 20 | +> from the request Host, so the railway.app baseline emits railway.app URLs; the payload is |
| 21 | +> byte-identical. |
| 22 | +
|
| 23 | +**Railway is still running and still the rollback target.** Nothing about it changed; the DNS |
| 24 | +record underneath still points at it, which is what makes route deletion an instant rollback. |
30 | 25 |
|
31 | | -### Order of operations for the flip |
| 26 | +### One thing to know before you roll back |
32 | 27 |
|
33 | | -1. **Set the record to Proxied** in Cloudflare, and confirm SSL/TLS mode is **Full** for this |
34 | | - hostname before doing so -- proxying moves TLS termination to Cloudflare, and a Flexible or |
35 | | - Full (strict) mismatch against Railway's origin cert breaks every request. With SSL correct |
36 | | - this step is transparent: traffic flows eyeball -> Cloudflare -> Railway, still Railway-served. |
37 | | - Verify with `curl -sI https://api.meshtastic.org/ | grep -i 'cf-ray\|server'` -- expect a |
38 | | - `cf-ray` AND `server: railway-hikari`. This step is independently reversible (toggle back to |
39 | | - grey; the record carries a 60s TTL). |
40 | | -2. **Only then** uncomment the `routes` block under `env.production` in `wrangler.jsonc` and run |
41 | | - the Deploy workflow with `environment: production`. That is the actual cutover. |
42 | | -3. Verify `server: cloudflare` on `api.meshtastic.org`, then soak before touching Railway. |
| 28 | +The production route was added **in the dashboard**, not via `wrangler.jsonc` -- the `routes` block |
| 29 | +under `env.production` there is still commented out. Two consequences: |
43 | 30 |
|
44 | | -Doing 2 before 1 is harmless but accomplishes nothing. Doing 1 alone is a safe, reversible |
45 | | -half-step that proves the SSL path before any traffic changes hands. |
| 31 | +- A `wrangler deploy --env production` does **not** re-create the route, so the "disable the Deploy |
| 32 | + workflow first" step below is belt-and-braces rather than load-bearing. It is still the right |
| 33 | + habit, and it becomes load-bearing the moment that block is uncommented. |
| 34 | +- Production routing currently lives only as dashboard state and is not described by the repo. |
| 35 | + Uncommenting the block to make config match reality is worth doing deliberately, not by accident. |
46 | 36 |
|
47 | | -## The one-line summary (post-flip) |
| 37 | +## The one-line summary |
48 | 38 |
|
49 | | -Once the record is proxied, `api.meshtastic.org` is served by a Cloudflare **Worker Route**, not a |
50 | | -Custom Domain, and the DNS record still points at Railway underneath. Removing the route hands |
| 39 | +`api.meshtastic.org` is served by a Cloudflare **Worker Route**, not a Custom Domain, and the DNS |
| 40 | +record still points at Railway underneath. Removing the route hands |
51 | 41 | traffic straight back to Railway with no DNS change and no TTL to wait out. |
52 | 42 |
|
53 | 43 | ## If the Worker is serving something wrong |
@@ -80,11 +70,10 @@ curl -sI https://api.meshtastic.org/resource/deviceHardware | grep -iE 'server|c |
80 | 70 | wrangler rollback --env production |
81 | 71 | ``` |
82 | 72 |
|
83 | | -Reverts to the previous Worker version. **Check there is one first:** every Deploy run before |
84 | | -2026-09-08 was `--env staging`, so the production Worker script did not exist at all. It has since |
85 | | -been deployed once (run 34267700592, sha 87c7325, no route attached), which means the script now |
86 | | -exists but has a single version -- `rollback` needs a *previous* one, so it only becomes a real |
87 | | -option after the flip deploy makes a second. Until then the route deletion above is the rollback. Because the five consumer-facing JSON documents are |
| 73 | +Reverts to the previous Worker version. This is available now: every Deploy run before 2026-09-08 |
| 74 | +was `--env staging` so the production script did not exist, but `meshtastic-api` has since been |
| 75 | +deployed more than once and therefore has a version to fall back to. Route deletion above remains |
| 76 | +the faster and broader rollback -- it does not depend on the Worker being healthy at all. Because the five consumer-facing JSON documents are |
88 | 77 | compiled **into the bundle**, this reverts data and routing together, atomically — there is no |
89 | 78 | window where a new Worker reads an old object. |
90 | 79 |
|
|
0 commit comments