Skip to content

Commit 8db39e7

Browse files
cboettigcboettig
authored
docs: app-maintainer guide for mirror failover during an S3 outage (#272)
* docs: app-maintainer guide for mirror failover during an S3 outage How to point an app at the MinIO mirror (minio.carlboettiger.info) when s3-west is down: the two independent surfaces (client-side layers via catalog/collection_url host swap; query via mcp_url -> a mirror head or per-request s3_endpoint), the get_schema https-glob caveat, reverting, and the DNS-failover zero-touch alternative. Refs #260/#264/#268. * docs: update mirror-failover query section for the source registry The get_schema https-glob caveat is resolved by the s3config source registry (#264/#271/#276): the STAC tools now return globbable s3:// paths for known mirrors and surface an in-band s3_endpoint/s3_scope hint for unknown ones; tiles honor S3_DEFAULT_ENDPOINT (#275); and mixing sources requires s3_scope (deterministic since #273). Replaces the stale caveat with the S3_SOURCES/S3_DEFAULT_ENDPOINT deploy recipe. * docs: mirror-head caveats — public-output writes (#279) and degraded start (#262)
1 parent 950f333 commit 8db39e7

1 file changed

Lines changed: 111 additions & 0 deletions

File tree

‎docs/guide/mirror-failover.md‎

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
# Running an app on a mirror during an S3 outage
2+
3+
When the primary NRP Ceph endpoint (`s3-west.nrp-nautilus.io`) is unavailable,
4+
an app can be pointed at a public **mirror** of the `public-*` buckets and keep
5+
working. This is app-driven — nothing needs to change on the primary server.
6+
7+
The reference mirror is **`minio.carlboettiger.info`** (MinIO): a drop-in copy of
8+
the NRP `public-*` buckets — same bucket names, same catalog structure,
9+
self-consistent asset hrefs, public/anonymous reads, with CORS + HTTP range
10+
enabled for browser access. (A partial AWS mirror also exists on source.coop; see
11+
[architecture.md](architecture.md) and issue #260. MinIO is the more complete
12+
drop-in.)
13+
14+
## The two data surfaces
15+
16+
An app reads data over **two independent paths**, and both must be pointed at the
17+
mirror to fully ride out an outage:
18+
19+
| Surface | Fetched by | Through the MCP server? |
20+
| --- | --- | --- |
21+
| Collection JSON, PMTiles, COGs (map layers) | the browser (and TiTiler) | **No** — client-side |
22+
| SQL analytics (`query` tool) | the LLM → MCP server → DuckDB | **Yes** |
23+
24+
Because these are independent, they are switched separately.
25+
26+
## 1. Map layers (client-side)
27+
28+
The map layers are fetched directly from the URLs in your app config
29+
(`layers-input.json`) — they never touch the MCP server. Point them at the mirror:
30+
31+
- `catalog`:
32+
`https://s3-west.nrp-nautilus.io/public-data/stac/catalog.json`
33+
→ `https://minio.carlboettiger.info/public-data/stac/catalog.json`
34+
- every `collection_url`: swap the host
35+
`s3-west.nrp-nautilus.io` → `minio.carlboettiger.info`
36+
(the path is unchanged — the bucket names are identical)
37+
38+
That is the only layer change. The mirror's collection JSONs are self-consistent —
39+
their PMTiles / COG / parquet asset hrefs already point at the mirror — so
40+
everything cascades. Leave `titiler_url` as `https://titiler.nrp-nautilus.io`:
41+
TiTiler reads the mirror's COG URLs server-side (verified).
42+
43+
## 2. Query / analytics (through the MCP server)
44+
45+
The server owns endpoint routing via a data-driven **source registry**
46+
(`s3config.py`, #264/#271): it rewrites known mirror hrefs to globbable `s3://`
47+
paths and creates the matching (anonymous, prefix-scoped) DuckDB secrets. The
48+
`query` paths themselves are the `s3://` form, and the STAC tools return them
49+
ready to use — you don't hand-edit query SQL. There are two ways to route
50+
queries to the mirror during an outage:
51+
52+
**a. Point `mcp_url` at a mirror-configured MCP head (recommended).** Deploy the
53+
server with `S3_DEFAULT_ENDPOINT=<mirror host>` (and, so the mirror's own hrefs
54+
rewrite cleanly, register it via `S3_SOURCES` — see [deployment.md](deployment.md)
55+
and issues #268/#264):
56+
57+
```
58+
S3_DEFAULT_ENDPOINT=minio.carlboettiger.info
59+
S3_SOURCES='[{"name":"minio","https_prefix":"https://minio.carlboettiger.info/","s3_prefix":"s3://"}]'
60+
STAC_CATALOG_URL=https://minio.carlboettiger.info/public-data/stac/catalog.json
61+
```
62+
63+
Point the app's `mcp_url` at this head. `s3://public-*` reads (and hex-tile
64+
*reads*, which now honor the same default endpoint — #275) resolve to the
65+
mirror, anonymously, with no per-query changes.
66+
67+
> **Two caveats on a mirror-configured head:**
68+
> - **Hex-tile builds write.** `register_hex_tiles` writes its pyramid output to
69+
> `s3://public-output` on the default backend. NRP Ceph accepts those writes
70+
> anonymously; the mirror bucket likely requires credentials, so expect builds
71+
> to fail at write time until the scoped write secret in
72+
> [#279](https://github.com/boettiger-lab/mcp-data-server/issues/279) is in
73+
> place. `query` reads are unaffected.
74+
> - **Booting mid-outage.** With `STAC_CATALOG_URL` swapped to the mirror as
75+
> above, the head starts normally. If you keep the Ceph catalog URL (or the
76+
> mirror lacks a root catalog), also set `STAC_ALLOW_DEGRADED_START=true`
77+
> (#262) or the fail-fast startup will crashloop until the primary returns —
78+
> discovery is empty in that mode, but inline/known-path queries all work.
79+
80+
**b. Or keep your `mcp_url` and pass routing per query (#264/#267).** For a
81+
source the server doesn't know, `get_stac_details` now returns the derived
82+
`s3://` path **and an in-band ⚠️ line telling you exactly what to pass** — e.g.
83+
`s3_endpoint='minio.carlboettiger.info', s3_scope='s3://public-<name>'`
84+
(anonymous; add `s3_key`/`s3_secret` if private). Following that instruction
85+
routes the read to the mirror.
86+
87+
> **When mixing sources, always pass `s3_scope`.** A per-request `s3_endpoint`
88+
> (or credentials) **without** a scope applies to *every* `s3://` path in that
89+
> query and disables the server default for the request (deterministic since
90+
> #273) — correct for a query hitting only your bucket, wrong for one mixing
91+
> your bucket with catalog data. The scope confines your endpoint to its prefix.
92+
93+
## Reverting
94+
95+
After the outage, flip the `catalog` / `collection_url` hosts back to
96+
`s3-west.nrp-nautilus.io` and point `mcp_url` back at the default MCP server.
97+
98+
## Zero-touch alternative (infrastructure)
99+
100+
If failover is handled at the DNS/proxy layer — resolving
101+
`s3-west.nrp-nautilus.io` to the mirror — then **all four paths** (collection
102+
JSON, PMTiles, COG, and query) redirect transparently with **no app or server
103+
changes at all**. That is the cleanest option when it's available, since every
104+
asset href and query path is already written against `s3-west`.
105+
106+
## Why it's this simple
107+
108+
The server is a stateless, env-configured data-access head; apps carry their own
109+
per-dataset links from any source. See
110+
[architecture/catalog-sourcing.md](../architecture/catalog-sourcing.md) for the
111+
"carry the links" model this failover relies on.

0 commit comments

Comments
 (0)