TESTING-DURING-CEPH-OUTAGE.md misses the cirrus MinIO mirror + cirrus MCP (full coverage, no href remapping) — documents only the lossier source.coop path

[cboettig]

Summary

headless/TESTING-DURING-CEPH-OUTAGE.md documents only the source.coop fallback. There is now a strictly better path — the cirrus MinIO mirror plus the cirrus duckdb-mcp — and grep -ri 'minio|cirrus' *.md in this repo returns nothing, so an operator following the doc during today's outage takes the lossier route without knowing an alternative exists. I hit exactly that: I followed the doc, found it required per-collection href remapping with partial coverage, went looking for something better, and found the mirror by reading mcp-data-server/k8s/cirrus-deployment.yaml.

The two paths, measured today (2026-07-30, s3-west 503 across the board)

	source.coop (documented)	cirrus MinIO (undocumented)
Bucket coverage	partial — doc lists confirmed/missing buckets; public-output not mirrored at all	ca-30x30 29/29 collections 200; tpl 29/29; bosl 15/15; wetlands 12/12; biodiversity 13/13
Path rewriting	per-collection href remap (public-<X>/… → cboettig/<X>/…), "not always 1:1", verify each	none — same bucket names, same paths, host swap only
MCP server	must use dev (dev-duckdb-mcp) for the #261 source.coop rewrite	duckdb-mcp.carlboettiger.info, reads its own in-cluster MinIO
Small-JSON reliability	intermittent Cloudflare 500s; doc adds a localhost-cache workaround	no failures observed across ~100 collection fetches
Data-read path	client→AWS over the public internet	inside the cirrus cluster over minio-svc…:9000 (on-node, faster by design)

Verified end-to-end: ca-30x30's regression slice, real committed config and system prompt, model qwen on dse-nimbus — 12/12 cells completed, no timeouts, and SUM(Acres) over the conserved-areas layer returned 26,471,500 / 52,375,900, matching the published 2025 assessment exactly.

Specific corrections needed

1. Add the cirrus path as the preferred one, keeping source.coop as the fallback for buckets MinIO lacks. Both are legitimate; the doc currently presents the fallback as the only option.
2. The dev-MCP requirement doesn't apply to it. "Use the DEV MCP server" is a consequence of needing the #261 source.coop href rewrite. With the cirrus server, prod-vs-dev is irrelevant — it resolves its own catalog and data from MinIO.
3. "Reliability add-on 1" (localhost JSON cache) is unnecessary on this path. It exists to dodge Cloudflare 500s on data.source.coop; MinIO served every collection JSON first try.
4. "Reliability add-on 2" is inverted. outage-systemprompt-addendum.md steers models away from get_stac_details/browse_stac_catalog because they "resolve through the offline top-level catalog and fail". On cirrus the server-side catalog is the healthy one — those tools work, and I confirmed an agent answering correctly through them alone (list_datasets → browse_stac_catalog → get_stac_details → query). Appending that addendum on this path steers models away from working tools.
5. The "Cluster matrix" section is right but incomplete. The Job still clones committed configs that point at s3-west, so it remains unusable. Worth pointing at the tool that now does the staging: geo-agent-benchmark/scripts/stage_app_config.py fetches an app's committed layers-input.json + system-prompt.md from GitHub and rewrites only the endpoints (verifying each collection URL and reporting drops), leaving the collection list and prompt verbatim so the run still measures the real app. The runbook lives in geo-agent-benchmark/AGENTS.md → "Running when NRP object storage is down"; this doc could simply link it rather than duplicate it.
6. State what remains on NRP on either path: this proxy itself, and kubectl for the proxy-key secret. run.js --proxy-endpoint https://vllm-nimbus.carlboettiger.info/v1 bypasses the proxy but needs NIMBUS_API_KEY and forfeits the request/response logging that LOGGING.md analysis depends on — usually the wrong trade.

Related

• mcp-data-server#346 — browse_stac_catalog advertises the in-cluster STAC_CATALOG_URL, so clients can't self-configure; that's why the mirror had to be found by reading deployment YAML.
• geo-agent#335 — the client still hard-crashes on an unreachable catalog after retries, which is what makes any of this load-bearing.

Filed from ca-30x30 regression work; no changes made to this repo.