DatasetCatalog.load() still fatal after retry exhaustion (and fetches the root even with collections: []) — blocks a working MCP-only degraded mode

[cboettig]

Summary

#287 shipped the retry half of its proposed fix — fetchJson now backs off (attempt 1/3, attempt 2/3) — but not the graceful-degradation half. After retries are exhausted load() still throws on the boot critical path, so a sustained object-store outage remains a hard boot failure. It also fetches the root catalog unconditionally, including when the app has declared it needs nothing from it.

This matters more than it did in July, because there is now a working fallback the app could degrade to: the MCP server carries its own STAC catalog and exposes browse_stac_catalog / get_stac_details. I verified below that an agent can answer correctly using only those, with no client-side catalog at all.

Evidence (today's s3-west 503 outage, ca-30x30 regression work)

A. Crashes even with zero collections. Config: catalog → the dead s3-west URL, collections: [], mcp_url → a healthy MCP server.

[Catalog] Transient fetch error … (attempt 1/3), retrying in 300ms: HTTP 503
[Catalog] Transient fetch error … (attempt 2/3), retrying in 600ms: HTTP 503
[headless] fatal: Error: HTTP 503: https://s3-west.nrp-nautilus.io/public-data/stac/catalog.json
    at DatasetCatalog.fetchJson (app/dataset-catalog.js:1036)
    at async DatasetCatalog.load (app/dataset-catalog.js:59)

The app asked for no collections, so nothing in that document was needed — the fetch is still fatal. Same for a config where every collection carries an explicit collection_url: load() fetches the root at line 59 before it ever checks whether the direct-fetch path covers everything.

B. MCP-only operation works and is correct. Same config, with catalog pointed at a local stub ({"type":"Catalog","links":[]}, served from 127.0.0.1) purely to get past the boot fetch. Still collections: [], no public object-store address anywhere in the config:

question: How many acres of California land are conserved at GAP status 1 or 2?
model:    qwen (dse-nimbus), thinking off
tools:    list_datasets → browse_stac_catalog → get_stac_details → query
result:   26,427,500 acres   (published 2025 assessment: 26,471,461 — 0.17% low)
elapsed:  40.3s, 4 tool calls

The agent found list_datasets empty, fell through to the MCP server's own catalog, read the schema, wrote correct SQL, and reported its method. That is the degraded mode working end-to-end — it just isn't reachable without a stub-catalog hack.

Why this is worth fixing

An in-cluster MCP deployment needs no publicly-reachable object store: it reads S3 over the cluster-internal endpoint, which is also faster (mcp-data-server/k8s/cirrus-deployment.yaml does exactly this). For a headless run there are no PMTiles or COGs to fetch, so the client needs no object-store address of its own either. Today it needs one anyway, solely to satisfy this boot fetch — which meant standing up a public mirror URL to run a benchmark that otherwise touches nothing public. (Filed alongside: mcp-data-server#346, so the server advertises a client-usable catalog URL.)

Proposed change

1. Degrade, don't throw. After retry exhaustion, log a warning and continue with an empty catalog. list_datasets returns [], get_schema returns its existing "Dataset not found … use get_stac_details" error (already worded to redirect), and the agent has a working path via the MCP STAC tools. A degraded app that answers questions beats a blank page.
2. Skip the root fetch when it is not needed — collections: [], or every entry carrying a collection_url. Cheap, and removes the failure mode entirely for configs that never traverse the catalog.
3. Optionally make catalog itself optional, so a headless/MCP-only config is just {"mcp_url": "…"}. Pairs with mcp-data-server#346 (server advertises its own catalog URL): the app could then self-configure from mcp_url alone.

Scope note for whoever picks this up

Degraded mode is not equivalent to normal operation and shouldn't be presented as such: the app's curated collection list is what feeds list_datasets, get_schema, and the dataset paths in the assembled system prompt. Dropping it changes agent behaviour — fine for a smoke test or an outage, wrong for a benchmark that is supposed to measure the production app. The goal here is "stays up and useful during an outage", not "catalog is optional in general".

Filed from ca-30x30 regression work; no changes made to this repo. Follows #287 (retry shipped, degradation didn't).