Refactor DatasetCatalog.load() to source from MCP instead of walking STAC locally

[cboettig]

Refactor DatasetCatalog.load() to source from MCP instead of walking STAC locally

Status — parked / backlog

Design captured below. Will be picked up when one of the forcing functions triggers, not before. The "MCP vs. direct STAC" question for DatasetCatalog.load() has been resolved twice already (#167, #171); the current production answer (direct-fetch + #192 inline-forwarding + #178's collection_url escape hatch) is adequate. Do not re-debate without a forcing function.

Decision history (so we don't go round in circles again)

• Catalog fetches all STAC children instead of only requested collections #150 — DatasetCatalog.load() walks the catalog directly, 1 level.
• Replace DatasetCatalog direct STAC fetching with MCP get_collection #167 (closed) — proposed routing through MCP get_collection. Rejected: MCP on critical boot path, OAuth-walled-app privacy issues, dual-href API change required.
• Boot: drop MCP get_collection preload; build system prompt from the direct STAC read #171 (closed) — dropped MCP get_collection preload that fed the system prompt. Direct-read became the single STAC consumer at boot.
• Pass cached STAC content inline to MCP get_schema (avoid catalog_url refetch) #191 / get_schema: forward cached STAC content inline to MCP #192 (closed, in production) — DatasetCatalog retains _rawStac; get_schema forwards cached JSON to MCP inline. Avoids the refetch without inverting the ownership.
• Document collection_url as the pattern for catalogs with intermediate Collection nodes #178 (closed won't-fix) — per-entry collection_url is the supported pattern for multi-level / mixed-private catalogs (Wyoming). Not a bug.
• Inject inline STAC content for LLM-issued get_stac_details / browse_stac_catalog calls #193 (parked) — inline-forwarding for LLM-issued get_stac_details / browse_stac_catalog. Privacy motivation dissolved with get_schema: forward cached STAC content inline to MCP #192; latency motivation unmeasured.
• This issue (Refactor DatasetCatalog.load() to source from MCP instead of walking STAC locally #197) — re-proposes the Replace DatasetCatalog direct STAC fetching with MCP get_collection #167 direction with batching + soft-fail-fallback + authenticated reads via mcp-data-server#117. Parked: prerequisite turned out to be 2–3 days, no forcing function, production is handled.

Goal

Make mcp-data-server the single owner of STAC topology. DatasetCatalog.load() stops walking the catalog and instead asks MCP for the structured collection content of each entry in layers-input.json. Unifies Wyoming-style private-data deployments with the public-data path; removes the 1-level walk and the collection_url per-entry escape hatch.

Why it's parked (current state is adequate)

• Public catalogs boot via direct fetch + 1-level child walk. Fast, parallel, no MCP dependency at boot.
• Multi-level / private catalogs use per-entry collection_url (Wyoming pattern). Deterministic, deployment-portable.
• MCP get_schema already gets cached STAC inline (get_schema: forward cached STAC content inline to MCP #192). No refetch.
• "Cleaner ownership" and "non-browser MCP-client reach" are real benefits, but speculative — no measured pain point.

Forcing functions to unpark

Reopen active work when one of these triggers:

1. A new deployment shape can't be expressed via collection_url — topology needs >1-level traversal and can't be flattened.
2. A non-browser MCP client (CLI, notebook, second app) needs authenticated reach to in-app private metadata.
3. mcp-data-server#117 ships for unrelated reasons (removes the biggest single prerequisite for free).
4. An independent DatasetCatalog refactor opens this up at no extra cost.

Hard prerequisites

1. mcp-data-server#117 — s3_key/s3_secret/s3_endpoint/s3_scope on the STAC tools. Without it, Wyoming-style apps still need inline forwarding for private leaves. Currently scoped at 2–3 days.
2. Batch endpoint on mcp-data-server — get_collections(catalog_url, collection_ids: [...]) returning {id: collection_dict, ...}. Non-negotiable for perf at 30+ datasets (see Performance). File separately.
3. mcp-credential-proxy extended to forward S3 creds on the new batch tool and the singular STAC tools.
4. Soft-fail fallback in geo-agent — keep the existing direct-fetch path as a fallback when the batched MCP call errors. Lets this land before all upstream prerequisites are buffed.

Design

Proposed change

Replace the walk with a single batched MCP call:

async load(appConfig) {
    this.catalogUrl = appConfig.catalog;
    const ids = appConfig.collections.map(c =>
        typeof c === 'string' ? c : c.collection_id);
    const optionsMap = buildOptionsMap(appConfig.collections);

    const dicts = await mcpClient.callTool('get_collections', {
        catalog_url: appConfig.catalog,
        catalog_token: appConfig.catalog_token,
        collection_ids: ids,
    });

    for (const [id, collection] of Object.entries(dicts)) {
        const options = optionsMap.get(id) || {};
        await this.processCollection(collection, options);
    }
}

_rawStac cache populated from these responses, so #192's inline-forwarding path for get_schema keeps working unchanged.

Performance

Batching is non-negotiable. For ~30+ datasets:

• Today: 1 root-catalog fetch + N parallel S3 leaf fetches → ~1–2 RTTs.
• Per-id MCP calls (naive): N round-trips with full session/JSON-RPC/allowlist overhead. Scales poorly under concurrent app boots.
• One batched MCP call: 1 RTT total. MCP serves cached entries from memory and parallelizes uncached S3 reads server-side.

Without batching, this is a perf downgrade.

Migration of layers-input.json

Most apps don't change. Top-level catalog + collections is exactly what the new path needs. Per-entry collection_url is deprecated but stays functional with a console warning. Per-entry catalog_token similarly stays.

Rollout

1. Refactor lands with fallback active. All deployed apps continue working unchanged.
2. mcp-data-server#117 + batch endpoint ship on their own timeline.
3. Downstream apps opt in (Wyoming first) by removing collection_url overrides.
4. After 1–2 release cycles, remove the fallback walk in a separate PR.

Test surface

• Demo / @main-pinned apps boot with new path against public catalog (MCP cache hit for everything). No collection_url needed.
• Wyoming (after migration) boots with s3://private-wyoming/... reached through mcp-data-server's authenticated S3 path.
• Fallback: simulate MCP outage → verify layers still load via direct-fetch on apps with collection_url; verify clean failure for apps without.
• Boot timing on a 30+ dataset app within 100ms of current.

Cross-repo sequence

mcp-data-server#117 → batch-endpoint issue (TBD) → mcp-credential-proxy extension → this issue → Wyoming layers-input.json simplification (separate PR in boettiger-lab/wyoming). Roll out via geo-agent-ops's normal pinned/unpinned validation order.

Collapses

• Document collection_url as the pattern for catalogs with intermediate Collection nodes #178 — depth questions move to mcp-data-server.
• Inject inline STAC content for LLM-issued get_stac_details / browse_stac_catalog calls #193's privacy motivation — MCP gets first-class authenticated access to in-app metadata.
• Per-entry collection_url workaround in downstream layers-input.json files.

[cboettig]

Prerequisite is bigger than initially scoped — adjusting expectations

mcp-data-server#117 had its design refined after the first review pass. The "small additive PR" framing didn't survive contact with pystac's recursion model: auth has to live inside _TimeoutStacIO.read_text_from_href (not a wrapper), the DuckDB connection lifecycle for the parallel-walker needs a real decision, error paths need credential scrubbing, and s3_scope semantics need to be pinned. Realistic effort estimate is 2-3 days, not an afternoon.

That doesn't change the direction of this issue, but it does change what gates what:

• The soft-fail-back-to-direct-fetch path in this refactor (described in the issue body) is now load-bearing, not a nice-to-have. It's what lets this issue land before mcp-data-server#117 is fully buffed, with private deployments staying on inline forwarding (current production state) until the mcp side is ready.
• A reasonable shipping order:
  1. First, this refactor lands with the fallback path active; all currently-deployed apps continue working unchanged (public catalogs through MCP cache, private deployments still using collection_url + inline forwarding).
  2. Then, mcp-data-server#117 + the batch endpoint ship, on their own timeline.
  3. Then, downstream apps (Wyoming first) opt into the all-MCP path by removing collection_url overrides and relying on credential-forwarded reads.
• The batch endpoint on mcp-data-server is still a hard prerequisite for this refactor (perf at 30+ datasets), independently of feat: add filter_by_query tool — bridge MCP SQL results to map filters #117.

If the inline-forwarding-only world turns out to be acceptable indefinitely (it works today; non-browser MCP clients reading private metadata isn't a current pain point), #117 could be deprioritized without blocking this work — Wyoming-style apps would just keep their current shape.

[cboettig]

Deferring — depends on boettiger-lab/mcp-data-server#117 (also deferred) plus a batch endpoint that would need to be filed and shipped, and the production case is already handled adequately by the existing direct-fetch + per-entry collection_url workaround pattern (most apps don't even need that — Wyoming-style mixed-private deployments do). The refactor's benefit is real but speculative: cleaner ownership, single STAC walker in MCP, non-browser client reach. None of those is currently a pain point.

Revisit when:

• A new deployment shape forces the question (e.g. an app whose catalog topology can't be expressed through collection_url overrides);
• mcp-data-server#117 ships for unrelated reasons; or
• We're already refactoring DatasetCatalog for another purpose.

Design captured above stays valid — the soft-fail-back fallback and the batch-endpoint-as-prerequisite framing are still the right approach if/when this gets picked up.

[cboettig]

Reopening as parked-not-rejected. The design captured in this issue and its comments is the considered way forward if/when there's a forcing function (a deployment shape that can't be expressed via collection_url, or non-browser MCP clients needing private metadata reach). Keeping the issue open preserves the design thinking and prevents re-debating; it does not signal active work. Status: backlog.