docs: reframe around the agent↔cloud-native-data bridge; add Vision + Roadmap (#170)

cboettig · web-flow · commit 51d8fda7140a · 2026-06-07T19:49:09.000-07:00
Reposition the docs from 'SQL access to geospatial data' to the project's role in the larger effort: the bridge that grounds coding agents in STAC metadata and confines them to validated cloud-native engines, runnable locally or on autoscaling k8s and drivable by small open models. - Reframe the home hero and feature cards - Add 'The bigger picture' (vision) page: the three-component ecosystem (data-workflows, mcp-data-server, jupyter-geoagent) and why a bridge - Add a forward-looking Roadmap page: array data via Zarr, hardware- accelerated engines (links the working mcp-gpu-data-server prototype and benchmark issue #42), broader domains, smaller open models - Fix license: MIT → BSD-3-Clause in README and site footer (matches #160)
diff --git a/README.md b/README.md
@@ -1,8 +1,10 @@
-# MCP DuckDB Geospatial Data Server
+# MCP Data Server
 
-**[Documentation](https://boettiger-lab.github.io/mcp-data-server/)**
+**[Documentation](https://boettiger-lab.github.io/mcp-data-server/)** · **[The bigger picture](https://boettiger-lab.github.io/mcp-data-server/guide/vision)**
 
-A [Model Context Protocol (MCP)](https://modelcontextprotocol.io/) server that provides SQL query access to large-scale geospatial datasets stored in S3. Built with DuckDB for high-performance analytics on H3-indexed environmental, biodiversity, and geospatial data.
+An open [Model Context Protocol (MCP)](https://modelcontextprotocol.io/) server that connects AI agents to cloud-native data: it grounds the agent in [STAC](https://stacspec.org/) metadata so it finds the right dataset and reads its schema, and confines it to validated cloud-native engines so it queries terabyte-scale data over S3 without downloading it, misreading it, or silently failing at scale. Today it serves SQL over Parquet via DuckDB with H3 spatial indexing; see the [roadmap](https://boettiger-lab.github.io/mcp-data-server/guide/roadmap) for array (Zarr) and hardware-accelerated engines.
+
+It is one of three open-source components — with [data-workflows](https://boettiger-lab.github.io/data-workflows/) (which produces the AI-ready data and metadata) and [jupyter-geoagent](https://jupyter-geoagent.readthedocs.io/) — that together make the cloud-native stack reachable by the AI tools researchers already use. Runs locally for sensitive data or on autoscaling Kubernetes for scale.
 
 ## Quick Start
 
@@ -314,7 +316,7 @@ Rather than maintaining a forked server deployment per app, private geo-agent ap
 
 ## License
 
-MIT License - See repository for details
+BSD-3-Clause License — see [LICENSE](LICENSE).
 
 ## Contributing
 
diff --git a/docs/.vitepress/config.js b/docs/.vitepress/config.js
@@ -1,12 +1,14 @@
 export default {
   title: 'MCP Data Server',
-  description: 'MCP server providing SQL access to large-scale geospatial datasets via DuckDB and S3.',
+  description: 'An open MCP server that connects AI agents to cloud-native data — grounding them in STAC metadata and validated query engines (DuckDB on S3).',
   base: '/mcp-data-server/',
 
   themeConfig: {
     nav: [
       { text: 'Guide', link: '/guide/quickstart' },
       { text: 'Datasets', link: '/guide/datasets' },
+      { text: 'Vision', link: '/guide/vision' },
+      { text: 'Roadmap', link: '/guide/roadmap' },
       { text: 'GitHub', link: 'https://github.com/boettiger-lab/mcp-data-server', target: '_blank' },
     ],
 
@@ -20,6 +22,13 @@ export default {
           { text: 'Programmatic Access (R & Python)', link: '/guide/programmatic-access' },
         ],
       },
+      {
+        text: 'About',
+        items: [
+          { text: 'The bigger picture', link: '/guide/vision' },
+          { text: 'Roadmap', link: '/guide/roadmap' },
+        ],
+      },
       {
         text: 'Operations',
         items: [
@@ -34,7 +43,7 @@ export default {
     ],
 
     footer: {
-      message: 'Released under the MIT License.',
+      message: 'Released under the BSD-3-Clause License.',
     },
 
     search: {
diff --git a/docs/guide/roadmap.md b/docs/guide/roadmap.md
@@ -0,0 +1,53 @@
+# Roadmap
+
+This page describes where the project is heading. Items here are **directions, not
+commitments** — the [Quick Start](./quickstart) and [Architecture](./architecture) pages
+describe what the server does *today* (DuckDB SQL over Parquet on S3, H3 spatial indexing,
+STAC-driven discovery). The roadmap is the bridge from that proven core toward the broader
+[vision](./vision).
+
+## Beyond tables: array data via Zarr
+
+Today the server queries **tabular** cloud-native data (Parquet) through DuckDB. A growing
+share of science data is **array** data — gridded rasters, imaging stacks, multidimensional
+cubes — for which [Zarr](https://zarr.dev) and xarray are the cloud-native standard. We aim
+to extend the server so an agent can discover and query Zarr arrays through the same
+STAC-grounded interface it uses for tables, without leaving the cloud-native path.
+
+This matters most for the **life sciences**, where spatially-resolved assays — spatial
+transcriptomics, whole-slide microscopy, 3D-genome imaging — are natively array-shaped and
+arrive at terabyte scale. Conventions like OME-Zarr and AnnData already point the way; the
+missing piece is the agentic layer that makes them reachable.
+
+## Hardware-accelerated engines (Polars / GPU)
+
+The server confines agents to **validated cloud-native engines**. DuckDB on CPU is the
+engine today. A working GPU-accelerated prototype —
+[`mcp-gpu-data-server`](https://github.com/boettiger-lab/mcp-gpu-data-server), built on
+Polars / cuDF with KvikIO for fast S3 reads and partition pruning — already exists and is
+being benchmarked head-to-head against this CPU server
+([tracked in issue #42](https://github.com/boettiger-lab/mcp-data-server/issues/42)).
+Early findings are nuanced: S3-I/O-bound queries favor CPU, while compute-heavy queries on
+already-loaded data favor GPU. The goal is to let an agent transparently use whichever
+engine fits the query, all behind the same MCP interface.
+
+## Broader domains
+
+The architecture is **domain-agnostic** — any data with a tabular or array structure and an
+indexable dimension fits the same pattern. Proven first at earth-observation scale, the
+work is being extended to the life sciences in collaboration with a lab that co-developed
+image-based spatial transcriptomics. As the [vision](./vision) describes, the aim is a
+single agentic interface to the cloud-native stack across scientific domains.
+
+## Smaller, more open models
+
+The server already injects all query guidance at call time, so compact open models can
+drive it. We are continuing to tune that guidance so increasingly small, locally-run open
+models can perform real analyses reliably — keeping sensitive data on local hardware and
+reducing dependence on closed models.
+
+---
+
+Have a use case that needs one of these sooner? Open an
+[issue](https://github.com/boettiger-lab/mcp-data-server/issues) — concrete use cases drive
+prioritization.
diff --git a/docs/guide/vision.md b/docs/guide/vision.md
@@ -0,0 +1,65 @@
+# The bigger picture
+
+The MCP Data Server is one part of a larger open-source effort to make
+**terabyte-scale science data reachable by the AI tools researchers already use.**
+
+## The problem
+
+The open-source stack for large-scale data is mature and widely adopted: cloud-native
+serializations (Parquet, Zarr), out-of-core query engines (DuckDB, Polars),
+machine-readable metadata ([STAC](https://stacspec.org)), and object stores streamed by
+range-request instead of downloaded. Yet in practice it stays underused, for two reasons:
+
+1. **Much data sits in legacy or proprietary formats** that lack the standardized
+   metadata these tools — and AI agents — need to interpret it.
+2. **Coding agents reach for the wrong tools.** As assistants like Claude Code become the
+   everyday interface for analysis, they default to the in-memory libraries most familiar
+   to them (e.g. `pandas`), which silently fail at scale and, without metadata, misread the
+   data. The mature stack sits idle not for lack of an audience, but because the agents that
+   audience now turns to don't reach for it.
+
+## Where this server fits
+
+This server is the **link that lets a coding agent drive the cloud-native stack.** It does
+two things a raw database connection cannot:
+
+- **Points the agent at STAC metadata**, so the model finds the right dataset and reads its
+  schema before querying — instead of guessing column names and silently returning wrong
+  answers.
+- **Exposes validated cloud-native engines** (DuckDB today; see the [roadmap](./roadmap))
+  so the model uses out-of-core, streaming query paths instead of loading everything into
+  memory.
+
+It runs **locally** for sensitive data or on **autoscaling Kubernetes** for scale, and the
+query guidance is injected at call time so even **small, locally-run open models** can drive
+it — reducing dependence on closed models and keeping data on the researcher's hardware.
+
+## The three components
+
+| Component | Role |
+|---|---|
+| **[data-workflows](https://boettiger-lab.github.io/data-workflows/)** | Transforms disparate legacy and proprietary datasets into AI-ready, cloud-native formats (Parquet, Zarr) with rich STAC metadata. |
+| **mcp-data-server** (this project) | Connects coding agents to that AI-ready data — grounding them in the STAC metadata and confining them to validated cloud-native engines. |
+| **[jupyter-geoagent](https://jupyter-geoagent.readthedocs.io/)** | A JupyterLab extension integrating with Jupyter-AI — a data persona users drive with commercial *or* fully open models run locally. |
+
+Each is independently useful; together they form an **AI-native bridge** to the
+cloud-native stack the sciences are converging on.
+
+## Why a bridge, not a fork
+
+The pieces this connects — Parquet, Zarr, DuckDB, Polars, STAC, Jupyter, and the
+[Model Context Protocol](https://modelcontextprotocol.io) — are individually mature
+open-source projects with wide adoption. What is new is the **AI-native layer** that lets
+researchers and their agents drive them *together*, at scale.
+
+The leverage is that Jupyter and agentic coding assistants are already everyday tools, so
+the bridge meets that existing base without asking anyone to learn a new toolchain: the
+agent a scientist already uses makes the move from in-memory libraries to the cloud-native
+stack on the user's behalf.
+
+These methods were built and proven at the largest **earth-observation** scales — working
+with partners including NASA, The Nature Conservancy, and California Fish & Wildlife. The
+same architecture is **general-purpose**, and is being extended to the **life sciences**,
+where spatially-resolved data (spatial transcriptomics, whole-slide microscopy,
+3D-genome imaging) now arrives at terabyte scale, beyond the in-memory tools most
+researchers rely on.
diff --git a/docs/index.md b/docs/index.md
@@ -3,30 +3,30 @@ layout: home
 
 hero:
   name: MCP Data Server
-  text: SQL access to geospatial data — for AI agents
-  tagline: An MCP server that lets any LLM client query petabytes of H3-indexed environmental and biodiversity data via DuckDB and S3. No database setup required.
+  text: The bridge between AI agents and cloud-native data
+  tagline: An open Model Context Protocol server that grounds coding agents in STAC metadata and confines them to validated cloud-native engines — so any LLM client can query terabyte-scale data over S3 without downloading it, misreading it, or silently failing at scale.
   actions:
     - theme: brand
       text: Quick Start
       link: /guide/quickstart
+    - theme: alt
+      text: The bigger picture
+      link: /guide/vision
     - theme: alt
       text: Browse Datasets
       link: /guide/datasets
-    - theme: alt
-      text: GitHub
-      link: https://github.com/boettiger-lab/mcp-data-server
 
 features:
-  - title: Zero Configuration
-    details: Point any MCP-compatible LLM client at the hosted endpoint. No database to install or configure.
-  - title: DuckDB on S3
-    details: Queries run directly against Parquet files in S3 using DuckDB — fast columnar analytics without moving data.
-  - title: H3 Spatial Indexing
-    details: All datasets use Uber's H3 hexagonal grid for efficient spatial joins and area calculations across resolutions.
-  - title: Isolated Execution
-    details: Each query runs in a fresh DuckDB instance. Credentials are request-scoped and never shared between clients.
-  - title: STAC Catalog
-    details: Datasets are discoverable through a standard STAC catalog. The agent browses and resolves S3 paths dynamically.
-  - title: Private Data Ready
-    details: Pass S3 credentials per-call to query private buckets alongside public data in the same query.
+  - title: Grounded in STAC metadata
+    details: The agent browses a standard STAC catalog to find the right dataset and read its column schema before writing a query — so it interprets the data correctly instead of guessing.
+  - title: Validated cloud-native engines
+    details: Queries run against Parquet on S3 with DuckDB — fast, out-of-core, columnar. The agent reaches for streaming engines instead of in-memory libraries that silently break at scale.
+  - title: Runs local or at scale
+    details: Run it on your own hardware for sensitive data, or on autoscaling Kubernetes for terabyte workloads. The same server, the same tools, either way.
+  - title: Drivable by small open models
+    details: The query guidance is injected at call time, so even compact, locally-run open models can drive the workflow — reducing dependence on closed models and keeping data on your hardware.
+  - title: Private data ready
+    details: Pass S3 credentials per call to query private buckets alongside public data. Credentials are request-scoped, never logged, and never shared between clients.
+  - title: Part of a larger effort
+    details: One of three open-source components — with data-workflows and jupyter-geoagent — that together make the cloud-native stack reachable by the AI tools researchers already use.
 ---
