experimental image: raster + zarr extensions on a separate tag and endpoint

cboettig · cboettig · commit eb331ebc42a5 · 2026-08-05T23:22:30.000Z
Adds a second image (Dockerfile.experimental -&gt; :experimental) layered on the
default one, carrying the duckdb raster and zarr community extensions, plus a
testing-only endpoint at experimental-duckdb-mcp.nrp-nautilus.io.

raster gives RT_ReadCells/RT_Read (GDAL) so a small GeoTIFF can be joined to
catalog GeoParquet directly, without a hex ingest first; zarr gives read_zarr.
Both stay off the default endpoint: raster pulls GDAL into the query path, and
RT_ReadCells materializes one row per pixel, which is only safe for small
rasters.

One codebase serves both tags. server.py LOADs whatever EXTRA_DUCKDB_EXTENSIONS
names per connection (unset on the default image, so behaviour is unchanged) and
injects experimental-extensions.md into the query tool description only where the
extras are actually loaded. Extension names are restricted to identifier
characters since LOAD takes no parameters.

The guidance carries the two traps found while testing: RT_ReadCells emits
GEOMETRY('EPSG:4326') while catalog GeoParquet is GEOMETRY('OGC:CRS84'), so the
join needs ST_SetCRS; and unfamiliar rasters need a cols*rows check before read,
with the hex asset as the answer for anything large.
diff --git a/.github/workflows/docker-experimental.yml b/.github/workflows/docker-experimental.yml
@@ -0,0 +1,90 @@
+name: Build experimental Docker image
+
+# Builds ghcr.io/boettiger-lab/mcp-data-server:experimental from
+# Dockerfile.experimental — the default image plus the `raster` and `zarr`
+# community extensions (issue #354).
+#
+# Deliberately NOT triggered by pushes to main. This tag backs a single testing
+# endpoint (experimental-duckdb-mcp) and is meant to move only when someone is
+# actually exercising it; an automatic rebuild on every main push would swap the
+# image under a live test session.
+#
+# BASE_TAG selects which default image to layer on. It defaults to :main, so the
+# experimental build inherits main's dependency and stock-extension layers while
+# `COPY . /app` overlays the code from *this* ref — that is what lets the tag carry
+# server changes before they merge. When main's deps move, re-run this workflow.
+on:
+  push:
+    branches: ['experimental/**']
+  workflow_dispatch:
+    inputs:
+      base_tag:
+        description: 'Default image tag to layer on (e.g. main, v0.8.12, or a sha)'
+        required: false
+        default: 'main'
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      packages: write
+    steps:
+      - uses: actions/checkout@93cb6efe18208431cddfb8368fd83d5badbf9bfd # v5.0.1
+
+      - name: Log in to GHCR
+        uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3.7.0
+        with:
+          registry: ghcr.io
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Set up Buildx
+        uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3.12.0
+
+      - name: Build and push
+        id: build
+        uses: docker/build-push-action@10e90e3645eae34f1e60eeb005ba3a3d33f178e8 # v6.19.2
+        with:
+          context: .
+          file: Dockerfile.experimental
+          push: true
+          tags: ghcr.io/boettiger-lab/mcp-data-server:experimental
+          build-args: |
+            BASE_TAG=${{ github.event.inputs.base_tag || 'main' }}
+            APP_VERSION=experimental
+            GIT_SHA=${{ github.sha }}
+          # Always pull, so a moving BASE_TAG (:main) is actually re-resolved
+          # rather than served from a stale local base.
+          pull: true
+          cache-from: type=gha
+          cache-to: type=gha,mode=max
+
+      - name: Smoke-test image
+        run: |
+          image="ghcr.io/boettiger-lab/mcp-data-server@${{ steps.build.outputs.digest }}"
+          docker pull "$image"
+          # Stock extensions must still work, and both extras must load and expose
+          # their entry points — a silently-missing extension would otherwise only
+          # surface as a confusing "function does not exist" at query time.
+          docker run --rm "$image" python -c "
+          import duckdb
+          c = duckdb.connect()
+          for e in ('httpfs', 'spatial', 'h3', 'raster', 'zarr'):
+              c.sql(f'LOAD {e}')
+          c.sql('SELECT h3_latlng_to_cell(37.8, -122.4, 5)')
+          c.sql('SELECT ST_Point(0, 0)')
+          for fn in ('RT_ReadCells', 'read_zarr'):
+              n = c.sql(f\"SELECT count(*) FROM duckdb_functions() WHERE function_name ILIKE '{fn}'\").fetchone()[0]
+              assert n, f'{fn} missing'
+          print('OK: httpfs/spatial/h3 + raster/zarr loaded and functional')
+          "
+
+      - name: Report image digest
+        run: |
+          {
+            echo '### Experimental image digest'
+            echo '```'
+            echo "${{ steps.build.outputs.digest }}"
+            echo '```'
+          } >> "$GITHUB_STEP_SUMMARY"
diff --git a/Dockerfile.experimental b/Dockerfile.experimental
@@ -0,0 +1,30 @@
+# Experimental image: the default server plus the `raster` and `zarr` community
+# extensions. Published as :experimental, deployed to experimental-duckdb-mcp
+# (k8s/experimental-*.yaml). Never promoted to the default endpoint — issue #354.
+#
+# Layered on the default image rather than duplicating it, so the Python deps and
+# the stock httpfs/spatial/h3 extension layer are shared and cannot drift. CI passes
+# BASE_TAG=<sha> so the experimental image is pinned to the exact base build from
+# the same run, not to whatever :main happens to be at pull time.
+ARG BASE_TAG=main
+FROM ghcr.io/boettiger-lab/mcp-data-server:${BASE_TAG}
+
+# Community extensions. These pull GDAL into the query path (raster) and add a
+# chunked-array reader (zarr) — the reason this image is kept separate from the
+# default one rather than gated behind an env var alone.
+RUN python -c "import duckdb; c = duckdb.connect(); c.sql('INSTALL raster FROM community; INSTALL zarr FROM community')"
+
+# Overlay this branch's code on the base image's dependency layers, so the
+# experimental tag can carry server changes that have not landed on main yet.
+COPY . /app
+
+# Read by server.py: LOADs these per connection and injects
+# experimental-extensions.md into the query tool description. Unset on the
+# default image, which therefore behaves exactly as before.
+ENV EXTRA_DUCKDB_EXTENSIONS=raster,zarr
+
+ARG APP_VERSION=experimental
+ARG GIT_SHA=unknown
+ENV APP_VERSION=$APP_VERSION GIT_SHA=$GIT_SHA
+
+CMD ["python", "server.py"]
diff --git a/experimental-extensions.md b/experimental-extensions.md
@@ -0,0 +1,53 @@
+
+### 🧪 EXPERIMENTAL EXTENSIONS (this deployment only)
+
+This endpoint loads two community extensions beyond the stock set. They are **not**
+available on the default endpoint — do not assume them elsewhere.
+
+#### `raster` — read GeoTIFF/COG directly
+
+`RT_ReadCells(path)` returns **one row per pixel**: `(id, x, y, geometry, col, row, band_1, …)`.
+`RT_Read(path)` returns one row per tile with an `RT_DATACUBE` band, for band algebra
+(`RT_CubeStats`, `RT_CubeClip`, `RT_CubeBurn`). Paths may be S3/HTTP via GDAL's
+`/vsicurl/` or `/vsis3/` prefixes.
+
+⛔ **Size guard — `RT_ReadCells` materializes every pixel.** A 92×81 fire-severity
+raster is 7,452 rows and returns in seconds; a continental COG is billions and will
+exhaust the pod. Before reading an unfamiliar raster, check its dimensions with
+`SELECT cols, rows FROM RT_Read(path)` and stop if `cols * rows` exceeds ~10 million.
+For anything larger, use the dataset's `…/hex/h0=*/…` asset — every raster collection
+in the catalog is already aggregated to H3 by the ingest pipeline, and that path is
+partition-pruned and far cheaper. `RT_ReadCells` is for small rasters not yet ingested.
+
+⚠️ **CRS type mismatch.** `RT_ReadCells` emits `GEOMETRY('EPSG:4326')`; catalog
+GeoParquet is `GEOMETRY('OGC:CRS84')`. Both hold lon/lat, but DuckDB rejects the join
+on the type difference. Re-tag the raster side — no reprojection is involved:
+
+```sql
+WITH px AS (
+  SELECT band_1 AS value, ST_SetCRS(geometry, 'OGC:CRS84') AS g
+  FROM RT_ReadCells('/vsicurl/https://example.org/severity.tif')
+  WHERE band_1 > 0                       -- drop nodata BEFORE the join
+)
+SELECT f.name, count(*) AS n_px, round(avg(px.value), 3) AS mean_value
+FROM px LEFT JOIN read_parquet('<geoparquet>') f ON f.geom.ST_Contains(px.g)
+GROUP BY 1;
+```
+
+Filter the raster's nodata sentinel in the CTE (see §7) — it is usually a large
+negative value like `-9999` that will wreck any average.
+
+For area, do not multiply a pixel count by a constant in a geographic CRS: pixel
+ground area varies with latitude. Assign cells with `h3_latlng_to_cell(y, x, res)`
+and sum `h3_cell_area()`, or transform to a projected CRS with
+`always_xy := true` (see §5).
+
+#### `zarr` — read Zarr stores
+
+`read_zarr(path)`, plus `read_zarr_groups(path)` and `read_zarr_metadata(path)` for
+discovery. Call the metadata/groups functions first to learn array names, shapes and
+chunking; the same size discipline as `RT_ReadCells` applies — a Zarr array is
+typically a full datacube, so slice on its coordinate columns before aggregating.
+
+**Report which extension produced a number** when you use either one, so the user
+knows the result came from the experimental path.
diff --git a/k8s/experimental-deployment.yaml b/k8s/experimental-deployment.yaml
@@ -0,0 +1,77 @@
+# Testing endpoint for the :experimental image (raster + zarr extensions, issue #354).
+# Not part of the prod or dev rotation — nothing should depend on this being up.
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: experimental-duckdb-mcp
+  namespace: biodiversity
+spec:
+  # Single replica on purpose: a testing endpoint on a shared cluster, and a
+  # single pod keeps a test session pinned to one DuckDB process.
+  replicas: 1
+  selector:
+    matchLabels:
+      app: experimental-duckdb-mcp
+  template:
+    metadata:
+      labels:
+        app: experimental-duckdb-mcp
+    spec:
+      affinity:
+        nodeAffinity:
+          requiredDuringSchedulingIgnoredDuringExecution:
+            nodeSelectorTerms:
+              - matchExpressions:
+                  - key: topology.kubernetes.io/region
+                    operator: In
+                    values: ["us-west"]
+      containers:
+      - name: server
+        image: ghcr.io/boettiger-lab/mcp-data-server:experimental
+        imagePullPolicy: Always
+        env:
+        - name: STAC_CATALOG_URL
+          value: "https://s3-west.nrp-nautilus.io/public-data/stac/catalog.json"
+        - name: MCP_PUBLIC_BASE_URL
+          value: "https://experimental-duckdb-mcp.nrp-nautilus.io"
+        - name: STAC_ALLOW_DEGRADED_START
+          value: "true"
+        # Also baked into the image; set here too so `kubectl describe` shows which
+        # extras a running pod believes it has without cracking open the image.
+        - name: EXTRA_DUCKDB_EXTENSIONS
+          value: "raster,zarr"
+        - name: POD_MEMORY_LIMIT
+          valueFrom:
+            resourceFieldRef:
+              containerName: server
+              resource: limits.memory
+        resources:
+          # Small requests (NRP's "Ignored" utilization bucket), and limits well
+          # below prod's 160Gi/64: this endpoint is for small rasters and Zarr
+          # slices. RT_ReadCells materializes every pixel, so a modest ceiling is
+          # a feature — an over-large read fails here instead of evicting
+          # co-tenants on the node.
+          requests:
+            memory: 2Gi
+            cpu: 250m
+          limits:
+            memory: 32Gi
+            cpu: 16
+        ports:
+        - containerPort: 8000
+        readinessProbe:
+          httpGet:
+            path: /healthz
+            port: 8000
+          initialDelaySeconds: 15
+          periodSeconds: 5
+          timeoutSeconds: 2
+          failureThreshold: 3
+        livenessProbe:
+          httpGet:
+            path: /healthz
+            port: 8000
+          initialDelaySeconds: 30
+          periodSeconds: 30
+          timeoutSeconds: 10
+          failureThreshold: 6
diff --git a/k8s/experimental-ingress.yaml b/k8s/experimental-ingress.yaml
@@ -0,0 +1,32 @@
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: experimental-mcp-server-duckdb-ingress
+  namespace: biodiversity
+  annotations:
+    haproxy-ingress.github.io/cors-enable: "true"
+    haproxy-ingress.github.io/cors-allow-origin: "*"
+    haproxy-ingress.github.io/cors-allow-methods: "GET, POST, OPTIONS"
+    haproxy-ingress.github.io/cors-allow-headers: "DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Authorization,mcp-session-id"
+    haproxy-ingress.github.io/cors-allow-credentials: "true"
+    haproxy-ingress.github.io/cors-max-age: "86400"
+    haproxy-ingress.github.io/balance-algorithm: leastconn
+    haproxy-ingress.github.io/timeout-client: "600s"
+    haproxy-ingress.github.io/timeout-server: "600s"
+    haproxy-ingress.github.io/timeout-tunnel: "3600s"
+spec:
+  ingressClassName: haproxy
+  tls:
+    - hosts:
+        - experimental-duckdb-mcp.nrp-nautilus.io
+  rules:
+    - host: experimental-duckdb-mcp.nrp-nautilus.io
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: experimental-mcp-server-duckdb
+                port:
+                  number: 8000
diff --git a/k8s/experimental-service.yaml b/k8s/experimental-service.yaml
@@ -0,0 +1,12 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: experimental-mcp-server-duckdb
+  namespace: biodiversity
+spec:
+  selector:
+    app: experimental-duckdb-mcp
+  ports:
+    - protocol: TCP
+      port: 8000
+      targetPort: 8000
diff --git a/server.py b/server.py
@@ -77,6 +77,21 @@ def parse_setup_sql(content):
 H3_RAW = load_text_file("h3-guide.md")
 ROLE_RAW = load_text_file("assistant-role.md")
 
+# Opt-in DuckDB extensions beyond the stock httpfs/spatial/h3 set (issue #354).
+# The experimental image (Dockerfile.experimental) installs raster + zarr and sets
+# EXTRA_DUCKDB_EXTENSIONS; the default image leaves it unset and behaves exactly as
+# before. LOAD is per-connection, like every statement in SETUP_SQL, so this is a
+# deployment knob rather than a code fork. Names are restricted to identifier
+# characters — they are interpolated into LOAD, which takes no parameters.
+_EXT_NAME_RE = re.compile(r"^[A-Za-z0-9_]+$")
+EXTRA_EXTENSIONS = [
+    e for e in (s.strip() for s in os.environ.get("EXTRA_DUCKDB_EXTENSIONS", "").split(","))
+    if e and _EXT_NAME_RE.match(e)
+]
+# Guidance for the extras is injected only where they are actually loaded, so the
+# default deployment's tool description is unchanged.
+EXTRAS_RAW = load_text_file("experimental-extensions.md") if EXTRA_EXTENSIONS else ""
+
 # -------------------------------------------------------------------------
 # 3. CONTEXT INJECTION (PROMPT ENGINEERING)
 # -------------------------------------------------------------------------
@@ -111,6 +126,7 @@ def parse_setup_sql(content):
 
 ### 📐 H3 SPATIAL MATH
 {H3_RAW}
+{EXTRAS_RAW}
 ---
 """
 
@@ -154,6 +170,14 @@ def get_isolated_db(s3_key: str = None, s3_secret: str = None, s3_endpoint: str
                 conn.sql(stmt)
             except Exception as e:
                 print(f"⚠️ Setup statement skipped: {stmt!r}: {e}", file=sys.stderr)
+        # Opt-in extras (experimental image only — empty list on the default tag).
+        # Warn rather than raise: a missing extension should degrade this connection's
+        # capability, not fail every query the replica serves.
+        for ext in EXTRA_EXTENSIONS:
+            try:
+                conn.sql(f"LOAD {ext}")
+            except Exception as e:
+                print(f"⚠️ Extra extension {ext!r} not loaded: {e}", file=sys.stderr)
         # Bound memory to the pod so a big aggregate spills instead of OOM-killing
         # the replica (#270). Not part of SETUP_SQL: the value is deployment-derived
         # (dbconfig reads POD_MEMORY_LIMIT/DUCKDB_MEMORY_LIMIT), not model guidance.
