Skip to content

Commit adec450

Browse files
cboettigcboettig
authored
docs+ci: dplyr/dbplyr + ibis examples on the source.coop mirror (#300)
* docs: dplyr/dbplyr + ibis examples reading the source.coop mirror Add a dataframe-idiom access route for R (dplyr/dbplyr) and Python (ibis) that points a local DuckDB straight at the public source.coop mirror (AWS us-west-2, anonymous) — no MCP server needed to read the data. Verbs push down into DuckDB; only collect()/execute() pulls rows. - programmatic-access.md: new "R — dplyr/dbplyr" and "Python — ibis" sections; document the public-<name> -> cboettig/<name> path mapping. - examples/query_dbplyr.R, examples/query_ibis.py: self-checking scripts. - Fix mcptools links (tidyverse -> posit-dev) and clarify the client is stdio-only (posit-dev/mcptools#88 deferred direct HTTP transport). * ci: standalone workflow running the dbplyr + ibis examples Self-checking example scripts against the public source.coop mirror (no secrets). Path-filtered on PRs plus a weekly cron to catch the mirror dataset/schema rolling forward. Separate from the pytest workflow since R isn't part of the codebase.
1 parent ac376af commit adec450

6 files changed

Lines changed: 275 additions & 4 deletions

File tree

‎.github/workflows/examples.yml‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
name: Examples
2+
3+
# The dataframe-idiom examples (R dbplyr, Python ibis) read the public
4+
# source.coop mirror directly with a local DuckDB — no MCP server, no secrets.
5+
# Each script self-checks (asserts on the result) so running it IS the test.
6+
# The weekly schedule is the real value: it catches the mirror path / dataset
7+
# rolling forward or a schema change before users hit it.
8+
on:
9+
push:
10+
branches: [ main, master ]
11+
paths: [ 'examples/query_dbplyr.R', 'examples/query_ibis.py', '.github/workflows/examples.yml' ]
12+
pull_request:
13+
branches: [ main, master ]
14+
paths: [ 'examples/query_dbplyr.R', 'examples/query_ibis.py', '.github/workflows/examples.yml' ]
15+
schedule:
16+
- cron: '0 6 * * 1' # Mondays 06:00 UTC
17+
workflow_dispatch:
18+
19+
jobs:
20+
dbplyr:
21+
name: R — dplyr/dbplyr
22+
runs-on: ubuntu-latest
23+
steps:
24+
- uses: actions/checkout@93cb6efe18208431cddfb8368fd83d5badbf9bfd # v5.0.1
25+
- uses: r-lib/actions/setup-r@d3c5be51b12e724e68f33216ca3c148b66d5f0b6 # v2
26+
with:
27+
use-public-rspm: true # binary packages — no source compiles
28+
- name: Install R packages
29+
run: Rscript -e 'install.packages(c("DBI", "duckdb", "dplyr", "dbplyr"))'
30+
- name: Run example
31+
run: Rscript examples/query_dbplyr.R
32+
33+
ibis:
34+
name: Python — ibis
35+
runs-on: ubuntu-latest
36+
steps:
37+
- uses: actions/checkout@93cb6efe18208431cddfb8368fd83d5badbf9bfd # v5.0.1
38+
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
39+
with:
40+
python-version: '3.11'
41+
- name: Install Python packages
42+
run: |
43+
python -m pip install --upgrade pip
44+
pip install 'ibis-framework[duckdb]'
45+
- name: Run example
46+
run: python examples/query_ibis.py

‎docs/guide/programmatic-access.md‎

Lines changed: 94 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,7 @@ asyncio.run(main())
4848

4949
### R
5050

51-
There is no R MCP client library yet. The server runs in stateless mode, so you can hit the JSON-RPC endpoint directly with `httr2`. Responses arrive as server-sent events (SSE).
51+
No R MCP client speaks HTTP directly ([`mcptools`](https://posit-dev.github.io/mcptools/) is stdio-only — see the [ellmer + mcptools](#r-ellmer-mcptools) section below). The server runs in stateless mode, so you can hit the JSON-RPC endpoint directly with `httr2`. Responses arrive as server-sent events (SSE).
5252

5353
```r
5454
library(httr2)
@@ -104,6 +104,98 @@ for (block in resp$result$content) {
104104
}
105105
```
106106

107+
### R — dplyr / dbplyr
108+
109+
If you'd rather write queries in `dplyr` than SQL, you don't need the MCP server at all: point a **local** DuckDB at the public [source.coop](https://source.coop) mirror of the same data and let `dbplyr` compile your `dplyr` verbs to DuckDB SQL. The mirror lives on AWS `us-west-2` (anonymous reads, reader doesn't pay egress), so it works from anywhere. Use the STAC catalog to *discover* paths and columns; read the Parquet directly here.
110+
111+
```r
112+
library(DBI)
113+
library(duckdb)
114+
library(dplyr)
115+
library(dbplyr)
116+
117+
con <- dbConnect(duckdb::duckdb())
118+
dbExecute(con, "INSTALL httpfs; LOAD httpfs;")
119+
120+
# Anonymous reads from the source.coop mirror (AWS us-west-2). The bucket name
121+
# has dots, so URL_STYLE 'path' is required for the TLS certificate to match.
122+
dbExecute(con, "
123+
CREATE SECRET source_coop (
124+
TYPE S3, KEY_ID '', SECRET '',
125+
ENDPOINT 's3.us-west-2.amazonaws.com', REGION 'us-west-2',
126+
URL_STYLE 'path', USE_SSL 'true',
127+
SCOPE 's3://us-west-2.opendata.source.coop'
128+
)")
129+
130+
# Register the Parquet dataset as a view, then treat it as a dplyr table.
131+
path <- "s3://us-west-2.opendata.source.coop/cboettig/overturemaps/2026-02-18.0/countries.parquet"
132+
dbExecute(con, sprintf("CREATE VIEW countries AS SELECT * FROM read_parquet('%s')", path))
133+
134+
countries <- tbl(con, "countries")
135+
136+
q <- countries |>
137+
filter(subtype == "country", is_land) |>
138+
select(country, name_en, subtype) |>
139+
arrange(name_en) |>
140+
head(10)
141+
142+
q |> show_query() # inspect the DuckDB SQL dbplyr generated
143+
q |> collect() # pull the result into a tibble
144+
145+
dbDisconnect(con, shutdown = TRUE)
146+
```
147+
148+
`dbplyr` pushes `filter`/`select`/`arrange`/`summarise`/joins down into DuckDB, so column and row pruning happen in the engine — only the final `collect()` pulls data into R.
149+
150+
::: tip
151+
The STAC catalog publishes NRP paths (`s3://public-<name>/…`); the source.coop mirror maps them to `s3://us-west-2.opendata.source.coop/cboettig/<name>/…`. Discover paths and columns via `get_stac_details` (see the [`httr2` example above](#r)) or the [web catalog](https://beta.source.coop), then translate the prefix. The usual DuckDB read rule still applies: always `read_parquet('s3://…')`, never a bare table name.
152+
:::
153+
154+
### Python — ibis
155+
156+
The Python parallel to `dbplyr`: [ibis](https://ibis-project.org) drives the same local DuckDB against the same source.coop mirror, and its deferred expressions compile to DuckDB SQL — nothing runs until `.execute()`.
157+
158+
```bash
159+
pip install 'ibis-framework[duckdb]'
160+
```
161+
162+
```python
163+
import ibis
164+
165+
con = ibis.duckdb.connect()
166+
con.raw_sql("INSTALL httpfs; LOAD httpfs;")
167+
168+
# Anonymous reads from the source.coop mirror (AWS us-west-2). The bucket name
169+
# has dots, so URL_STYLE 'path' is required for the TLS certificate to match.
170+
con.raw_sql("""
171+
CREATE SECRET source_coop (
172+
TYPE S3, KEY_ID '', SECRET '',
173+
ENDPOINT 's3.us-west-2.amazonaws.com', REGION 'us-west-2',
174+
URL_STYLE 'path', USE_SSL 'true',
175+
SCOPE 's3://us-west-2.opendata.source.coop'
176+
)""")
177+
178+
path = "s3://us-west-2.opendata.source.coop/cboettig/overturemaps/2026-02-18.0/countries.parquet"
179+
# EXCLUDE the GEOMETRY column: ibis's type mapper can't represent DuckDB's
180+
# GEOMETRY type during schema inference. (The attribute columns are all we need.)
181+
con.raw_sql(f"CREATE VIEW countries AS SELECT * EXCLUDE (geometry) FROM read_parquet('{path}')")
182+
183+
countries = con.table("countries")
184+
185+
expr = (
186+
countries
187+
.filter((countries.subtype == "country") & countries.is_land)
188+
.select("country", "name_en", "subtype")
189+
.order_by("name_en")
190+
.limit(10)
191+
)
192+
193+
print(ibis.to_sql(expr)) # inspect the DuckDB SQL ibis generated
194+
print(expr.execute()) # run it, returns a pandas DataFrame
195+
```
196+
197+
Like `dbplyr`, ibis pushes `filter`/`select`/`order_by`/aggregations/joins down into DuckDB, so only the final `.execute()` pulls rows into Python. The same path mapping and `read_parquet` rules from the R tip above apply.
198+
107199
## LLM tool use
108200

109201
Let the model discover datasets, write SQL, and interpret results autonomously. The MCP tools (`browse_stac_catalog`, `get_stac_details`, `query`) are registered as callable tools so the model decides when and how to use them.
@@ -151,7 +243,7 @@ asyncio.run(main())
151243

152244
### R — ellmer + mcptools
153245

154-
[`mcptools`](https://github.com/tidyverse/mcptools) is an MCP client for R that plugs MCP tools into `ellmer` chats. It speaks stdio, so we bridge to the remote HTTP server with [`mcp-remote`](https://www.npmjs.com/package/mcp-remote) (requires Node.js on PATH).
246+
[`mcptools`](https://posit-dev.github.io/mcptools/) is an MCP client for R that plugs MCP tools into `ellmer` chats. Its client (`mcp_tools()`) speaks **stdio only** — direct HTTP-transport support was proposed in [posit-dev/mcptools#88](https://github.com/posit-dev/mcptools/issues/88) but deliberately deferred (the maintainers still [recommend `mcp-remote`](https://posit-dev.github.io/mcptools/reference/client.html#connecting-to-remote-http-servers)). So we bridge to the remote HTTP server with [`mcp-remote`](https://www.npmjs.com/package/mcp-remote) (requires Node.js on PATH).
155247

156248
```r
157249
library(mcptools)

‎examples/README.md‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,14 @@
11
# Examples
22

3-
Four small scripts showing two ways to talk to the duckdb-geo MCP server:
3+
Small scripts showing ways to talk to the duckdb-geo MCP server (plus one that
4+
skips it and reads the same public data directly):
45

56
| File | What it does |
67
|---|---|
78
| [query.py](query.py) | Direct MCP `query` tool call from Python (no LLM) |
89
| [query.R](query.R) | Direct MCP `query` tool call from R via JSON-RPC over HTTP |
10+
| [query_dbplyr.R](query_dbplyr.R) | Query the source.coop mirror with dplyr/dbplyr via a local DuckDB (no MCP server) |
11+
| [query_ibis.py](query_ibis.py) | Query the source.coop mirror with ibis via a local DuckDB (no MCP server) |
912
| [agent_langchain.py](agent_langchain.py) | LangGraph ReAct agent that calls MCP tools via tool use |
1013
| [agent_ellmer.R](agent_ellmer.R) | ellmer chat that calls MCP tools via tool use |
1114

‎examples/query.R‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,9 @@
11
# Minimal R example: call the duckdb-geo MCP `query` tool directly.
22
#
33
# No LLM involved -- this just speaks MCP over streamable HTTP and runs SQL.
4-
# There is no R MCP client library, so we hit the JSON-RPC endpoint via httr2.
4+
# No R MCP client speaks HTTP directly (mcptools is stdio-only), so we hit the
5+
# JSON-RPC endpoint via httr2. If you only want to *read* the data with dplyr and
6+
# don't need the MCP tools, see query_dbplyr.R instead.
57
# The server runs in stateless mode, so no session handshake is required.
68
#
79
# Install:

‎examples/query_dbplyr.R‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
# R example: query the public data with dplyr / dbplyr (no MCP server, no LLM).
2+
#
3+
# R users who prefer dplyr over raw SQL don't need the MCP server to *read* the
4+
# data: point a local DuckDB at the public source.coop mirror (AWS us-west-2,
5+
# anonymous) and dbplyr compiles your dplyr verbs to DuckDB SQL. Use the MCP
6+
# server's STAC tools (see query.R -> get_stac_details) to discover paths and
7+
# columns; read the data here. filter/select/arrange/summarise/joins push down
8+
# into DuckDB -- only the final collect() pulls rows into R.
9+
#
10+
# Install:
11+
# install.packages(c("DBI", "duckdb", "dplyr", "dbplyr"))
12+
#
13+
# Run (self-checking -- exits non-zero if the query route breaks):
14+
# Rscript query_dbplyr.R
15+
16+
library(DBI)
17+
library(duckdb)
18+
library(dplyr)
19+
library(dbplyr)
20+
21+
con <- dbConnect(duckdb::duckdb())
22+
dbExecute(con, "INSTALL httpfs; LOAD httpfs;")
23+
24+
# Anonymous reads from the source.coop mirror (AWS us-west-2). The bucket name
25+
# has dots, so URL_STYLE 'path' is required for the TLS certificate to match.
26+
dbExecute(con, "
27+
CREATE SECRET source_coop (
28+
TYPE S3, KEY_ID '', SECRET '',
29+
ENDPOINT 's3.us-west-2.amazonaws.com', REGION 'us-west-2',
30+
URL_STYLE 'path', USE_SSL 'true',
31+
SCOPE 's3://us-west-2.opendata.source.coop'
32+
)")
33+
34+
# Register the Parquet dataset as a view, then treat it as a dplyr table.
35+
path <- "s3://us-west-2.opendata.source.coop/cboettig/overturemaps/2026-02-18.0/countries.parquet"
36+
dbExecute(con, sprintf(
37+
"CREATE VIEW countries AS SELECT * FROM read_parquet('%s')", path))
38+
39+
countries <- tbl(con, "countries")
40+
41+
q <- countries |>
42+
filter(subtype == "country", is_land) |>
43+
select(country, name_en, subtype) |>
44+
arrange(name_en) |>
45+
head(10)
46+
47+
cat("--- DuckDB SQL generated by dbplyr ---\n")
48+
cat(dbplyr::sql_render(q), "\n\n")
49+
50+
result <- collect(q)
51+
cat("--- result ---\n")
52+
print(as.data.frame(result))
53+
54+
dbDisconnect(con, shutdown = TRUE)
55+
56+
# Smoke test: the pushdown query returns exactly the 10 rows we asked for,
57+
# with the expected columns. Fails loudly (non-zero exit) if the route breaks.
58+
stopifnot(
59+
nrow(result) == 10L,
60+
identical(names(result), c("country", "name_en", "subtype")),
61+
all(result$subtype == "country")
62+
)
63+
cat("\nOK: dplyr/dbplyr -> DuckDB -> S3 Parquet route works\n")

‎examples/query_ibis.py‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
"""Python example: query the public data with ibis (no MCP server, no LLM).
2+
3+
The Python parallel to query_dbplyr.R. Users who prefer a dataframe API over raw
4+
SQL don't need the MCP server to *read* the data: point a local DuckDB at the
5+
public source.coop mirror (AWS us-west-2, anonymous) and let ibis compile its
6+
deferred expressions to DuckDB SQL. Use the MCP server's STAC tools (see
7+
query.py -> get_stac_details) to discover paths and columns; read the data here.
8+
filter/select/order_by/aggregations/joins push down into DuckDB -- only the
9+
final .execute() pulls rows into Python.
10+
11+
Install:
12+
pip install 'ibis-framework[duckdb]'
13+
14+
Run (self-checking -- exits non-zero if the query route breaks):
15+
python query_ibis.py
16+
"""
17+
import ibis
18+
19+
con = ibis.duckdb.connect()
20+
con.raw_sql("INSTALL httpfs; LOAD httpfs;")
21+
22+
# Anonymous reads from the source.coop mirror (AWS us-west-2). The bucket name
23+
# has dots, so URL_STYLE 'path' is required for the TLS certificate to match.
24+
con.raw_sql(
25+
"""
26+
CREATE SECRET source_coop (
27+
TYPE S3, KEY_ID '', SECRET '',
28+
ENDPOINT 's3.us-west-2.amazonaws.com', REGION 'us-west-2',
29+
URL_STYLE 'path', USE_SSL 'true',
30+
SCOPE 's3://us-west-2.opendata.source.coop'
31+
)"""
32+
)
33+
34+
path = (
35+
"s3://us-west-2.opendata.source.coop/"
36+
"cboettig/overturemaps/2026-02-18.0/countries.parquet"
37+
)
38+
# EXCLUDE the GEOMETRY column: ibis's type mapper can't represent DuckDB's
39+
# GEOMETRY type during schema inference. (The attribute columns are all we need.)
40+
con.raw_sql(
41+
f"CREATE VIEW countries AS SELECT * EXCLUDE (geometry) FROM read_parquet('{path}')"
42+
)
43+
44+
countries = con.table("countries")
45+
46+
expr = (
47+
countries.filter((countries.subtype == "country") & countries.is_land)
48+
.select("country", "name_en", "subtype")
49+
.order_by("name_en")
50+
.limit(10)
51+
)
52+
53+
print("--- DuckDB SQL generated by ibis ---")
54+
print(ibis.to_sql(expr))
55+
56+
df = expr.execute()
57+
print("\n--- result ---")
58+
print(df)
59+
60+
# Smoke test: the pushdown query returns exactly the 10 rows we asked for, with
61+
# the expected columns. Fails loudly (non-zero exit) if the route breaks.
62+
assert len(df) == 10, f"expected 10 rows, got {len(df)}"
63+
assert list(df.columns) == ["country", "name_en", "subtype"], df.columns
64+
assert (df["subtype"] == "country").all()
65+
print("\nOK: ibis -> DuckDB -> S3 Parquet route works")

0 commit comments

Comments
 (0)