You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Sets up a VitePress site (matching geo-agent style) under docs/ with
GitHub Actions deployment to GitHub Pages. Covers quickstart, datasets,
private data access, deployment, and architecture.
Closes#31
Co-authored-by: Carl Boettiger <cboettig@berkeley.edu>
The `.md` files in this repo are **not documentation** — they are curated prompt content loaded by `server.py` at startup and injected directly into MCP tool descriptions and prompts at runtime. The agent (LLM) reads them as instructions, not humans.
129
+
130
+
| File | How it is used |
131
+
|---|---|
132
+
|`query-setup.md`| SQL parsed and executed in every fresh DuckDB connection before a query runs |
133
+
|`query-optimization.md`| Injected verbatim into the `query` tool description |
134
+
|`h3-guide.md`| Injected verbatim into the `query` tool description |
135
+
|`assistant-role.md`| Served as the `geospatial-analyst` MCP prompt (role + response style) |
136
+
137
+
Editing these files changes what the agent is told to do. They must be written for a stateless LLM — short, concrete, and unambiguous. See `AGENTS.md` for editing rules.
128
138
129
139
### Key Design Patterns
130
140
131
-
1.**Prompt Engineering**: Injects strict SQL rules into tool descriptions to guide LLM behavior
132
-
2.**Isolation Engine**: Each query gets a fresh DuckDB connection for security
133
-
3.**Context Injection**: Documentation is embedded into MCP resources and tool descriptions
- Dataset questions: Use the `list_datasets` tool or browse the [public STAC catalog](https://s3-west.nrp-nautilus.io/public-data/stac/catalog.json)
301
+
- Dataset questions: Use the `browse_stac_catalog` tool or browse the [public STAC catalog](https://s3-west.nrp-nautilus.io/public-data/stac/catalog.json)
The `.md` files in this repo are **not documentation** — they are curated prompt content loaded by `server.py` at startup and injected directly into MCP tool descriptions and prompts. The agent (LLM) reads them as instructions.
6
+
7
+
| File | How it is used |
8
+
|---|---|
9
+
|`query-setup.md`| SQL extracted and executed in every fresh DuckDB connection before a query runs |
10
+
|`query-optimization.md`| Injected verbatim into the `query` tool description |
11
+
|`h3-guide.md`| Injected verbatim into the `query` tool description |
12
+
|`assistant-role.md`| Served as the `geospatial-analyst` MCP prompt (role and response style) |
13
+
14
+
Editing these files changes what the agent is told to do. They must be written for a stateless LLM — short, concrete, and unambiguous. See `AGENTS.md` for editing rules.
15
+
16
+
## Two-process design
17
+
18
+
The server is designed around two distinct agent processes. See `AGENTS.md` for the full details.
19
+
20
+
**Process 1 — Real-time MCP tool (small LLM)**
21
+
22
+
Handles user requests in real time. Has no memory between requests. Its only context is what is injected at call time from the prompt files above.
23
+
24
+
**Process 2 — Asynchronous evaluation (Claude)**
25
+
26
+
Reviews logs from real user sessions, identifies slow or incorrect queries, diagnoses root causes, and updates the small LLM's prompt files. This separation prevents misdiagnoses (e.g. blaming DuckDB internals when the real cause was a query violating a rule in `query-optimization.md`) from corrupting the prompt files.
27
+
28
+
## Isolation engine
29
+
30
+
Each query runs in a fresh `duckdb.connect(":memory:")`:
31
+
32
+
```python
33
+
@contextmanager
34
+
defget_isolated_db(...):
35
+
conn = duckdb.connect(database=":memory:")
36
+
try:
37
+
# Run setup SQL from query-setup.md
38
+
# Inject per-request S3 credentials if provided
39
+
yield conn
40
+
finally:
41
+
conn.close()
42
+
```
43
+
44
+
No state, credentials, or query results persist between requests.
45
+
46
+
## Context injection
47
+
48
+
Prompt content is embedded into tool descriptions so that MCP clients which don't support `prompts/list` (e.g. VS Code) still receive the guidance. The `query` tool description includes the full text of `query-optimization.md` and `h3-guide.md`.
49
+
50
+
## STAC catalog integration
51
+
52
+
Dataset discovery is handled by `stac.py`. The agent calls `browse_stac_catalog` to list available datasets, then `get_stac_details` to resolve S3 paths and column schemas. S3 paths are never hardcoded in the server or guessed by the agent.
The agent discovers dataset paths and schemas dynamically — use the `browse_stac_catalog` and `get_stac_details` tools rather than hardcoding any S3 paths.
10
+
11
+
## Current datasets
12
+
13
+
| Dataset | Description |
14
+
|---|---|
15
+
|**GLWD**| Global Lakes and Wetlands Database |
16
+
|**Vulnerable Carbon**| Conservation International carbon vulnerability data |
17
+
|**NCP**| Nature Contributions to People biodiversity scores |
18
+
|**Countries & Regions**| Global administrative boundaries (Overture Maps) |
19
+
|**WDPA**| World Database on Protected Areas |
20
+
|**Ramsar Sites**| Wetlands of International Importance |
21
+
|**HydroBASINS**| Global watershed boundaries (levels 3–6) |
22
+
|**iNaturalist**| Species occurrence range maps |
23
+
|**Corruption Index 2024**| Transparency International data |
24
+
25
+
## H3 spatial indexing
26
+
27
+
All datasets are indexed using [Uber's H3 hexagonal grid system](https://h3geo.org).
The server supports private STAC catalogs and private S3 buckets. Credentials are supplied per-call and scoped to that request only — they are never logged, cached, or shared between clients.
Pass the same `catalog_url` and `catalog_token` to `get_stac_details` as well.
20
+
21
+
::: tip Serving a private catalog
22
+
If you use oauth2-proxy for browser access, add a parallel nginx `auth_request` bypass for the `/stac/` path that accepts a static shared token via the `Authorization` header. This allows the MCP server to fetch catalog metadata without a browser OAuth session.
23
+
:::
24
+
25
+
## Private S3 data
26
+
27
+
Pass S3 credentials directly to the `query` tool:
28
+
29
+
```json
30
+
{
31
+
"tool": "query",
32
+
"arguments": {
33
+
"sql_query": "SELECT * FROM read_parquet('s3://my-private-bucket/data/**') LIMIT 10",
34
+
"s3_key": "YOUR_ACCESS_KEY_ID",
35
+
"s3_secret": "YOUR_SECRET_ACCESS_KEY",
36
+
"s3_endpoint": "minio.example.org"
37
+
}
38
+
}
39
+
```
40
+
41
+
`s3_endpoint` defaults to `s3-west.nrp-nautilus.io` if omitted.
42
+
43
+
## Mixing private and public data
44
+
45
+
Use `s3_scope` when a query reads from both a private and the public S3 endpoint, so DuckDB routes each path to the correct endpoint:
46
+
47
+
```json
48
+
{
49
+
"tool": "query",
50
+
"arguments": {
51
+
"sql_query": "SELECT a.h8, b.value FROM read_parquet('s3://private-wyoming/...') a JOIN read_parquet('s3://public-data/...') b ON a.h8 = b.h8 AND a.h0 = b.h0",
52
+
"s3_key": "YOUR_ACCESS_KEY_ID",
53
+
"s3_secret": "YOUR_SECRET_ACCESS_KEY",
54
+
"s3_endpoint": "minio.example.org",
55
+
"s3_scope": "s3://private-wyoming"
56
+
}
57
+
}
58
+
```
59
+
60
+
## Security properties
61
+
62
+
| Concern | How it is handled |
63
+
|---|---|
64
+
| Credential bleed between clients | Each request uses a separate `duckdb.connect(":memory:")` — secrets are connection-scoped and destroyed on close |
65
+
| Credentials in server logs |`CREATE SECRET` statements are constructed internally and never written to stderr |
66
+
| Credentials in transit | All traffic is TLS-terminated at the ingress |
67
+
| Credential persistence |`stateless_http=True` — no session state survives between requests |
68
+
69
+
## Deploying private apps without a separate server
70
+
71
+
Private geo-agent apps can share the public MCP server endpoint and pass credentials per-call. This avoids maintaining a separate server deployment per app while ensuring all apps benefit from server improvements automatically.
![boettiger-lab-llm-agent[bot]](https://avatars.githubusercontent.com/u/13372268?v=4&size=40){kind=avatar}
0 commit comments