feat(catalog): compact catalog index for large apps to shrink the cold prompt (#294)

[cboettig]

What

Part 1 of #294. DatasetCatalog.generatePromptCatalog() switches to a compact index above catalog_index_threshold datasets (default 8): one line per dataset — `id` — **title** — one-line summary (+ map-layer ids for map datasets). It drops the front-loaded full description, read_parquet paths, provider, and about-url. Small apps keep the full front-load unchanged.

Why it's safe (zero added round-trips)

Everything the compact form omits — paths, columns, full description — arrives via get_schema(dataset_id), a call the model already makes before every query (verified in the issue's gold transcripts: models call get_schema+query, never browse_stac_catalog). So the front-loaded copy is pure redundancy; removing it also reinforces the get-schema-first discipline (AGENTS.md: never guess S3 paths).

Measured

Real geo-agent-template catalog (27 datasets loaded): catalog block ~9.8k → ~2.0k tokens (79.5% smaller).

Headless A/B on nimbus qwen (CD-16 funding question, run.js with GEO_AGENT_DIR pointed at this branch vs main):

metric	full (main)	compact	delta
turn-1 cold prompt_tokens	29,467	20,335	−31%
total prefill tokens (turn)	525,735	139,737	3.8×
LLM time	77.4s	39.8s	1.9×
tool flow	browse→get_stac_details×2→get_schema→query×7	list_datasets→get_schema→query	✓

The compact run navigated the catalog correctly (get_schema supplied the path that's no longer front-loaded), confirming the core design assumption. This matches the issue's controlled benchmark (cirrus 2.1× faster, identical answer; nimbus 6/6 correct).

Honest note on the A/B: the two runs diverged on the final step — full answered directly; compact noticed CD-16 exists in 6 states and asked which one (a reasonable disambiguation, cf. #274). That's final-step model nondeterminism on an ambiguous question (MoE isn't bit-reproducible at temp 0), not a catalog-caused regression — the compact run retrieved the same underlying data. Accuracy is covered by the issue's 6/6 controlled benchmark; this A/B contributes the token/latency evidence.

Config

catalog_index_threshold (default 8; set very high to always front-load). Documented in docs/guide/configuration.md.

Scope

This is the compact-index half. Part 2 of #294 — gating the 16 map-tool schemas (~5–6k tok) to relevant turns — is a separate, thornier change and not included here. The query-description / MCP-side half is mcp-data-server#293.

Tests

7 new cases in test/dataset-catalog.test.js (threshold behavior, compact shape, dropped fields, override, parent containers, size-reduction assertion, _oneLine). Existing single-dataset catalog tests stay in full mode, unchanged. Full suite: 485 passing.

Refs #294

[github-actions]

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	56.47%	4916 / 8705
🔵	Statements	56.44%	4916 / 8709
🔵	Functions	63.34%	178 / 281
🔵	Branches	83.69%	1073 / 1282

File Coverage

File	Stmts	Branches	Functions	Lines	Uncovered Lines
Changed Files
app/dataset-catalog.js	91.15%	81.18%	100%	91.15%	87-89, 115-117, 142-151, 171-173, 183-184, 433-438, 467-508, 811-817, 830-835
app/main.js	0%	0%	0%	0%	1-494

Generated in workflow #158 for commit 2503ec1 by the Vitest Coverage Report Action