h3-guide.md, query-optimization.md and query-setup.md carry worked SQL examples that are injected into the query tool description. Nothing checks that they run. #364 shipped a coarse-overlay example that could not execute — it read nland from …-hex-weights-res8 after data-workflows#522 moved that column — and it went out in a tagged release.
Why the model gate cannot catch this
The natural assumption is that a question exercising the same path would have caught it. It would not. car-19-ace-biorank5 does overlay a res-8 feature against the conserved-areas weights, and in the 2026-08-09 gate — run against dev at 19b3c709, the build with the broken example — both models answered it correctly:
glm-5.2: {'land_area_km2': 3, 'hex-weights-res8': 1}
deepseek: {'hex-weights-res8': 1, 'land_area_km2': 3}
They took the asset name from the guide and the column names from the per-dataset STAC description, which was already correct. A model that can route around a broken example will, so the gate goes green whether the example works or not. That is why geo-agent-benchmark#21 was closed: the coverage was already there, and adding more questions cannot fix this.
It also explains the low blast radius of #364 — degraded, not blocked. But "models mostly recover" is not a property to rely on, and the weakest models are the least likely to.
Proposal
A CI job that extracts every ```sql block from the guidance files and executes it against the catalog.
The examples use placeholders (<cwhr13 hex-fractions>, <ecoregion hex-res8>, <feature filter>), so this needs a small committed fixture mapping placeholder → real path/predicate. Two rules keep the fixture honest:
- a block whose placeholders are all mapped must execute, or CI fails
- an unmapped placeholder is a failure, not a skip — otherwise the check quietly passes everything as examples drift (this is the same trap geo-agent-benchmark#9 flags for its own key mapping)
EXPLAIN is not enough on its own — it does bind columns, which is what #364 needed, but running the statement also catches a path that no longer exists. Given these are small scoped queries, running them is affordable; if cost becomes an issue, EXPLAIN catches the specific #364 shape.
Why this is the right layer
The guidance files are runtime artifacts, not documentation — server.py loads them at startup and injects them. An example that does not execute is a defect in a shipped artifact, and it should be caught the way any broken code is: by running it in CI, before merge. The model gate measures whether guidance steers; this measures whether it is valid. Both are needed and neither substitutes.
Related: #367 (sequencing for this guidance area), #245 (make guidance validation a pre-merge gate — this is a cheap, deterministic first slice of that), geo-agent-benchmark#21 (closed, with the measurement above).
h3-guide.md,query-optimization.mdandquery-setup.mdcarry worked SQL examples that are injected into thequerytool description. Nothing checks that they run. #364 shipped a coarse-overlay example that could not execute — it readnlandfrom…-hex-weights-res8after data-workflows#522 moved that column — and it went out in a tagged release.Why the model gate cannot catch this
The natural assumption is that a question exercising the same path would have caught it. It would not.
car-19-ace-biorank5does overlay a res-8 feature against the conserved-areas weights, and in the 2026-08-09 gate — run against dev at19b3c709, the build with the broken example — both models answered it correctly:They took the asset name from the guide and the column names from the per-dataset STAC description, which was already correct. A model that can route around a broken example will, so the gate goes green whether the example works or not. That is why geo-agent-benchmark#21 was closed: the coverage was already there, and adding more questions cannot fix this.
It also explains the low blast radius of #364 — degraded, not blocked. But "models mostly recover" is not a property to rely on, and the weakest models are the least likely to.
Proposal
A CI job that extracts every ```sql block from the guidance files and executes it against the catalog.
The examples use placeholders (
<cwhr13 hex-fractions>,<ecoregion hex-res8>,<feature filter>), so this needs a small committed fixture mapping placeholder → real path/predicate. Two rules keep the fixture honest:EXPLAINis not enough on its own — it does bind columns, which is what #364 needed, but running the statement also catches a path that no longer exists. Given these are small scoped queries, running them is affordable; if cost becomes an issue,EXPLAINcatches the specific #364 shape.Why this is the right layer
The guidance files are runtime artifacts, not documentation —
server.pyloads them at startup and injects them. An example that does not execute is a defect in a shipped artifact, and it should be caught the way any broken code is: by running it in CI, before merge. The model gate measures whether guidance steers; this measures whether it is valid. Both are needed and neither substitutes.Related: #367 (sequencing for this guidance area), #245 (make guidance validation a pre-merge gate — this is a cheap, deterministic first slice of that), geo-agent-benchmark#21 (closed, with the measurement above).