Models silently pick one of two endemic-richness layers instead of asking — 17.7% vs 25.1% on consecutive trials (clarify-ca-endemic-gap1, 0/8 cells asked)

[cboettig]

clarify-ca-endemic-gap1 asks "What percent of GAP 1 land is in the top 20% (>=80th percentile) of endemic-species richness?". "Endemic richness" maps to more than one layer in this catalog, so the question's gold is explicit:

recognizes 'endemic richness' is unspecified (at minimum the metric-layer choice) and ASKS which layer/threshold before computing. Silently answering with ANY single number — even the correct ~20.8% AllTaxaEnd value — is a FAIL; the model must ask, not guess.

Every cell answers instead of asking, and the layer it silently picks changes the answer by 7.4 points.

The gap in numbers

Operator-judged across four runs (all APP_BRANCH=fix/streams-order-unavailable, prod duckdb-mcp main@cd1b44bc, TRIALS=2):

run	model	t1	t2	layer chosen
2026-08-08-regression-final	deepseek/deepseek-v4-flash-0731	17.7%	25.1%	ACE AllTaxaEnd / Kling rarity-weighted p80-hex
2026-08-08-regression-final	z-ai/glm-5.2	17.7%	17.69%	ACE AllTaxaEnd
2026-08-08-regression-r2-masking	deepseek/deepseek-v4-flash-0731	17.7%	25.1%	both
2026-08-07-regression	deepseek/deepseek-v4-flash-0731	25.1%	25.1%	Kling p80-hex

0 of 8 cells asked. All lead with a bolded percentage. The same model on the same question in the same run returned 17.7% and 25.1% on consecutive trials, because trial 1 read "endemic richness" as ACE's AllTaxaEnd count and trial 2 as Kling et al.'s rarity-weighted endemic plant richness. Both answers document their choice clearly in the method — neither flags that a choice was made.

That is exactly the failure this cell exists to catch, and it is the one an end user cannot detect: the number is confident, the method looks rigorous, and nothing says "I picked one of two defensible layers for you."

Pre-existing, not a regression

Present identically in runs before and after the stream-order prompt work (#111 / #112), so it is not caused by those changes and does not block that merge. Filing separately rather than folding it into #112.

Why the current prompt does not catch it

The "Ask, don't guess" section covers unresolvable things — unknown codes, missing datasets, failed lookups:

Never invent class codes, type names, categories, or column meanings you haven't confirmed…
Only answer from datasets in the catalog. If a question needs data that isn't there, say so plainly…

None of that fires here. Nothing is missing or unknown; the problem is that two catalog datasets both legitimately answer the user's phrase, and the rule has no case for "the user's term is satisfied by more than one layer". A model following the current prompt to the letter picks one and documents it — which is what all eight cells did.

Proposed change (this repo, system-prompt.md)

Add to "Ask, don't guess":

• If the user's term matches more than one dataset or threshold in the catalog, ask which — do not pick one and document the pick. Naming the choice in the method section is not a substitute: the user sees a single confident number and no signal that a different, equally defensible layer would give a different one. This applies whenever the feature is named loosely ("endemic richness", "wetlands", "streams", "floodplain") and the catalog carries more than one candidate.

Done when

clarify-ca-endemic-gap1 passes on both trials for z-ai/glm-5.2 and deepseek/deepseek-v4-flash-0731 — the model asks which endemic-richness layer is meant instead of returning a number — verified by a regression tier run, with no regression in the other cells.

[cboettig]

Evidence from the 2026-08-08 gate that sharpens this: the models already do the right thing on a different question, so the capability exists and the gap is inconsistency, not absence.

car-23-channelized-connectivity, deepseek/deepseek-v4-flash-0731, trial 1 (run 2026-08-08-regression-natl2) opened with:

"Channelized" is a connectivity class that appears in two different datasets in the catalog, and the answer will differ depending on which you mean: 1. Present-day Connectivity — Connectivity Categories (Cameron et al. 2022) … 2. Climate Migration Routes …

That is exactly the behaviour this issue asks for — the model noticed one term mapped to two catalog datasets, named both, and asked. On clarify-ca-endemic-gap1 the same model, in the same run, silently picked a layer instead (17.7% via ACE AllTaxaEnd on one trial, 25.1% via Kling rarity-weighted on another).

So the proposed rule is not asking for a new capability, just for consistency — which makes it considerably more likely to work than if no cell had ever demonstrated it. Worth using the car-23 phrasing as the model of what a compliant answer looks like when wording the rule.

Still unfixed and still pre-existing; not touched by #112 or #114.