Benchmark: nimbus qwen (Qwen3.6-35B-A3B-NVFP4) full gold-standard — 19 pass / 1 partial / 2 fail

[cboettig]

Result: 19 pass / 1 partial / 2 fail (22 answer questions)

First full run of the standing gold baseline (#40) against nimbus qwen = nvidia/Qwen3.6-35B-A3B-NVFP4 (DSE, vllm-nimbus.carlboettiger.info). Sibling to the OpenRouter open-model benchmark (#54).

Environment

• Model/route: nimbus qwen, called direct (--proxy-endpoint https://vllm-nimbus.carlboettiger.info/v1 --model qwen) via headless/run.js.
• Data: 10 questions off the MinIO mirror (dev MCP, minio-default) during the tail of the Ceph outage; the other 12 off restored Ceph (committed app configs + prod MCP) so values match the gold snapshot. See data-workflows#360 for the 3 mirror-gap datasets that forced the split.
• Throughput: ~70–105 tok/s decode; 0 tool-call format failures across all 22 (vLLM qwen3_xml_patched + geo-agent#288 held — was ~1/4 malformed before those landed).
• Grading: per-question against headless/baseline/golden.json accept bands; gold is operator-verified via own DuckDB queries, not consensus (Treat the geo-agent harness (not just guidance text) as the optimization target for open-model training #42 methodology).

Scorecard

#	id	gold	nimbus qwen	verdict	s
1	bio-ramsar-countries	UK 176, MX 144, IN 94, CN 82	exact + flagged count-distinct trap	✅	26
2	bio-ecoregion-vuln-carbon	W. Siberian taiga 13.85 Gt #1	exact	✅	127
3	bosl-fishing-displaced	~1.96M hrs, ~75% longlines	2.02M hrs, 73.8% longlines	✅ (+3%)	335
4	bosl-sargasso-seamounts	141	141	✅	23
5	tplca-cd-ballot-funding-2010	separate statewide vs local	separated, LA/coastal top	✅	151
6	tplca-cd-failed-measures	separate statewide; SD CDs top local	flagged statewide+SD, ranking left statewide-polluted	⚠️	111
7	tpl-nj-municipalities	~261 munis / ~444 / ~$1.95B	261 / 445 / $1.96B	✅	68
8	wet-india-vuln-carbon	~1.1 Pg C	1.095 Pg C, top classes match	✅	162
9	wet-ramsar-criterion9	64 sites; non-avian 1% def	3,590 (rows); wrong def	❌	33
10	wet-top10-hydrobasins-composite	basin 6030007000 #1	6030007000 #1, sound method	✅	248
11	ca-gap12-acres	26.47M	26.43M + flagged 52M trap	✅	18
12	ca-ecoregion-most-conserved	Mojave 7.37M	Mojave 7.37M	✅	20
13	glob-pct-land-protected	~16.5%	16.5%, dedup h8	✅	206
14	glob-people-in-pas	~357M	~357M	✅	662
15	glob-least-represented-ecoregions	~0% arid set	Chhota-Nagpur/Qaidam/Horn of Africa	✅	366
16	glob-top5-pa-mammal-richness	Albertine Rift/Rwenzori	Rwenzori/Uganda (didn't apply ≥3-cell threshold)	✅	609
17	tplca-cd16-funders	~$36.1M; Prop40/12/LWCF	$36.1M; Prop40 11.75/Prop12 7.61/LWCF 5.64	✅	53
18	tplca-assembly-acreage	AD34 563,641; AD36 451,429	exact	✅	66
19	tplca-lwcf-sd2	~$285.5M	$285.5M (site count 187 vs 48, attribution note)	✅	89
20	tpl-almanac-landcover	herb/needleleaf/decid/ag; herb→CO shrub→CA	classes match; herb→CO, shrub→CA	✅	481
21	tpl-tx-cd-federal-funding	CD-34/CD-14 top ~$29.5M	CD-34 top but $32.6B (~1000× overcount)	❌	114
22	tpl-boulder-funders	Boulder sales-tax ~$321M +NRCS/USFS/GOCO	$321.2M + NRCS 6.3/USFS 4.9/GOCO 3.1	✅	42

The 2 failures — both classic geo-agent traps

• Add gemma4-nimbus provider entry for audio-capable Gemma 4 endpoint #9 wet-ramsar-criterion9: counted 3,590 rows (COUNT(*) WHERE Criterion9) instead of 64 distinct sites (COUNT(DISTINCT ramsarid)) — the same site-has-many-rows shape it correctly handled in Add request_id to correlate request/response log pairs #1 — and fabricated the criterion definition ("scientific/educational value" vs the real non-avian 1%-population rule). Data verified correct. → data-workflows#356.
• Add direct Anthropic provider (OpenAI-compatible endpoint) #21 tpl-tx-cd-federal-funding: joined per-tpl_id funding to the sites hex table then SUM(amount), replicating each site's amount across all its h10 cells → $32.6B vs ~$29.5M (~1000×). CD-34 top rank survived; magnitude nonsense. Same dataset as Add geo-agent-training as a local skill #22 (Boulder, PASS) — the difference is Add geo-agent-training as a local skill #22 filters sites by county name (no hex fan-out) while Add direct Anthropic provider (OpenAI-compatible endpoint) #21 attributes via hex. The dedup-before-sum-over-hex trap.

1 partial

• Unified log analysis script with per-app filtering #6 tplca-cd-failed-measures: separated statewide from local conceptually and flagged San Diego, but its primary ranking left statewide measures in (no clean local-only CD ranking).

Takeaway

nimbus qwen is strong on direct filter/aggregate (exact on 12+ questions incl. all the funding/acreage lookups) with two systematic weak spots: (1) dedup-before-sum over hex partitions, (2) coded-column semantics (fabricates when STAC has no description). Both are addressable via guidance / STAC metadata, not model changes — see the geo-agent guidance issue (linked below) and data-workflows#356 / #360.

Transcripts (all 22) archived in the session scratchpad. Refs: #40 (baseline), #54 (open-model benchmark), #42 (harness-as-target), data-workflows#360, data-workflows#356.

[cboettig]

Guidance follow-up for the two trap failures (#9 coded-column semantics + count-distinct, #21 dedup-before-sum over hex): boettiger-lab/geo-agent#293.

[cboettig]

Second full run (uniform, all-ceph via prod MCP): 17 pass / 1 partial / 4 fail

Re-ran all 22 on stable ceph (prod MCP + committed configs), origin gold_nimbus_v2. Same model, temp 0. Scored 17/1/4 vs the first run's 19/1/2 — the delta is entirely run-to-run variance on hard hex-aggregation / attribution questions:

question	run 1	run 2 (uniform)
tplca-q2 (ballot attribution)	✅ $491M LA	❌ $5.6B Bay Area (double-count)
tplca-q3 (failed measures)	⚠️ partial	✅ CD49/50/48 exact
wet-q1 (India carbon)	✅ 1.095 Pg	❌ 6.5 Gt (~6× h8/h9 overcount)
wet-q2 (Criterion 9)	❌ 3,590 + wrong def	⚠️ 34 + correct def
wet-q3 (hydrobasin)	✅ basin 6030007000	❌ timeout (set THREADS=100)
tpl-q2 (TX funding)	❌ 1000× overcount	❌ asked-instead (but stated the dedup fix)

Easy/medium questions were exact both times; only the hard multi-step ones move.

Root cause of the variance (temp is 0)

geo-agent/app/agent.js pins temperature: 0 and omits seed/top_p, so decoding is greedy — but greedy ≠ deterministic. The logits vary run-to-run from batch-dependent FP nondeterminism in the shared vLLM server (non-associative reductions under continuous batching), amplified by this model being MoE (A3B) + NVFP4 (near-ties are closer, flip more easily). A single flipped token early in a 15–25-call trajectory changes the SQL the model writes → the whole downstream path diverges. seed wouldn't help (nondeterminism is upstream of sampling). Hard questions have many decision-point forks; easy ones have ~one path → stable.

Implication + mitigation

Single-trial scoring is noisy on ~6/22; the baseline should run N=3–5 trials/question, scored by pass-rate. And since the model produces the right answer sometimes, the durable fix is tool-use self-verification, not determinism — prototype tracked in #70 (grain/dedup advisory + subset-vs-whole sanity bound + critic pass, A/B'd in headless). Trap root causes: geo-agent#293.