README.md

Bonsai 2 27B quality: what the runtime controls, measured

Summary

The complaint about Bonsai 2 is code and agentic work, and most of that gap is runtime, not compression. Details and every measurement are in the sections below.

• KV precision. Per token, q4_0 KV flips the top token on 1 in 48 positions at depth, q8_0 on 1 in 160 (KL 0.00218 vs 0.00017 against f16 KV; chart). On the tasks that matter at depth the two are equal: at 245k exact code copies 20/20 and needles 24/24 with q4_0 (q8_0: 20/20, 23/24), HumanEval 158 for both, perplexity +0.21 % (docs/research/depth-20261010/ D29). Because q4_0 keeps ~99 % of the 262k window in VRAM on a 12 GB card, it is the 12 GB default from bundle-20261016 (245k decode 4x q8_0); BONSAI_CTK=q8_0 keeps the per-token precision when the window is short.
• q3_0 on 8 GB cards. q3_0 is the 8 GB default from bundle-20261015b. Against an f16 K/V cache it has a mean KL of 0.008 at 16k (q4_0 0.0021, q8_0 0.00024), the same top token 96 % of the time, perplexity +0.45 % at 16k (kv-kl-20261018). On tasks: HumanEval 164 163/164 (RTX 2060 SUPER, P131), and at 245k exact copies 20/20 and needles 23/24, equal to the dense read item by item (P148).
• Reasoning effort. The GGUF template defaults to xhigh (an extra "think carefully" line: runaway thinking, empty answers). low behaves close to xhigh. medium is the model's natural thinking and beats both, and thinking off, at every output cap from 2k up (Killy's HumanEval grid: 160-161 of 164). Default: medium with a 20k thinking budget and a force-close message so a trip still yields the answer.
• Reasoning budget report (patch 0035). Every response to a request that ran with a thinking budget says whether the budget closed the thinking: timings.reasoning_budget_exhausted (true / false), timings.reasoning_n (tokens inside the thinking, forced tokens included) and the OpenAI field usage.completion_tokens_details.reasoning_tokens, streaming and non-streaming. A client can count forced closes without searching reasoning_content for the budget message. A request may send its own reasoning_budget_tokens; when that is above the server's budget, the output cap becomes that budget + 4096 (also for a request with no max_tokens, which before was cut at the server default inside its thinking). Check: python bench\budget_report_probe.py --base http://127.0.0.1:18080 --key-file artifacts\api_key.txt. On agent work the budget rarely acts: AppWorld (168 tasks, 2835 model calls) has no call above 3,021 completion tokens.
• Harness-proofing. Cline, Kilo and Open WebUI send effort: "high", which the template rejects: HTTP 500, 0 of 164. Apps with 256-4096 token caps end a thinking model mid-thought (17-137 of 164). The server maps unknown effort words to medium and raises small caps to the thinking budget. Replays of Killy's rows against this server: Killy's plates, replayed: effort: "high" 0 -> 160 of 164 (the same server with harness-proofing off: HTTP 500, 0), a 4096-token cap 137 -> 157, medium 161.
• Tool calls. The native format is Qwen3-Coder XML with raw string parameters, grammar-constrained by the server: 9 of 9 parsed vs 1 of 9 for JSON-in-content. For file-writing agents, thinking off parsed 8 of 9 vs 6 of 9 at medium (which spent 10-17k tokens thinking first): send enable_thinking: false per request.
• Generated pages: run them. Killy's voxel pagoda at medium, 6 samples: 3 correct on the first try, and all 6 after bench/pagoda_plate.py --repair rendered each page headless and sent back what it saw (console errors, voxels floating, structure out of view). The first-try misses were one-token slips (a space inside a hex literal, an import map missing its imports key). Agents that run their output get this for free. (pagoda details)

Other pages here: Benchmark your own model the way this repo does.

About this page

262,144 tokens is the trained maximum. The bundle serves that window on 8 GB and 12 GB cards. The 12 GB default K/V cache is q4_0 since bundle-20261016 (BONSAI_CTK=q8_0 for the previous q8_0 recipe); the 8 GB default is q3_0 since bundle-20261015b; cards with more than 14 GB keep q8_0. The sections below are dated: some describe the recipes of their time (98,304 with q8_0, then 262k with q8_0 and a tiered cache). Current setups: 12 GB, 8 GB.

Kernel work made this model fast on a 12 GB card. This page is about the other complaint people have, that Bonsai 2 is worse than its 98% retention claim on code and agentic work, and about which part of that gap is serving software rather than compression. Every number here is measured on the original PrismML Ternary-Bonsai-2-27B-PTQ1_0.gguf with the binaries in this repo on an RTX 4070 12 GB. The recipe changes that come out of it are the defaults of start-server.ps1.

Nothing here touches the weights. Same 1.75 bits per weight, same file.

The frame: Killy's failure census

@net_termina ran 34 adversarial agentic tasks against Bonsai 2 and the real Qwen3.8-27B teacher (teacher 25 correct, Bonsai 15) and classified all 19 Bonsai failures:

Failures	Cause	Nature
7	tool-call JSON the harness could not parse: an escape dropped 4k-12k characters into a code string	serving format
8	ran out of steps	mostly serving: runaway thinking, retry loops after a bad call
4	understood the task, got it wrong	compression

His reasoning benchmarks survived compression almost untouched (GSM8K 480 vs 480, passkey 40/40, MATH-500 452 vs 461) because reasoning is redundant and self-correcting. Exact syntax is not: one wrong token deep inside a string kills a call and nothing repairs it. That points the runtime at two things: emit fewer wrong tokens (KV precision), and never let a wrong token become an unparseable call (format + grammar).

One caveat he also established and that applies to every number below: the "F16" GGUF in PrismML's repo is the ternary weights re-expanded to 16 bit (every 128-weight group holds one magnitude, exactly 42 of 128 are zero, embeddings and lm_head included). KL against it is KL of the model against itself. So this page only measures the model against itself under different runtime settings. Absolute distance to the teacher needs the real Qwen3.8-27B weights.

1. KV cache precision (measured, changed the default)

Every published 12 GB recipe for this model, ours included until now, runs -ctk q4_0 -ctv q4_0 to fit the full 262,144-token window. Nobody had measured what that costs on this model.

Method: llama-perplexity KL divergence, wikitext-2 test set, 16,384-token chunks x 4 (65k tokens processed, the second half of each chunk scored, so every scored token sits at 8k-16k depth). The reference is the same model with f16 K/V; each quantized cache is scored token-by-token against it. bench/kv_kl_sweep.ps1 reproduces it (base run 1.5 min, each q8_0/q4_0 config 1.5 min; the mixed config takes 44 min because mixed K/V types are not on the CUDA flash-attention path and fall back to the CPU).

K / V cache	Mean KL vs f16 KV	Top-1 token agrees with f16 KV	Max KL on one token	PPL vs f16 KV
q8_0 / q8_0	0.00017	99.38%	0.18	-0.03%
q8_0 K / q4_0 V	0.00094	98.59%	3.22	-0.06%
q4_0 / q4_0	0.00218	97.93%	0.94	+0.18%

Reading it:

• Perplexity does not see the difference (+0.18%), which is why the q4_0 recipe spread without anyone flagging it. The argmax does: q4_0 flips the top token on 1 in 48 positions at depth, q8_0 on 1 in 160. Twelve times the divergence. For prose that is a synonym; for code, an escape sequence, a closing quote or a JSON key it is the failure Killy counted. It also compounds: a long agentic session re-reads its own earlier output through the same cache on every turn.
• K precision is a bit more than half the damage, but V at 4 bits still costs 5x q8_0 and produced the single worst token in the set (KL 3.2, an 83-point probability swing). Mixed K/V is not a shortcut, and today it is also 10x slower on CUDA.

Memory. Bonsai 2 has 16 full-attention layers (every 4th of 64), 4 KV heads x 256 dims, so K+V is 32,768 values per token:

Window	q4_0 KV	q8_0 KV	VRAM in use with the 6.3 GB MTP file and its draft context (measured)
65,536	1.2 GB	2.3 GB	q8_0: 9.6 GB
98,304	1.8 GB	3.4 GB	q8_0: 10.8 GB
131,072	2.4 GB	4.6 GB	q4_0: 9.9 GB; q8_0: 11.96 GB, pages
262,144	4.8 GB	9.1 GB	q4_0: 11.96 GB, pages; q8_0 needs 16 GB

"Pages" is the finding that moved the default. 128k/q8_0 + draft allocates without complaint on a 12 GB card, and a fresh-context probe runs at full speed, but at 11.96 GB Windows (WDDM) has begun paging the cache to system memory: decode at 32k depth measured 30.1 tok/s against 47.3 with the same setup in a 64k window, and 12.6 tok/s at 100k. PrismML's own build at 11.35 GB did not page. The line is somewhere between 11.4 and 11.9 GB in use, and a recipe has to stay under it at every depth, not just on the probe that gets screenshotted.

Superseded 2026-09-27: the default is now the full 262,144 window with q8_0 K/V on 12 GB, through a tiered KV cache (first ~113k positions in VRAM, the rest in pinned system RAM, bit-identical output); see Q8_FULL_CONTEXT.md. What follows is the all-VRAM recipe it replaced (BONSAI_TIER=0).

Default until then: 98,304 tokens with q8_0 K/V, 10.8 GB in use, no paging at any depth. On 12 GB: BONSAI_CTX=131072 BONSAI_CTK=q4_0 for a longer window with the q4_0 noise; BONSAI_CTX=262144 BONSAI_CTK=q4_0 BONSAI_SPEC=0 for the full window (the draft context is what pushes 262k over the line). 16 GB and up: 262k with q8_0.

Not yet done: iq4_nl K/V. Same 4.5 bits per value as q4_0 with a non-linear codebook, usually a large KL improvement at identical memory. It is not implemented in the CUDA flash-attention kernels (the in-place q4_0/q8_0 MMA path in this repo would need an iq4_nl variant), so it has to be measured through the slow CPU fallback first. If it recovers most of the q4_0 gap it becomes the free 262k answer on 12 GB.

2. Runaway thinking (the quality gap that is not compression)

The GGUF chat template (tokenizer.chat_template) is explicit:

if enable_thinking is undefined or true:
    resolved_reasoning_effort = reasoning_effort|default('xhigh')
    xhigh  -> extra system line "think carefully... prioritize correctness..."
    low    -> extra system line "keep your thinking brief..."
    medium -> thinking on, no extra line
if enable_thinking is false:
    emit <think></think>   # pre-closed; the model answers immediately

xhigh is the silent default. That is Killy's "reasoning madness": no stop sign, the model fills half the KV with <think>, then either emits nothing (SVG plate: 10/12 subjects blank when thinking was off and the template still had no answer, red/thinking cells empty from budget blow-up) or a fragment. He also flagged that published Terminal-Bench / DeepSWE numbers may have been crushed the same way.

His coding bake-off on the original ternary file, same output cap on both arms (pass counts as published; thinking-off is flat because it never spends the extra tokens):

cap	MBPP think-off	MBPP medium	HumanEval think-off	HumanEval medium
10,224	37.72	35.44	14.77	12.77
20,488	37.73	42.22	14.77	15.00
40,966	37.73	44.11	14.77	15.88
81,922	37.73	45.33	14.77	15.99

Medium is worse than off at 10k and pulls ahead from 20k. That is the budget, not the weights: he said medium closed the gap to the 27B teacher, and that a single code prompt reasoned through ~50% of the KV before the first line of code. Omead (Prism) independently said medium is the setting they kept.

start-server.ps1 therefore:

• chat default reasoning_effort=medium (BONSAI_EFFORT) — never leave it unset (xhigh)
• --reasoning-budget 20480 (BONSAI_THINK_BUDGET) — first cap in his table where medium wins; 4096 was our old default and is below his break-even
• --reasoning-budget-message "Now produce the complete answer." so a force-close still writes the SVG/code instead of an empty content field
• agent harnesses BONSAI_THINK=0 (enable_thinking=false). With tools and thinking on, even at low + 9k tokens, write_file spent the whole budget inside <think> and never called (finish_reason: length). Thinking-off delivered a parseable call 9 of 9 times (next section)

Per-request chat_template_kwargs still override. The budget is a host-side sampler and turns off GPU-side sampling (~4% decode; BONSAI_THINK_BUDGET=-1 gets it back).

Client max_tokens counts thinking. A request with max_tokens: 4096 and thinking on dies inside <think> before </think> — the server budget never gets a chance. Measured on this card, same four SVG prompts plus one short code task (bench/reason_ab.py):

arm	empty	SVG landed	code landed
think-off	0	4/4	yes
low	4	0/4	yes
medium	1	4/4	no (thought through the 4096 cap)
xhigh	4	0/4	yes

OpenAI clients that leave max_tokens at 4k will reproduce Killy's blanks even with the new server default. Chat needs max_tokens ≥ 20k when thinking is on. Agents should keep thinking off and can stay at a small cap.

Thinking length and the system turn (measured 2026-10-10 to 10-13)

Since 09-21 the GGUF template adds a system turn with short work rules ("Save long command output to a file ...") to every request. HumanEval 164, medium, greedy, raw server, the same engine for every arm (bench/he_template_one.sh, docs/receipts/he_template_ab.log):

what the prompt has	pass	mean tokens	median	over 4k tokens
work rules in the system turn (the template as served)	160/164	2312	576	18
no work rules, no system turn	159/164	932	710	3
the same work rules as a prefix of the first user turn	160/164	956	676	3
system turn: "Reasoning effort is set to medium. Think as much as the task needs, then answer." + the rules	161/164	2327	599	19
a community template's "terse" system block, no rules	160/164	2454	594	21
work rules in the system turn, thinking budget 4096	160/164	1178	589	18 (capped near 4.1k)
work rules in the system turn, thinking budget 2048	160/164	913	606	0

• A system turn, with any text, gives a long thinking tail on about 11 % of the tasks (18-21 runs past 4k tokens against 3). The median hardly moves. Without a system turn, or with the same rules in the user turn, the tail goes away and the score stays.
• An effort sentence does not fix it (2327 mean with the medium sentence).
• A thinking budget cuts the tail with the rules kept: 2048 keeps 160/164 at 913 mean tokens. The budget-message force-close lets the answer through.
• The launcher maps every reasoning_effort except medium to medium (BONSAI_EFFORT_ALLOWED), so a request's low does not reach the template.
• No change shipped (measured 2026-10-11/13). Both fixes lose agent work. With a coding client's system prompt, a template that moves the system text into the first user turn cuts HumanEval tokens (2209 -> 1476 mean, 158/164 both), but the workspace tool tasks drop from 13/15 to 11/15. A 2048 thinking budget drops them from 13/15 to 6/15. The served template and the 20480 budget stay. Every run: docs/research/thinking-20261010/.

3. Tool-call syntax (measured: format and grammar, not the model)

Bonsai 2's native tool-call format is Qwen3-Coder XML, not JSON:

<tool_call>
<function=write_file>
<parameter=path>
parser.py
</parameter>
<parameter=content>
...raw file content, no escaping...
</parameter>
</function>
</tool_call>

String parameters are raw text terminated by </parameter>; there is no escaping to get wrong. The llama-server in this bundle parses that format with a PEG parser and, when a request carries tools, arms a lazy grammar (common_chat_params_init_qwen3_coder): string parameters accept anything up to the closing tag, object/array parameters are JSON-schema-constrained, so an unescaped quote inside a JSON value is simply not a legal next token. Under this path the model cannot produce a call the server fails to parse. The failure Killy measured needs a harness that makes the model write Hermes-style {"name":..., "arguments":{...}} with the code escaped inside a JSON string.

bench/toolcall_stress.py reproduces both paths against this server with payload-heavy tasks (250-line modules full of quotes, backslashes, regexes and f-strings; three multi-line edits as a JSON array parameter), thinking off, sampling at the recommended temp 1.0 / top-p 0.95 / top-k 20:

Arm	Format	Calls parsed	Finished within 9,000 tokens	Payload compiles as Python	Median payload	Decode
A	native XML + server grammar (tools in request)	9 / 9	9 / 9	3 / 9 (see below)	22,551 chars	66.9 tok/s
B	Hermes JSON in content, no grammar	1 / 9	5 / 9	0 / 9	14,543 chars	77.5 tok/s

Three runs x three tasks per arm (artifacts/eval/toolcall_stress.json). Every JSON-arm failure was Killy's: the object closes one bracket short at the very end of a 3k-23k character payload (Expecting ',' delimiter at char 23319 of 23319), or a string is never terminated (Unterminated string starting at char 74), or the model never gets out of the payload at all (4 of 9 hit the 9,000-token cap). None of those can happen on the grammar path, and none did.

"Payload compiles" is a stricter bar than "call parsed" and mostly measures the model, not the call. The six grammar-arm misses break down as: two apply_edits tasks where the model called write_file first to create the file it was asked to edit (a legitimate plan; the calls were well-formed), and four write_file payloads with a real Python bug inside the 250-line module: a """ opened inside a """ docstring, a bare ' inside a '...' raw regex, a C:\Users path in a non-raw string. Those errors exist in either format, reach the file intact, and an agent loop sees them as an ordinary Python error on the next step and can fix them. An unparseable call is different: it ends the step with nothing to repair. That is the column that matters, and it went from 8 of 9 failing to 0 of 9.

Serving advice that falls out of this, for any harness in front of this server: send tools and let the server format and parse the call. Do not prompt the model to write JSON tool calls in content, and do not run it behind a server that lacks the Qwen3-Coder XML grammar.

Cost: grammar-constrained sampling runs on the host, so decode with tools attached is slower than plain decode: 66.9 vs 77.5 tok/s above, about 14%, paid only on requests that carry tools. (Both arms are below the 93 tok/s plain-chat figure because these payloads are long code with speculative acceptance around 0.66 on the teacher MTP graft / 0.71 on the ProCreations on-policy head, rather than the short-answer probe.)

4. Speed at depth, and where the draft head stops paying (measured, changed the default)

Every speed claim for this model, this repo's earlier ones included, is a fresh-context probe. Agentic sessions live at 20k-80k tokens. bench/served_depth.ps1 runs the same three prompts behind 32k and 64k tokens of varied filler (sized with the server's tokenizer, not guessed) and reads the server's own timings. RTX 4070, 96k window, q8_0 K/V, 400-token answers:

decode tok/s	fresh	32k deep	64k deep
PrismML build	54.0	40.7	31.9
bundle kernels, no draft	68.3	49.1	(38 est.)
bundle + MTP draft, always on	101.2	45.3	27.2
bundle + MTP draft to 24k, then plain (default)	100.7	47.6	36.9

The draft head is +85% on a fresh context and a net loss past ~28k tokens. Deep in the context a step is bound by reading the KV cache; the two draft passes (the MTP layer has its own attention over the whole context) and the 3-column verify read more of it without shortening the step, and acceptance does not rise to compensate. Rather than pick between fresh speed and deep speed, the bundle adds --spec-draft-depth-max to llama-server (patch 0020): the slot drafts while the sequence is shorter than N and decodes one token per step after, skipping the draft-context catch-up too. The recipe sets 24,576.

At 64k the bundle's lead shrinks to 16% because attention dominates and the flash-attention kernel is shared with PrismML's build. That kernel at depth, with q8_0 K/V and head size 256, is the next thing to open up; nothing in this bundle touches it beyond the in-place K/V read.

What was not found

• No evidence of a numerics problem in the CUDA path that would degrade the model relative to CPU reference: every kernel in the bundle is checked against the CPU implementation (test-backend-ops, all PTQ1_0 shapes). Greedy output with each optimization on and off matches on this 4070. GGML_CUDA_BATCH_INVARIANT=1 is what makes a token verified in an MTP batch use the same layout and the warp-reduce epilogue as a token decoded alone. Without the flag, the faster four-accumulator epilogue can flip a late near-tie (5080: "coastal waters" vs "coastal areas") while staying coherent. The flag does not make every mmap near-tie identical, and it does not cover attention: past ~32k (and at every depth with GGML_CUDA_FA_DEEP_MMA=256, the current serve default) the stream-k KV split follows the padded KV length, which a verify batch can move one tile ahead of a later single-token decode. Measured 2026-09-26 at 40k: a code continuation matched, a prose one diverged, both coherent. See docs/Q8_FULL_CONTEXT.md.
• Nothing here changes the 4-of-19 "understood the task, got it wrong" bucket. That is the compression, and the only lever on it is training against the real teacher, which is out of scope for a runtime.

Reproduce

# KV precision (needs artifacts\eval\wiki.test.raw: wikitext-2-raw-v1 from huggingface.co/datasets/ggml-org/ci)
bench\kv_kl_sweep.ps1 -Model models\Ternary-Bonsai-2-27B-PTQ1_0.gguf -Bin bin -Configs q8_0:q8_0,q4_0:q4_0

# tool-call syntax, against a running start-server.ps1 (port 8080 by default)
python bench\toolcall_stress.py --base http://127.0.0.1:8080 --key (Get-Content artifacts\api_key.txt -Raw).Trim() --n 3