BPB: use renderer-reported content bytes as the denominator

[YujiaBao]

Follow-up to #823 / #824.

Problem

The BPB denominator counts the UTF-8 bytes of the decoded trained tokens. #824 excluded registered special tokens from both sides, but scaffolding that renderers inject as ordinary text still counts as target bytes. The clearest case: Kimi renderers put a <think></think> block in every trained assistant message, and <think>/</think> are not in all_special_ids for the Kimi K2 tokenizer (where <think> is even three regular tokens: <, think, >) nor for Qwen3.

Scaffolding is near-zero-loss once the template is learned, so its bytes act as free denominator padding: measured BPB ≈ true BPB × B/(B+C) for B content bytes and C scaffolding bytes per turn. On a short turn ("2+2 equals 4." = 13 bytes), Kimi's trained span decodes to <think></think>2+2 equals 4. — 15 of 28 denominator bytes (54%) are scaffolding. The deflation varies with the chat template, biasing exactly the cross-model comparisons BPB exists for; templates with more markup look artificially better.

Token-level filtering can't fix this: renderers deliberately encode scaffolding + content as one concatenated string so tokenization matches the HF chat template, so no exact per-token content mask exists (and BPE can merge across the boundary). The message level is where content bytes are exact.

Fix

Renderers report content bytes. New helper message_content_byte_count(message, include_thinking=..., include_tool_calls=...) counts the payload that survives round-tripping through parse_response: text parts, thinking parts (when actually rendered), and tool-call function names/arguments. Each built-in renderer sets RenderedMessage.content_byte_count using flags that match what it actually rendered (e.g. thinking stripped from history → not counted; Nemotron's injected {reasoning effort: ...} suffix → discounted). For the same rendered content the count is independent of how the template formats it — which is what makes the denominator comparable. (Renderers that legitimately train on different content count different bytes: gpt-oss preserves historical thinking that Qwen3/Kimi strip, so cross-model comparisons are cleanest on data without thinking in historical turns; documented in the sweep README.)

A new entry point carries the total; dispatch matches Python attribute lookup. Renderer.build_supervised_example_with_metadata returns a SupervisedExample(model_input, weights, trained_content_bytes) where the count sums over loss-weighted messages. It is a @final dispatcher over a single overridable implementation chain, _build_supervised_example_impl (same protected-hook style as _render_tool_calls); kimi_k2 / gpt_oss / nemotron3 override the hook. The legacy build_supervised_example keeps its tuple signature and routes into the same chain. If any class in the MRO overrides the legacy method — the pre-existing extension point, at any layer, including overrides that call super().build_supervised_example() — the dispatcher routes through it exactly as a direct call would (reporting None for content bytes), so the two public entry points return identical tokens in every hierarchy, and no existing subclass silently changes its training tokens.

The count rides on the datum without touching the wire. DatumWithContentBytes is a tinker.Datum subclass with a plain attribute — deliberately not a loss_fn_inputs entry: the SDK serializes datums from model_input + loss_fn_inputs only (and forward_backward_custom rejects unknown loss_fn_inputs keys), so the field never reaches the service and datums pass through forward_backward unchanged. datum_from_model_input_weights drops the count if max_length truncation removes any loss-weighted token, since it would no longer describe the surviving span.

compute_bpb prefers content bytes. For datums carrying a count:

bpb = -sum(logprob_i for i where weight_i > 0) / (ln(2) * content_bytes)

The numerator is the model's entire trained code length — scaffolding pays its real bits, which vanish as the template is learned and are charged symmetrically to every model (no dependence on which tokens happen to be registered as special). The denominator is fixed by the trained content, not the template. Datums without a count (custom renderers overriding only the legacy method, or trained spans cut by max_length) fall back per-datum to the previous token-byte computation, so everything keeps working and train_mean_bpb / test/bpb keep their names; the sweep README documents the fallback.

With this, the same conversation scored under Kimi K2 and Qwen3 renderers yields the identical denominator (asserted in tests), and think-tag scaffolding remains trained but contributes no bytes.

Review

Beyond unit tests, the change went through an independent multi-lens review (metric math, dispatch/backward-compat, per-renderer accounting, SDK integration, test quality, docs accuracy, whole-diff pass) with two adversarial verifiers per finding. Confirmed findings — a dispatch edge where an impl-hook override layered above a legacy override diverged from direct-call behavior, the Nemotron effort-suffix bytes, and two over-broad doc claims — are all fixed in the last commit; the dispatcher now defers to any legacy override exactly as Python attribute lookup does.

Test

• renderers/content_bytes_test.py (new): kimi/qwen3 agree on content bytes for the same conversation; Kimi's <think></think> is trained but not counted; thinking counted only where rendered (last message vs. stripped history, both renderers); untrained messages contribute nothing; role_colon separators excluded; both entry points return identical tokens/weights. Dispatch regression tests: legacy-only overrides fall back to None with identical tokens; a legacy override layered on KimiK2Renderer is honored on both entry points; an impl-hook override layered above a legacy override still agrees; super().build_supervised_example() from a legacy override is recursion-free; impl-hook overrides report content bytes on both paths.
• renderers/nemotron3_test.py: the rendered {reasoning effort: low} suffix is excluded from content bytes.
• supervised/common_test.py: content-byte mode hand computation (numerator includes trained special tokens); weight-magnitude invariance (mean reduction); scaffolding pads the fallback denominator but not the content one; mixed batches fall back per-datum; length-mismatch raises; DatumWithContentBytes keeps loss_fn_inputs to exactly {weights, target_tokens}, pickles, and survives mean reduction; truncation into the trained region drops the count while truncation of untrained tokens keeps it.
• Verified the metadata attribute never reaches the wire: the SDK's pydantic and proto conversions rebuild datums from model_input + loss_fn_inputs only.
• uv run --extra dev pytest tinker_cookbook/renderers/ tinker_cookbook/supervised/ tinker_cookbook/recipes/chat_sl/ → 557 passed, 46 skipped (skips pre-existing, HF-gated); tests/downstream_compat/ → 499 passed; ruff check / ruff format --check clean; pyright 0 errors on all touched files.

🤖 Generated with Claude Code

https://claude.ai/code/session_01LC4d9VwvhNY3DX7EbmpMmv