fix(tokens): one token-limit table for CLI and proxy; catalog fills gaps only#108
Merged
Conversation
…aps only The proxy carried its own hardcoded max-token ladder (deepseek|haiku|gpt-oss -> 8192, everything else -> 16384) that never consulted MODEL_MAX_OUTPUT, so proxy users were capped at 16K on models the CLI already knew could emit far more — Kimi K3 at 65,536 and GPT-5.6 Sol at 128,000 were losing 4-8x. Both paths now read getMaxOutputTokens(). Separately, both getMaxOutputTokens() and getContextWindow() now consult the live gateway catalog — but ONLY for model ids with no static entry. This is the qwen3.7-max class: a real paid model nobody has catalogued, which silently compacted at 128k and capped output at 16k. The catalog is a gap-filler. It is deliberately NOT a source of truth, which is the opposite of what the obvious design would do. Two reasons, both verified live 2026-07-20: 1. The gateway's own numbers are wrong for models we do know. It reports max_output 8192 for claude-haiku-4.5 (Anthropic documents 64000) and 64000 for claude-sonnet-4.6 (documented 128000). They are also not enforced — Franklin sends max_tokens 16384 to haiku today and the gateway accepts it, so the field is unreliable metadata, not a contract. 2. Where the gateway is right, we deliberately disagree. Every Anthropic model is pinned to a 200k context window against the advertised 1M because the gateway's 1M beta header is not enabled and anything larger 413s (src/agent/tokens.ts:213-218). Letting the catalog win would reintroduce exactly the bug that comment prevents. Cache access is a synchronous cache-only peek that never fetches, with a fire-and-forget warm on the first static-table miss, so nothing is added to startup latency and a cold or unreachable gateway degrades to today's behavior. Four tests pin the precedence in both directions: static entries survive a warm catalog, and an uncatalogued id picks up the catalog's values. Known-wrong static values left alone on purpose: haiku-4.5 max output is 16384 here vs Anthropic's documented 64000, and sonnet-4.6 is 64000 vs 128000. Raising them needs a real long-output run against a gateway whose own metadata we just established is unreliable — separate change, separate verification. Local suite 634 pass.
This was referenced Jul 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two token-limit paths disagreed, and neither could learn about a model nobody had catalogued.
The proxy had its own ladder.
src/proxy/server.tscappedmax_tokenswith a hardcoded substring chain —deepseek|haiku|gpt-oss → 8192, everything else→ 16384— that never consultedMODEL_MAX_OUTPUT. Proxy users were pinned at 16K on models the CLI already knew could emit far more: Kimi K3 (65,536) and GPT-5.6 Sol (128,000) were losing 4-8×. Both paths readgetMaxOutputTokens()now.Nothing filled the gaps.
getMaxOutputTokens()andgetContextWindow()now consult the live gateway catalog for ids with no static entry — theqwen/qwen3.7-maxclass, a real paid model that silently compacted at 128k and capped output at 16k until someone hand-added it.Why the catalog is a gap-filler and not the source of truth
This inverts the obvious design.
src/gateway-models.tssays "Gateway is the single source of truth" in its header, so wiring it in as authoritative is the natural move. It's wrong, for two reasons — both verified against the live catalog on 2026-07-20:1. The gateway's numbers are wrong for models we already know.
claude-haiku-4.5claude-sonnet-4.6claude-opus-4.8They're also not enforced — Franklin sends
max_tokens: 16384to haiku today, comfortably above the advertised 8192, and the gateway accepts it. The field is unreliable metadata, not a contract.2. Where the gateway is right, we deliberately disagree. Every Anthropic model is pinned to a 200k context window against the advertised 1M, because the gateway's 1M beta header is not enabled and anything larger 413s (
src/agent/tokens.ts:213-218). Letting the catalog override would reintroduce exactly the bug that comment exists to prevent.So: static table wins; catalog only answers for ids we have never heard of.
How
peekGatewayModel(id)— synchronous, cache-only, never fetches. Safe from the hot path and from sync functions.warmGatewayModelsCache()— fire-and-forget, kicked on the first static-table miss. No startup latency added; a cold or unreachable gateway degrades to exactly today's behavior.Left alone on purpose
haiku-4.5max output is 16,384 here vs Anthropic's documented 64,000, andsonnet-4.6is 64,000 vs 128,000. Both are under-estimates, not 400 risks. Raising them needs a real long-output run against a gateway whose metadata this PR just established is unreliable — separate change, separate verification.Tests
Four new tests pin the precedence in both directions: static entries survive a warm catalog, an uncatalogued id picks up the catalog's values, and the proxy ladder cannot come back.
Local suite: 634 pass, 0 fail.
Gateway-side follow-ups (not in this PR)
max_outputis wrong forclaude-haiku-4.5(8192 vs 64000) andclaude-sonnet-4.6(64000 vs 128000).qwen/qwen3.7-maxrejectstool_choicewith400 Invalid request: 400 Provider returned error, naming nothing. Franklin's runtime retry keys on the error text mentioningtool_choice, so an opaque relay defeats it — v3.35.1 had to hardcode the model into a blocklist instead. Passing the provider's original message through would let the generic recovery work.