Skip to content

fix(tokens): one token-limit table for CLI and proxy; catalog fills gaps only#108

Merged
VickyXAI merged 1 commit into
mainfrom
fix/gateway-token-limits
Jul 20, 2026
Merged

fix(tokens): one token-limit table for CLI and proxy; catalog fills gaps only#108
VickyXAI merged 1 commit into
mainfrom
fix/gateway-token-limits

Conversation

@VickyXAI

Copy link
Copy Markdown
Contributor

What

Two token-limit paths disagreed, and neither could learn about a model nobody had catalogued.

The proxy had its own ladder. src/proxy/server.ts capped max_tokens with a hardcoded substring chain — deepseek|haiku|gpt-oss → 8192, everything else → 16384 — that never consulted MODEL_MAX_OUTPUT. Proxy users were pinned at 16K on models the CLI already knew could emit far more: Kimi K3 (65,536) and GPT-5.6 Sol (128,000) were losing 4-8×. Both paths read getMaxOutputTokens() now.

Nothing filled the gaps. getMaxOutputTokens() and getContextWindow() now consult the live gateway catalog for ids with no static entry — the qwen/qwen3.7-max class, a real paid model that silently compacted at 128k and capped output at 16k until someone hand-added it.

Why the catalog is a gap-filler and not the source of truth

This inverts the obvious design. src/gateway-models.ts says "Gateway is the single source of truth" in its header, so wiring it in as authoritative is the natural move. It's wrong, for two reasons — both verified against the live catalog on 2026-07-20:

1. The gateway's numbers are wrong for models we already know.

model Anthropic gateway Franklin
claude-haiku-4.5 64,000 8,192 16,384
claude-sonnet-4.6 128,000 64,000 64,000
claude-opus-4.8 128,000 128,000 128,000 ✓

They're also not enforced — Franklin sends max_tokens: 16384 to haiku today, comfortably above the advertised 8192, and the gateway accepts it. The field is unreliable metadata, not a contract.

2. Where the gateway is right, we deliberately disagree. Every Anthropic model is pinned to a 200k context window against the advertised 1M, because the gateway's 1M beta header is not enabled and anything larger 413s (src/agent/tokens.ts:213-218). Letting the catalog override would reintroduce exactly the bug that comment exists to prevent.

So: static table wins; catalog only answers for ids we have never heard of.

How

  • peekGatewayModel(id) — synchronous, cache-only, never fetches. Safe from the hot path and from sync functions.
  • warmGatewayModelsCache() — fire-and-forget, kicked on the first static-table miss. No startup latency added; a cold or unreachable gateway degrades to exactly today's behavior.

Left alone on purpose

haiku-4.5 max output is 16,384 here vs Anthropic's documented 64,000, and sonnet-4.6 is 64,000 vs 128,000. Both are under-estimates, not 400 risks. Raising them needs a real long-output run against a gateway whose metadata this PR just established is unreliable — separate change, separate verification.

Tests

Four new tests pin the precedence in both directions: static entries survive a warm catalog, an uncatalogued id picks up the catalog's values, and the proxy ladder cannot come back.

Local suite: 634 pass, 0 fail.

Gateway-side follow-ups (not in this PR)

  • max_output is wrong for claude-haiku-4.5 (8192 vs 64000) and claude-sonnet-4.6 (64000 vs 128000).
  • Upstream errors are swallowed. qwen/qwen3.7-max rejects tool_choice with 400 Invalid request: 400 Provider returned error, naming nothing. Franklin's runtime retry keys on the error text mentioning tool_choice, so an opaque relay defeats it — v3.35.1 had to hardcode the model into a blocklist instead. Passing the provider's original message through would let the generic recovery work.

…aps only

The proxy carried its own hardcoded max-token ladder (deepseek|haiku|gpt-oss
-> 8192, everything else -> 16384) that never consulted MODEL_MAX_OUTPUT, so
proxy users were capped at 16K on models the CLI already knew could emit far
more — Kimi K3 at 65,536 and GPT-5.6 Sol at 128,000 were losing 4-8x. Both
paths now read getMaxOutputTokens().

Separately, both getMaxOutputTokens() and getContextWindow() now consult the
live gateway catalog — but ONLY for model ids with no static entry. This is
the qwen3.7-max class: a real paid model nobody has catalogued, which silently
compacted at 128k and capped output at 16k. The catalog is a gap-filler.

It is deliberately NOT a source of truth, which is the opposite of what the
obvious design would do. Two reasons, both verified live 2026-07-20:

1. The gateway's own numbers are wrong for models we do know. It reports
   max_output 8192 for claude-haiku-4.5 (Anthropic documents 64000) and 64000
   for claude-sonnet-4.6 (documented 128000). They are also not enforced —
   Franklin sends max_tokens 16384 to haiku today and the gateway accepts it,
   so the field is unreliable metadata, not a contract.

2. Where the gateway is right, we deliberately disagree. Every Anthropic model
   is pinned to a 200k context window against the advertised 1M because the
   gateway's 1M beta header is not enabled and anything larger 413s
   (src/agent/tokens.ts:213-218). Letting the catalog win would reintroduce
   exactly the bug that comment prevents.

Cache access is a synchronous cache-only peek that never fetches, with a
fire-and-forget warm on the first static-table miss, so nothing is added to
startup latency and a cold or unreachable gateway degrades to today's
behavior.

Four tests pin the precedence in both directions: static entries survive a
warm catalog, and an uncatalogued id picks up the catalog's values.

Known-wrong static values left alone on purpose: haiku-4.5 max output is 16384
here vs Anthropic's documented 64000, and sonnet-4.6 is 64000 vs 128000.
Raising them needs a real long-output run against a gateway whose own metadata
we just established is unreliable — separate change, separate verification.

Local suite 634 pass.
@VickyXAI
VickyXAI merged commit 28605bf into main Jul 20, 2026
2 checks passed
@VickyXAI
VickyXAI deleted the fix/gateway-token-limits branch July 20, 2026 22:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant