Skip to content

Commit 7ecb2b4

Browse files
VickyXAI1bcMax
andauthored
docs(tokens): correct the note claiming gateway max_output is unenforced (#109)
* docs(tokens): correct the note claiming gateway max_output is unenforced The comment added in 3.35.2 said the gateway's max_output values are 'not enforced: Franklin sends max_tokens 16384 to haiku today and the gateway accepts it.' Accepting the request and honoring it are different things. The gateway clamps with Math.min(request.max_tokens, model.maxOutput) on both handlers and prices the quote off the clamped ceiling — the request is accepted rather than rejected, so the clamp stays invisible until a reply is long enough to hit it. The smoke test that produced that claim returned ten tokens. The conclusion the comment supports is unchanged: the static tables stay authoritative, because the gateway's values are wrong for models we know. Only the reasoning was wrong, and a wrong reason is how a right rule gets reverted later. * docs(tokens): make the correction itself precise Two things the first pass got imprecise, in a comment whose entire job is precision about this mechanism: - It cited 8192 / 64000 in the present tense. Those exact values are being corrected upstream (BlockRunAI/blockrun#266), so the comment was written pre-stale. Reframed as a dated observation, with the durable point stated separately: the catalog can be wrong, and a wrong value here is not cosmetic. - It called the clamp 'invisible' without qualification. The handler does log 'capping to <limit>' — server-side, where no caller sees it (chat/completions/route.ts:1343). Invisible from here, not silent everywhere. Someone who finds that warn should not conclude the comment is wrong again. --------- Co-authored-by: 1bcMax <195689928+1bcMax@users.noreply.github.com>
1 parent fff9c12 commit 7ecb2b4

1 file changed

Lines changed: 15 additions & 5 deletions

File tree

src/gateway-models.ts

Lines changed: 15 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -92,11 +92,21 @@ export function clearGatewayModelsCache(): void {
9292
* Deliberately ignores the TTL: a stale gateway record still beats no record
9393
* at all, and the only callers are fallbacks for models we have no static
9494
* entry for. Callers must treat the static tables as authoritative — the
95-
* gateway's own metadata is not reliable (verified 2026-07-20: it reports
96-
* max_output 8192 for claude-haiku-4.5, which Anthropic documents as 64000,
97-
* and 64000 for claude-sonnet-4.6, documented as 128000). It is also not
98-
* enforced: Franklin sends max_tokens 16384 to haiku today and the gateway
99-
* accepts it. Treat this as "better than a blind default", nothing more.
95+
* gateway's own metadata has been wrong for models we already know. As of
96+
* 2026-07-20 it reported max_output 8192 for claude-haiku-4.5 (Anthropic
97+
* documents 64000) and 64000 for claude-sonnet-4.6 (documented 128000).
98+
* Those two specific numbers are being corrected upstream, so do not treat
99+
* them as current — the point that outlives them is that the catalog can be
100+
* wrong, and a wrong value here is not cosmetic.
101+
*
102+
* Correction to an earlier note here: these values are NOT inert metadata.
103+
* The gateway clamps with them — Math.min(request.max_tokens, model.maxOutput)
104+
* in both the messages and chat/completions handlers — and derives its price
105+
* quote from the clamped ceiling. An over-cap request is accepted rather than
106+
* rejected: the handler logs "capping to <limit>" server-side and continues,
107+
* so from here the clamp is invisible until a reply is long enough to hit it.
108+
* That is why a short smoke test against a wrongly-capped model looks healthy.
109+
* Treat this as "better than a blind default", nothing more.
100110
*/
101111
export function peekGatewayModel(id: string): GatewayModel | null {
102112
if (!cache) return null;

0 commit comments

Comments
 (0)