Repository navigation
fix(chat): inject cache breakpoints in marker-less claude passthrough - #18025
Draft
patrykkopycinski wants to merge 2 commits into
Draft
patrykkopycinski wants to merge 2 commits into
patrykkopycinski wants to merge 2 commits into
Conversation
Anthropic-format requests forwarded through the non-CC pure passthrough branch reached upstream with zero cache_control markers whenever the client shipped none (e.g. Cursor) — re-billing the full prompt prefix every turn (678M uncached prompt tokens/week observed on one gateway). ensurePassthroughCacheBreakpoints now runs in the plain passthrough branch for Anthropic-native upstreams (claude + anthropic-compatible-*) and injects the standard breakpoint set only when the body carries no markers anywhere: system tail ttl=1h, second-to-last user message and last assistant message plain ephemeral. A client that manages its own markers stays byte-untouched. The CC semantic-passthrough branch keeps its own contract (ensureCacheControlOnLastUserMessage). Wiring is pinned by a handleChatCore-level test with a mocked upstream: injection asserted on the marker-less body, zero-injection on the client-managed body, and a mutation arm (call disabled → test red) proved the wiring bites. extended-cache-ttl beta is already in ANTHROPIC_BETA_BASE, so ttl=1h markers are accepted upstream.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this fixes
Marker-less Anthropic-format clients (e.g. Cursor) send zero
cache_controlmarkers through the pure claude→claude passthrough branch, so every turn re-bills the full prompt prefix. On one gateway this measured 678M uncached prompt tokens/week onprovider=claude.Change
ensurePassthroughCacheBreakpoints(new helper inclaudeHelper.ts) now runs in the plain passthrough branch ofchatCore/claudePassthroughBody.tsfor Anthropic-native upstreams (claude+anthropic-compatible-*):ttl=1h, second-to-last user message + last assistant message plain ephemeralensureCacheControlOnLastUserMessage)extended-cache-ttlbeta is already inANTHROPIC_BETA_BASE, sottl=1hmarkers are accepted upstreamTests
tests/unit/claude-passthrough-cache-rescue.test.ts— helper contract (inject-on-marker-less, zero-injection on client-managed)tests/unit/claude-passthrough-cache-wiring.test.ts— handleChatCore-level wiring test with mocked upstream; includes a mutation arm (call disabled → test red)