feat(claude-code): agent-aware prompt-cache keepalive - #5446
feat(claude-code): agent-aware prompt-cache keepalive#5446ArshansGithub wants to merge 17 commits into
Conversation
Opt-in agent-aware prompt-cache keepalive settings with presence-aware defaults (before-expiry 5m, only-when-agents-active true, liveness claude-code-tasks, max-probes 6, max-tokens 1) and validation.
New internal/runtime/keepalive package: one pending probe per Claude Code session, superseded by the next real request, gated on task liveness and on the session still being bound to the same credential, capped at max-probes consecutive probes. Probe bodies keep tools/system/messages and every cache_control marker byte-identical and only rewrite max_tokens, stream and thinking. Liveness reads Claude Code task state (~/.claude/tasks/<session>/*.json, both the full-UUID and session-<prefix> directory spellings) and recent task output files under /private/tmp/claude-<uid>.
…on path The Claude executor records every confirmed Claude Code request that carried an explicit 1h cache_control TTL. The probe replays the stored client body through Manager.Execute with the credential pinned, so it gets the same translation, cloaking, header and cache_control handling a real request does and warms the same per-account entry. Session affinity gains a non-refreshing BoundAuthID lookup so the scheduler can tell "still bound", "binding lost" and "routing is not session-sticky" apart. The scheduler is installed at startup and reconfigured on every config reload. Also ignores the local cliproxyapi-patched build output.
Credential selection does not key the session affinity cache by the bare Claude Code session id; it canonicalizes it first, for example to "claude:<session>:agent:main", and publishes that under canonical_session_id. Looking the binding up by the bare id always missed, so every probe was skipped as auth-binding-lost. The scheduler now carries both identities: the Claude Code session id keys the scheduler and the liveness check, the canonical identity keys the binding lookup. An empty canonical identity skips the binding check rather than failing it. Also announces the keepalive settings only when they change, so startup no longer logs the same line twice.
A probe travels the ordinary request path, so the observation hook was recording it as a real request. That reset the consecutive-probe budget on every probe, which would let an idle session probe forever and defeat max-probes. The prober now marks its execution metadata and the hook skips it.
Adds GET /v0/management/cache-keepalive, a read-only route behind the same auth as the rest of /v0/management. It returns every tracked session with its auth, ttl, last request, next probe, probe counts and last outcome, plus process-wide counters for scheduled, fired, hits, misses, errors and skips by reason. Each probe now emits one info line carrying the whole outcome, so `grep cache-keepalive main.log` needs no other source. A probe whose own cache_read_input_tokens is zero logs at warning with the upstream diagnosis: the entry it was meant to refresh had already expired, which is the signal that keepalive is not working for that session. A session that stops probing stays listed with active:false and the reason, so "nothing to do" is distinguishable from "silently broken". Retiring drops the stored request body immediately, so no request content is reachable through the endpoint, and the retired history is capped.
Ending a session recorded the reason into last_probe, which erased the outcome of the probe it had just sent. The reason now has its own retired_reason field, and last_probe only carries a skip when the skip happened instead of a probe.
The claude-code-tasks check was reading ~/.claude/tasks/<session>/*.json, which is Claude Code TodoWrite state, not subagent state. A session with a running subagent frequently has no todo file at all, so the check reported it dead and no probe was ever sent. The primary signal is now the per-agent task output file under <output-dir>/<project-slug>/<session>/tasks/*.output. Every background agent and shell has one, and a running agent keeps writing to it. Many are symlinks into the project subagent transcripts, where the real writes land, so the check follows the link: the link timestamp lags the target and reading it would report a busy agent as gone. The todo files stay as a secondary OR, commented as such. Freshness is measured against a new agent-idle-window (default 10m), not the cache TTL. An agent silent for an hour is finished, not busy, and probing for it would burn the budget on a dead session.
A probe that reads far less than the real request did refreshed only a fragment of the prefix, which is the same malfunction as reading nothing. The observed request cache_read_input_tokens is now recorded as the baseline, a probe reading under half of it is a miss, and the warning line reads "probe MISSED" and carries diagnostics.cache_miss_reason. The baseline is only available on the non-streaming path, where the response usage is in hand at observation time. A zero baseline leaves the outright check, which is the previous behaviour.
The rule is "below half", so a probe reading exactly half the baseline is a hit. The table asserted the opposite and failed; the implementation was right.
A real Claude Code session rejected every probe with "clear_thinking_20251015 strategy requires thinking to be enabled or adaptive". The probe removes thinking, because extended thinking forces a minimum max_tokens far above the probe budget, but Claude Code also sends a context_management edit that requires thinking to be present. Neither participates in the cached prefix, so the probe now drops both together, keeping any other context_management edit and removing the block only when nothing is left.
The probe was guessing at four candidate paths for the cache-diagnosis field. The shape is confirmed from captures: diagnostics.cache_miss_reason sits beside usage in a non-streaming body and inside the message_start event message when streaming, carrying type and cache_missed_input_tokens. Both are now read and the guesses are gone, and the missed token count reaches the warning line and the endpoint. This duplicates applyClaudeCacheMissReason from the cache-stats branch, which is not on this branch to import; the comment points at it so the two can be folded together once both land.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 29bffbeb02
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| previous := s.sessions[in.SessionID] | ||
| generation := uint64(1) | ||
| if previous != nil { | ||
| generation = previous.generation + 1 |
There was a problem hiding this comment.
When a root agent and one of its subagents both send qualifying requests, they share the bare Claude Code session ID even though BindingSessionID distinguishes their agent identities. This lookup therefore replaces the root agent's stored body, model, and timer with the subagent's state; the eventual probe warms the subagent's cache instead of the blocked orchestrator cache that this feature is intended to preserve. Key scheduler entries by the canonical agent identity and model while retaining the bare session ID for filesystem liveness checks.
Useful? React with 👍 / 👎.
| if !c.onlyWhenAgentsActivePresent { | ||
| c.OnlyWhenAgentsActive = true | ||
| } |
There was a problem hiding this comment.
When an SDK caller supplies config.Config directly through Builder.WithConfig, setting OnlyWhenAgentsActive: false cannot set the private presence flag, so WithDefaults always changes the requested false value back to true. Only YAML decoding can currently populate this flag, making the exported option behave differently for programmatic SDK configuration; use a representation that can distinguish unset from false without relying solely on UnmarshalYAML state.
Useful? React with 👍 / 👎.
| cutoff := now().Add(-window) | ||
| for _, root := range l.OutputDirs { | ||
| for _, dirName := range sessionDirNames(sessionID) { | ||
| entries, errGlob := filepath.Glob(filepath.Join(root, "*", dirName, "tasks", "*.output")) |
There was a problem hiding this comment.
For a qualifying request whose X-Claude-Code-Session-Id contains glob metacharacters, filepath.Glob interprets the client-controlled ID as a pattern rather than a directory name. For example, * matches task output from every session, so activity in an unrelated session makes the attacker's session appear live and triggers unnecessary paid probes; path separators can also move the lookup outside the intended session directory. Validate the identifier or escape it before inserting it into the glob.
Useful? React with 👍 / 👎.
| if _, terminal := terminalTaskStatuses[status]; !terminal { | ||
| log.Debugf("cache-keepalive: liveness hit | session=%s source=todo-state file=%s status=%s", | ||
| truncateSession(sessionID), entry, status) | ||
| return true |
There was a problem hiding this comment.
If a TodoWrite JSON file remains in pending or in_progress after its task has actually finished, this fallback reports the session live indefinitely because it checks only the status and ignores both the file timestamp and agent-idle-window. When no recent task-output file exists, that stale record alone causes probes to continue until max-probes, contrary to the intended active-agent gate; apply a freshness check or require corroboration before treating this secondary signal as live.
Useful? React with 👍 / 👎.
| } | ||
|
|
||
| probeStart := s.now() | ||
| result, errProbe := prober.Probe(context.Background(), ProbeRequest{ |
There was a problem hiding this comment.
If configuration is disabled or the service shuts down while a probe is running, Stop can cancel only pending timers because the network request uses context.Background(). An unresponsive upstream can consequently leave its goroutine, connection, and credential work alive after service shutdown; derive this context from a scheduler-owned lifecycle context and cancel it in Stop, rather than introducing a post-connection timeout.
AGENTS.md reference: AGENTS.md:L58-L58
Useful? React with 👍 / 👎.
ExtendedCacheTTL only reports a TTL the client wrote out, so a request that
carried a bare {"type":"ephemeral"} marker was indistinguishable from one
that cached nothing. Both were dropped, which is correct while only the 1h
pool is probed and wrong the moment 5m sessions become eligible.
RequestCacheTTL resolves the wire default instead: any cache_control marker
selects at least the 5m pool. TTLTier names the pool, which is what the
scheduler will key the lead time and the probe budget by.
probe-5m (auto|always|never, default auto), probe-5m-models, before-expiry-5m (45s) and max-probes-5m (30). The 5m pool needs its own lead time because 5m minus the 1h default of 5m is zero, and its own budget because 6 probes cover 25 minutes there against 6 hours on the 1h pool. Validation rejects a before-expiry-5m at or above the 5m TTL, which would otherwise drop every 5m session silently, and skips both 5m knobs when probe-5m is never.
The 1h-only rule assumed cache reads cost 0.1x base input, where the twelve reads an hour a 5m window needs are a wash against the 1.25x write they avoid. claude-fable-5-1 and claude-mythos-5-1 read at 0.025x base, so those twelve reads cost ~0.3x of the context against that same 1.25x write: probing is about four times cheaper than letting the entry expire. probe-5m=auto therefore schedules a 5m session when the request model is on the cheap-cache-read list, always takes every session, never keeps the old rule. The tier picks the lead time and the probe budget; the probe body, the liveness gate and the binding check are the 1h path unchanged, and the liveness gate is still what bounds the spend. Both tiers already scheduled from the request's start rather than the response, which matters far more at 45s of margin than at 5m; the reschedule after a probe measures from the probe's own start for the same reason. The tier and the probe-5m decision now appear in every scheduled and probe log line and in the management payload, and the two skips are counted under skipped_by_reason so a policy that drops every 5m session is not silent.
The old text asserted the 5m pool is never worth refreshing. That is true at a 0.1x-base cache read and false at 0.025x, so the doc now carries the arithmetic both ways rather than the conclusion alone, along with the new keys, the tier fields in the log lines and the management payload, and why the 5m lead time is 45s rather than minutes.
|
Follow-up pushed: the keepalive now covers 5-minute sessions, not just the 1h pool. The 1h-only rule assumed cache reads cost 0.1x base input, where the ~12 reads an hour a 5m window needs (1.2x the context) are a wash against the 1.25x write they avoid. On New keys under
The probe body, the liveness gate and the binding check are the 1h path unchanged; the liveness gate is still what bounds the spend.
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d758e36ec2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // BindingUnknown means routing is not session-sticky, so there is no binding | ||
| // to check. The scheduler proceeds: with no affinity there is no binding to lose. | ||
| BindingUnknown BindingState = iota |
There was a problem hiding this comment.
When routing.session-affinity remains at its default false and multiple Claude credentials participate in round-robin routing, BindingUnknown lets a probe proceed pinned to credential A, but the next real request may be routed to credential B. The probe then incurs a cache read without preserving the cache entry that the user subsequently hits; keepalive must maintain its own routing binding or require session affinity for multi-credential pools.
AGENTS.md reference: AGENTS.md:L3-L3
Useful? React with 👍 / 👎.
| s.cfg = cfg | ||
| if cfg.Enabled { | ||
| s.mu.Unlock() | ||
| return |
There was a problem hiding this comment.
When an enabled configuration is hot-reloaded from probe-5m: always or an eligible auto model list to probe-5m: never or an ineligible list, this early return leaves every existing 5-minute timer active. fire never reevaluates the 5-minute policy, so those sessions continue sending and rescheduling paid probes until their budget is exhausted despite the new configuration; cancel or retire sessions that are no longer eligible when applying the config.
Useful? React with 👍 / 👎.
| if c.BeforeExpiry <= 0 { | ||
| return fmt.Errorf("claude-code.cache-keepalive.before-expiry must be positive") | ||
| } |
There was a problem hiding this comment.
If an operator sets before-expiry: 1h (or larger), validation accepts the configuration even though every supported 1-hour observation computes a non-positive delay and is silently skipped by Observe. This makes an enabled keepalive configuration do nothing; validate that the 1-hour lead time is below the supported 1-hour TTL, just as before-expiry-5m is bounded below five minutes.
Useful? React with 👍 / 👎.
1 participant
Branch:
feat/claude-code-cache-keepalive— 13 commits offdev(18e01a7), HEAD29bffbeb.Summary
With
promptCacheTtl: "1h", an orchestrator session that blocks on a subagent for longer than the TTL pays a full cache write of its whole context on the return turn instead of a read. On the 1h pool that write costs roughly ten times the read it replaces, so refreshing the entry with one minimal request per window is close to free. A client-side hook cannot do this, because it cannot guarantee which account the refresh lands on; the proxy holds the last request body, the credential and the session-to-account binding, so its probe warms the same per-account entry the next real request will hit. The feature is opt-in and off by default, and only requests from a confirmed Claude Code client that carried an explicitcache_controlttl of1hare ever probed. A 5m session is never probed: on that pool the thirteen reads an hour cost more than the single write they avoid.Independent of the cache-stats and version-floor branches. It reads the cache-miss diagnostics itself rather than importing the cache-stats helper; see the note below.
Diff
26 files, +3,637 / −0.
internal/runtime/keepalive/keepalive.go(new)internal/runtime/keepalive/keepalive_test.go(new)docs/cache-keepalive.md(new)sdk/cliproxy/service_cache_keepalive.go(new)internal/runtime/keepalive/liveness_claude_code.go(new)internal/runtime/keepalive/liveness_claude_code_test.go(new)internal/config/claude_cache_keepalive.go(new)internal/runtime/executor/helps/claude_cache_keepalive_test.go(new)internal/config/claude_cache_keepalive_test.go(new)internal/api/handlers/management/cache_keepalive_test.go(new)internal/runtime/executor/helps/claude_cache_keepalive.go(new)internal/runtime/keepalive/liveness_claude_code_symlink_test.go(new)sdk/cliproxy/service_cache_keepalive_test.go(new)config.example.yamlsdk/cliproxy/auth/selector_session_binding.go(new)internal/runtime/keepalive/default.go(new)internal/api/handlers/management/cache_keepalive.go(new)internal/runtime/executor/claude_executor_execute.gointernal/runtime/executor/claude_executor_stream.goservice_lifecycle.go,service_config.go,config_load.go,parse.go,sdk_config.go,server_management.go,README.md)No deletions; the two executor files gain one Observe call and a start timestamp each.
Behaviour
cache_control.ttl: "1h", the body is stored and a timer is armed forttl − before-expiry.max-probesconsecutive probes were already sent, if no agent is live, or if the auth binding was lost or moved. Otherwise the stored body is sent withmax_tokens: 1and withthinkingand the thinking-dependentcontext_managementedit removed, since neither is in the cached prefix and the API rejects one without the other. Tools, system, messages,cache_controland betas go out byte-identical.Manager.Executewithpinned_auth_id, so translation, headers, retries and token refresh are the same path a real request takes. The probe marks its own context so it cannot observe itself and rearm.claude-code-taskslooks for any<output-dir>/<project-slug>/<session>/tasks/*.output(symlinks resolved) modified withinagent-idle-window. The~/.claude/taskstodo list is only a secondary OR, since it is not subagent state. Documented caveat: a silent process does not advance its output file, so the window must exceed the expected silence.liveness: alwaysskips the check entirely, at a cost of up tomax-probesreads per idle session.cache-keepalive:prefix on every schedule, probe and skip, with a reason.probe MISSEDat WARN carryingcache_miss_reasonandcache_missed_input_tokenswhen the probe's read is zero or below half the observed baseline.GET /v0/management/cache-keepalivereports per-sessionnext_probe_at,probes_sent,last_probeandretired_reasonplus global counters. Retired sessions stay listed under a cap so idle and broken are distinguishable; stored bodies are dropped on retirement.Config
Applied on hot reload; disabling the block cancels every timer and logs
cache-keepalive: disabled.Verification
123 packages: 93 with tests, all passing; 30 with no test files; 0 failures. 63 tests cover the scheduler, the liveness reader including the symlink case, the config, the management handler and the diagnostics reader.
Behaviour verified on a scratch instance against api.anthropic.com with a real
claudesession (haiku,--strict-mcp-config) and a ticking background task: probes fired while the task ran, reportingstatus=hitwithcache_read_input_tokensequal to the real request's, then ano-live-agentsskip once the task ended and the window passed. A second run took two hits and stopped atmax-probes, with the endpoint reflecting both the live and the end state. Probing a real body is what surfaced thecontext_managementrejection that the final probe shape fixes.config.example.yamldocuments only theclaude-code.cache-keepaliveblock on this branch; nousage-cache-statskeys are present.Note for reviewers
claudeCacheMissReasoninsdk/cliproxy/service_cache_keepalive.goreadsdiagnostics.cache_miss_reasonfrom both the non-streaming body and the streamingmessage_startevent. It duplicatesapplyClaudeCacheMissReasonfrom the per-session cache-statistics branch, which is not a dependency of this one. The comment in the file points at the other implementation so the two can be folded into a shared helper once both land.Why: see https://github.com/ArshansGithub/claude-code-fable-usage
Context: https://github.com/ArshansGithub/claude-code-fable-usage.