Skip to content

refactor: cache-aligned summarization for compaction requests#2752

Open
lloydzhou wants to merge 3 commits intocharmbracelet:mainfrom
lloydzhou:main
Open

refactor: cache-aligned summarization for compaction requests#2752
lloydzhou wants to merge 3 commits intocharmbracelet:mainfrom
lloydzhou:main

Conversation

@lloydzhou
Copy link
Copy Markdown

@lloydzhou lloydzhou commented Apr 29, 2026

Summary — Cache-Aligned Summarization

Align the summarization agent's prefix (system prompt, tools, cache-control markers) with the main agent so that providers supporting prompt caching reuse the already-cached prefix on compaction requests. Providers that don't support explicit cache-control markers silently ignore them — no behavior change.

将摘要 agent 的前缀(system prompt、tools、cache-control markers)与主 agent 对齐,使支持 prompt caching 的 provider 在压缩请求时复用已缓存的前缀。不支持显式 cache-control 标记的 provider 会静默忽略,行为无变化。

Cost Example (Claude Sonnet 4, $3.00/$0.30/$15.00 per MTok)

A compaction request with 100k input tokens: 5k prefix (system prompt + tools) + 94k old messages + 1k summary instruction.

Input cost Output cost Total
Without caching 100k × $3.00 = $0.300 500 × $15.00 = $0.0075 $0.3075
Cache-Aligned 99k × $0.30 + 1k × $3.00 = $0.0327 500 × $15.00 = $0.0075 $0.0402
Savings ~87%

This reuse strategy is inspired by bash-agent's cache-aware DP compaction.

此复用策略灵感来源于 bash-agent 的缓存感知 DP 压缩算法

What Changed

  • buildSystemPrompt() — shared system prompt builder (base prompt + MCP instructions), extracted from inline code in Run.
  • buildAgent() — shared agent factory: calls buildSystemPrompt(), copies tools (with cache control on the last tool), creates fantasy.Agent. Both Run and Summarize now call this instead of constructing agents independently.
  • applyCacheControl(messages) — shared cache marking: clears stale provider options, marks the system message boundary and last 2 messages with cache control. Both PrepareStep closures now call this.
  • buildSummaryPrompt — summary instructions are now embedded in the user prompt (via templates/summary.md) rather than as a separate system prompt, so the main agent's system prompt prefix is preserved.

变更内容

  • buildSystemPrompt() — 共享 system prompt 构建器(基础提示 + MCP 指令),从 Run 内联代码中提取。
  • buildAgent() — 共享 agent 工厂:调用 buildSystemPrompt()、复制 tools(最后一个 tool 添加 cache control)、创建 fantasy.AgentRunSummarize 均通过此方法构建 agent,不再各自独立构造。
  • applyCacheControl(messages) — 共享缓存标记:清除过期的 provider options,标记 system message 边界和最后 2 条消息的 cache control。两个 PrepareStep 闭包均调用此方法。
  • buildSummaryPrompt — 摘要指令改为嵌入在 user prompt 中(通过 templates/summary.md),而非作为独立的 system prompt,从而保留主 agent 的 system prompt 前缀用于缓存命中。

@charmcli
Copy link
Copy Markdown
Contributor

charmcli commented Apr 29, 2026

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@lloydzhou
Copy link
Copy Markdown
Author

I have read the Contributor License Agreement (CLA) and hereby sign the CLA.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants