Skip to content

Implement prompt caching across LLM providers #83

Description

@jrswab

Problem

Axe re-transmits the full system prompt (skill + files + memory), tool definitions, and message history on every turn of a multi-turn agent run. No provider currently sends cache hints, so users pay full input-token costs for static content on every API call. For agents with large SKILL.md files and many tool calls, this is a significant cost multiplier.

Goal

Add provider-side prompt caching support so that repeated static context (system prompt, tool schemas) is cached by the LLM provider after the first turn, reducing input-token costs on subsequent turns. Cache usage must be observable in verbose mode and JSON output.

Provider Support

Provider Mechanism Request Change Response Change
Anthropic Explicit cache_control: {type: "ephemeral"} Add cache hints to system blocks and tool defs Parse cache_creation_input_tokens / cache_read_input_tokens
OpenAI Automatic prefix caching None (automatic) Parse usage.prompt_tokens_details.cached_tokens
Bedrock (Claude) cachePoint blocks Add cache points to system/tool config Parse cacheReadInputTokens / cacheWriteInputTokens
Gemini Context caching API Out of scope (requires stateful two-phase API) Out of scope
Ollama N/A None None
OpenCode Pass-through No gateway-level control assumed No changes

Spec

See docs/plans/049_prompt_caching_spec.md for full requirements and implementation plan.

Acceptance Criteria

  • Cache annotations added to provider.Request and cache metrics to provider.Response
  • Anthropic provider sends cache_control on system prompt and parses cache usage
  • OpenAI provider reports cached_tokens from API responses
  • Bedrock provider sends cachePoint blocks and parses cache usage
  • Runner automatically enables caching on every request without agent TOML changes
  • Cache read/write tokens tracked cumulatively across conversation turns
  • Cache token counts visible in verbose output and JSON result envelope
  • Graceful degradation: unsupported providers/models continue to work unchanged
  • Tests added for each provider's cache request construction and response parsing

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions