|
| 1 | +--- |
| 2 | +pr: block/goose#8916 |
| 3 | +sha: 00c2141debc4eff86146ed4450ba2249a20ceec2 |
| 4 | +verdict: merge-as-is |
| 5 | +reviewed_at: 2026-04-29T18:31:00Z |
| 6 | +--- |
| 7 | + |
| 8 | +# fix(bedrock): cache trailing message for stable prefix across agent turns |
| 9 | + |
| 10 | +## Context |
| 11 | + |
| 12 | +In `crates/goose/src/providers/bedrock.rs` (`BedrockProvider::converse`, |
| 13 | +around line 232), the previous code placed prompt-cache breakpoints on |
| 14 | +the first three visible messages |
| 15 | +(`const MESSAGE_CACHE_BUDGET: usize = 3; let cache_count = … visible_messages.len().min(MESSAGE_CACHE_BUDGET)`) |
| 16 | +and then iterated `enumerate()` setting cache=true for `idx < cache_count`. |
| 17 | +This PR replaces that with a single trailing-message cache point. |
| 18 | + |
| 19 | +The author's reasoning matches Anthropic's prompt caching contract: |
| 20 | +cache reads walk *backward* from the breakpoint, hashing the prefix. |
| 21 | +A cache point pinned to position 0..3 means everything appended after |
| 22 | +position 3 has to be reprocessed every turn — linear growth in turn |
| 23 | +count. A trailing breakpoint means the next turn's lookback (≤20 |
| 24 | +blocks) finds the previous turn's write, and only the new content |
| 25 | +between turns gets fresh processing. |
| 26 | + |
| 27 | +## What's good |
| 28 | + |
| 29 | +- The diff is exactly the change described in the comment — no |
| 30 | + drive-by refactors. The new `last_idx = visible_messages.len().checked_sub(1)` |
| 31 | + + `cache_last && Some(idx) == last_idx` pattern is the cleanest |
| 32 | + Rust expression of "set the flag for exactly the last element, |
| 33 | + or none if the list is empty." |
| 34 | +- The misleading old comment ("caching recent messages would shift |
| 35 | + positions each turn, causing misses") is replaced with an accurate |
| 36 | + description of the lookup model and a documentation link. Future |
| 37 | + maintainers won't re-introduce the original mistake based on that |
| 38 | + comment. |
| 39 | +- The author correctly identified that the existing test surface |
| 40 | + (`providers::formats::bedrock` per-message helpers and |
| 41 | + `providers::bedrock::test_caching_*` enable-flag tests) doesn't |
| 42 | + actually exercise the *placement* of the cache point — it exercises |
| 43 | + whether `to_bedrock_message_with_caching` honors the boolean. |
| 44 | + Adding a placement test would be valuable but is not strictly |
| 45 | + required for correctness; the change itself is mechanically obvious. |
| 46 | +- The system-prompt cache point is left untouched, which is correct |
| 47 | + — system prompts are stable across turns, so a head-anchored |
| 48 | + breakpoint is the right policy there. |
| 49 | + |
| 50 | +## Concerns / nits |
| 51 | + |
| 52 | +- For agent loops that add >20 blocks per turn (large multi-step |
| 53 | + tool batches), the trailing-breakpoint strategy degrades to |
| 54 | + full-prefix reprocessing because the lookback window is exceeded. |
| 55 | + This is documented behavior, not a bug, but worth a comment in |
| 56 | + the code or a follow-up that emits a debug log when this |
| 57 | + threshold is approached. |
| 58 | +- `enable_caching && last_idx.is_some()` could be folded into a |
| 59 | + `let cache_last = enable_caching && !visible_messages.is_empty();` |
| 60 | + for one less `Option` ceremony. Style nit only. |
| 61 | + |
| 62 | +## Verdict |
| 63 | + |
| 64 | +`merge-as-is` — the analysis is correct, the diff is minimal, the |
| 65 | +old comment was wrong and is now right. The 20-block-window |
| 66 | +caveat is a follow-up consideration. |
0 commit comments