feat(cache): cache-aware dedup with preserve_cache_prefix option

[Siddhant-K-code]

Closes #50

What

Adds preserve_cache_prefix to /v1/dedupe so the dedup pipeline cannot reorder or remove chunks that appear before a cache_control breakpoint. Without this, Distill can silently invalidate Anthropic prompt cache prefixes while improving context quality.

Changes

pkg/cache/prefix.go (new)

• PartitionForCacheAwareDedup(chunks) — detects cache_control markers in chunk metadata, returns PrefixPartition with Prefix (frozen), Suffix (dedup-eligible), PrefixHash, FrozenPrefixTokens, MarkerCount
• hasCacheControl handles string, map, bool, and nil marker values
• PrefixAwareStats type for response stats

cmd/api.go

• DedupeChunk gains cache_control string field
• DedupeOptions type with preserve_cache_prefix bool
• DedupeRequest gains Options DedupeOptions
• DedupeStats gains cache_prefix_frozen, cache_prefix_tokens, cache_prefix_hash, suffix_input_count, suffix_output_count
• handleDedupe: partitions chunks when preserve_cache_prefix=true, runs cluster/select/MMR only on the suffix, prepends frozen prefix to output

pkg/cache/prefix_test.go (new) — 6 tests covering no-marker passthrough, single/multiple markers, marker at end, hash stability, hash change detection

API

POST /v1/dedupe
{
  "chunks": [
    {"id": "sys", "text": "You are a helpful assistant.", "cache_control": "ephemeral"},
    {"id": "msg1", "text": "What is 2+2?"},
    {"id": "msg2", "text": "What is 2 plus 2?"}
  ],
  "options": {"preserve_cache_prefix": true}
}

Response:

{
  "chunks": [{"id": "sys", ...}, {"id": "msg1", ...}],
  "stats": {
    "cache_prefix_frozen": true,
    "cache_prefix_tokens": 64,
    "cache_prefix_hash": "a3f2c1d4e5b6f7a8",
    "suffix_input_count": 2,
    "suffix_output_count": 1
  }
}

Why opt-in

Not all workloads use prompt caching. The current full-reorder behaviour produces better context quality when caching is not a concern. preserve_cache_prefix=false (default) leaves existing behaviour unchanged.