Skip to content

Session-granularity prefix-cache affinity producer (tokenizer-free, stock-engine) #1980

Description

@vMaroon

What would you like to be added:

Part of #. Stage 1 (L1 + L2): a DataProducer, session-prefix-cache-producer, that maintains prefix-cache affinity at session granularity and feeds the existing prefix-cache-scorer slot. No tokenizer, no per-block state, no engine changes.

  • Identity from content. A session id is a chain of hashes over framed content blocks (message boundaries plus fixed-size text chunking, so a conversation growing inside one message extends its chain rather than re-keying). Hash-prefix equality holds exactly when content prefixes are byte-identical — the KV-reuse condition — so continuations, forks, and template sharing resolve structurally. Declared client ids (session headers, prompt_cache_key, previous_response_id) alias the chain.
  • Estimate-seeded index. session -> {engine -> extent}: the prefix is assumed to live where the turn was served (PreRequest), refined by engine-reported usage on completion (ResponseBody).
  • Signal in the existing slot. Produce publishes covered fraction per endpoint as the standard PrefixCacheMatchInfo; the generic prefix-cache-scorer binds via prefixMatchInfoProducerName. Composed with the standard queue and load scorers.

Scope is L1 + L2 only. L3 (truthful extents from engine-labeled events), L4 (retention), and the engine-side steps are tracked in #.

Why is this needed:

Prefix-cache routing already works — approx-prefix-cache-producer (tokenizer plus per-engine LRU simulation) and precise-prefix-cache-producer (truthful, from KV events). This adds a third option with a different profile: it derives session identity from request content, so affinity is tokenizer-free, deterministic on template-sharing traffic, and works against stock engines with no KV-event infrastructure. Its larger purpose is the session identity itself — the coordinate the orchestration roadmap in # (keeping, placement, program-aware scheduling) builds on; affinity is its first consumer. Prototyped and benchmarked on the optimized-baseline guide setup (8x Qwen3-32B, TP=1, H200): best-in-class prefix reuse across the session suites, ~20% faster fan-out burst TTFT p50.

And more importantly, better fit towards session-centric orchestration.

Metadata

Metadata

Assignees

No one assigned

    Labels

    needs-triageIndicates an issue or PR lacks a triage label and requires one.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions