What would you like to be added:
Part of #. Stage 1 (L1 + L2): a DataProducer, session-prefix-cache-producer, that maintains prefix-cache affinity at session granularity and feeds the existing prefix-cache-scorer slot. No tokenizer, no per-block state, no engine changes.
- Identity from content. A session id is a chain of hashes over framed content blocks (message boundaries plus fixed-size text chunking, so a conversation growing inside one message extends its chain rather than re-keying). Hash-prefix equality holds exactly when content prefixes are byte-identical — the KV-reuse condition — so continuations, forks, and template sharing resolve structurally. Declared client ids (
session headers, prompt_cache_key, previous_response_id) alias the chain.
- Estimate-seeded index.
session -> {engine -> extent}: the prefix is assumed to live where the turn was served (PreRequest), refined by engine-reported usage on completion (ResponseBody).
- Signal in the existing slot.
Produce publishes covered fraction per endpoint as the standard PrefixCacheMatchInfo; the generic prefix-cache-scorer binds via prefixMatchInfoProducerName. Composed with the standard queue and load scorers.
Scope is L1 + L2 only. L3 (truthful extents from engine-labeled events), L4 (retention), and the engine-side steps are tracked in #.
Why is this needed:
Prefix-cache routing already works — approx-prefix-cache-producer (tokenizer plus per-engine LRU simulation) and precise-prefix-cache-producer (truthful, from KV events). This adds a third option with a different profile: it derives session identity from request content, so affinity is tokenizer-free, deterministic on template-sharing traffic, and works against stock engines with no KV-event infrastructure. Its larger purpose is the session identity itself — the coordinate the orchestration roadmap in # (keeping, placement, program-aware scheduling) builds on; affinity is its first consumer. Prototyped and benchmarked on the optimized-baseline guide setup (8x Qwen3-32B, TP=1, H200): best-in-class prefix reuse across the session suites, ~20% faster fan-out burst TTFT p50.
And more importantly, better fit towards session-centric orchestration.
What would you like to be added:
Part of #. Stage 1 (L1 + L2): a
DataProducer,session-prefix-cache-producer, that maintains prefix-cache affinity at session granularity and feeds the existingprefix-cache-scorerslot. No tokenizer, no per-block state, no engine changes.sessionheaders,prompt_cache_key,previous_response_id) alias the chain.session -> {engine -> extent}: the prefix is assumed to live where the turn was served (PreRequest), refined by engine-reportedusageon completion (ResponseBody).Producepublishes covered fraction per endpoint as the standardPrefixCacheMatchInfo; the genericprefix-cache-scorerbinds viaprefixMatchInfoProducerName. Composed with the standard queue and load scorers.Scope is L1 + L2 only. L3 (truthful extents from engine-labeled events), L4 (retention), and the engine-side steps are tracked in #.
Why is this needed:
Prefix-cache routing already works —
approx-prefix-cache-producer(tokenizer plus per-engine LRU simulation) andprecise-prefix-cache-producer(truthful, from KV events). This adds a third option with a different profile: it derives session identity from request content, so affinity is tokenizer-free, deterministic on template-sharing traffic, and works against stock engines with no KV-event infrastructure. Its larger purpose is the session identity itself — the coordinate the orchestration roadmap in # (keeping, placement, program-aware scheduling) builds on; affinity is its first consumer. Prototyped and benchmarked on the optimized-baseline guide setup (8x Qwen3-32B, TP=1, H200): best-in-class prefix reuse across the session suites, ~20% faster fan-out burst TTFT p50.And more importantly, better fit towards session-centric orchestration.