[RFC][Frontend/CLI]: Offline prefix-cache workload analyzer

[raravind007]

Motivation

vLLM's automatic prefix caching reuses KV-cache blocks only when the computed block-hash chain matches at full-block granularity. The hash includes the parent block hash, the current block tokens, and optional extra keys such as LoRA IDs, multimodal input hashes, and cache salts. As a result, small prompt-formatting differences, per-request metadata, different chat templates, LoRA/multimodal metadata, or cache-salt differences can prevent reuse even when prompts look similar to a human.

Today, users can inspect prefix-cache behavior through runtime logs, Prometheus metrics, and benchmark scripts, but there is no user-facing, no-GPU way to analyze a request dataset before deployment and answer whether its prompt structure is cache-friendly. This makes it difficult to reason about RAG, agent, tool-calling, and multi-turn workloads before spending GPU time on a serving run.

Related user pain: #29944, #12077, #24394, and the forum thread "How log kvcache usage and prefix hit rate in offline infer?".

Proposal

Add an offline prefix-cache analyzer CLI, tentatively:

vllm analyze-prefix-cache \
  --model meta-llama/Meta-Llama-3-8B-Instruct \
  --input requests.jsonl \
  --format openai-chat \
  --block-size 16 \
  --output-format text

Given a JSONL of requests and a model/tokenizer, the analyzer would:

• Parse OpenAI batch-style requests and/or plain prompt JSONL.
• Apply the selected chat template and tokenize each request.
• Compute the same full-block hash-chain structure used by vLLM prefix caching, reusing existing request/block hashing utilities rather than reimplementing hashing logic.
• Group requests by shared hash-chain prefixes.
• Report offline cacheability estimates, including:
  • total prompt tokens
  • total full-block tokens
  • estimated reusable full-block tokens
  • estimated cacheability ratio
  • top shared prefix groups
  • request groups that diverge after a common prefix
• Support both human-readable text output and machine-readable JSON output.

This is static analysis over tokenized requests and existing hashing utilities. It should not instantiate the engine, execute model weights, or require GPU access.

Design question: internal API usage

The hashing logic this proposal depends on (hash_block_tokens, generate_block_hash_extra_keys, get_request_block_hasher) currently lives in vllm/v1/core/kv_cache_utils.py — internal to the engine core. A CLI/frontend module importing directly from v1/core crosses a layering boundary that may not be acceptable as-is. Before implementation, we'd like maintainer input on whether this warrants a small, stable public helper for offline hash-chain construction, versus importing the internal utilities directly for a first version.

Scope for v1

• CLI command for offline workload analysis.
• Text output by default.
• JSON output for automation and CI/capacity-planning workflows.
• Support for plain prompt JSONL and OpenAI-compatible chat/batch request JSONL.
• Block-size parameter support, with optional comparison across multiple block sizes.
• Mechanical divergence reporting: show where hash chains stop matching and which request groups share common full-block prefixes.
• Documentation with RAG, agent, tool-calling, and multi-turn examples.

Explicit non-goals for v1

• Exact runtime prefix-cache hit-rate prediction.
• Modeling scheduler behavior, eviction, memory pressure, request arrival order, preemption, KV offload, or disaggregated serving.
• Natural-language cause classification such as "this miss was caused by timestamps" or "this miss was caused by per-user IDs."
• Full multimodal/LoRA/cache-salt inference unless the relevant metadata is present in the input format and can be passed through the same hashing path.
• Engine behavior changes.

The analyzer should present results as an offline upper-bound or cacheability estimate, not as a guarantee of production hit rate.

Alternatives considered

Existing runtime metrics — Runtime metrics and logs can show prefix-cache behavior after a server or offline inference run, but they do not provide a pre-flight analysis tool for a request corpus.

benchmarks/benchmark_prefix_caching.py — Benchmarks prefix-caching performance by running an actual model with fixed prompts or ShareGPT-style prompts. This proposal is different: it does not run inference, does not require GPU, and analyzes a user's request dataset structurally before deployment.

Hash microbenchmarks — Existing hash benchmarks are useful for comparing hashing throughput and algorithm choices, but they do not estimate cacheability of a real prompt workload.

Open questions

• Should this be a top-level command such as vllm analyze-prefix-cache, or should it live under an existing CLI family such as vllm bench prefix-cache-analyze?
• What input formats should v1 support beyond plain prompts and OpenAI-compatible request JSONL?
• Should JSON output be considered stable in v1, or marked experimental?
• How much LoRA, multimodal, and cache-salt metadata should v1 support?
• Should block-size comparison be built into v1, or added later?

Happy to implement this if the design direction is acceptable.

[harsh543]

I'd like to take a first pass at this. Plan for v1, scoped down from the
full proposal to something reviewable:

• Plain-prompt JSONL only for now ({"prompt": ..., "id": ...} per line);
  OpenAI chat/batch JSONL as a fast-follow once the report shape is agreed.
• Reuses hash_block_tokens / get_hash_fn_by_name / init_none_hash
  directly from vllm/v1/core/kv_cache_utils.py rather than waiting on a
  new public helper, per your note in "Design question: internal API
  usage" — happy to help design that helper afterward once there's a
  second caller to generalize from.
• No LoRA/multimodal/cache-salt extra-key support in v1, matching the
  non-goals section.
• vllm analyze-prefix-cache --model ... --input requests.jsonl as a
  top-level subcommand (matches your primary proposal); open to vllm bench prefix-cache-analyze if that's the maintainers' preference.

Have a working draft locally. Want me to open it as a PR?

[raravind007]

Now that #48369 exists, the layering question from "Design question: internal API usage" above has a concrete shape.

prefix_cache_analysis.py's _full_block_hash_chain re-derives the same block-splitting + parent-chaining logic that get_request_block_hasher's inner closure already does incrementally, per generation step, in vllm/v1/core/kv_cache_utils.py. Rather than adding a new public module, I'd suggest extracting a small function alongside hash_block_tokens in kv_cache_utils.py — e.g. build_full_block_hash_chain(token_ids, block_size, hash_fn) -> list[BlockHash] — that both the analyzer and the real request-hashing path can call. That gives it a legitimate second caller to generalize from immediately, rather than waiting for one to show up later. Left more detail on this, plus one scaling finding on the grouping logic, in a review on #48369.

Separately: once #48369 lands, I'll pick up OpenAI chat/batch JSONL input support, and I'm happy to help extract the build_full_block_hash_chain helper proposed above. @harsh543 if you want either of those or had already planned to take them, let's coordinate rather than duplicate.

[NickLucche]

Hey, why would this analyzer need to live within vLLM codebase?

[raravind007]

Good question! this is the same tension I flagged in the RFC under “Design question: internal API usage,” so let me make the rationale more explicit.

The analyzer’s value depends on computing the same hash chain that the V1 scheduler computes: the same hash_block_tokens semantics, parent chaining, init_none_hash behavior, hash-function selection, and tokenization path—and the same chat-template path once that input format lands. That coupling is what makes the output trustworthy, but it also makes an out-of-tree implementation harder to maintain.

An external tool would generally have two options:

1. Import directly from vllm/v1/core/kv_cache_utils.py, creating a dependency on private internals with no stability guarantee.
2. Reimplement the hashing path, which creates a risk of semantic drift as prefix-cache behavior evolves.

Either approach could result in the analyzer no longer matching the engine, potentially without an obvious failure. Keeping the analyzer in-tree makes that coupling explicit and allows changes to the hashing path and the analyzer to be reviewed and tested together.

Moving the CLI out of tree would not entirely remove the in-tree requirement. A maintainable external analyzer would likely still need a small, stable helper—for example, an API that constructs the full block-hash chain. My concern with exposing such an API without an in-tree consumer is that compatibility issues may be less visible over time.

The CLI itself would have a relatively small footprint: CPU-only, no engine behavior changes, no GPU dependency, and no new heavy dependencies. In that sense, I see it as similar to other diagnostic or planning commands such as vllm bench or collect-env.

That said, I’m not attached to the CLI location. If maintainers prefer a smaller in-tree surface, a reasonable fallback would be to expose the stable hash-chain helper in vLLM and initially place the analyzer externally or under tools/. The main property I would like to preserve is that the analysis uses the engine’s exact hashing semantics rather than an independent approximation.

[harsh543]

@raravind007 Sounds good on both — go ahead and take the OpenAI chat/batch JSONL support and the build_full_block_hash_chain extraction, I hadn't started either. Happy to review once you've got something up. Will ping if that changes.

[harsh543]

@raravind007 Sounds good on both — go ahead and take the OpenAI chat/batch JSONL support and the build_full_block_hash_chain extraction, I hadn't started either. Happy to review once you've got something up. Will ping if that changes.

[raravind007]

Thanks !! will open the helper first, since it also addresses the layering question above.