Spike: benchmark and design vLLM prompt-token reuse for rehydrated history

[leseb]

Problem

Praxis rehydrates multi-turn history by loading stored JSON messages and sending the expanded request to the inference backend. The backend then tokenizes the full prompt again, including the unchanged conversation prefix.

The hypothesis is that reusing a backend-produced token representation could reduce CPU usage and latency for long conversations. This needs measurement and a backend contract before changing the response-store schema.

Current findings

• The rehydrate filter does not tokenize. It restores JSON messages into ResponsesState; tokenization happens in the inference backend.
• Standard OpenAI-compatible responses expose token counts, not the full prompt token-ID sequence needed for reuse.
• vLLM can expose raw prompt and output token IDs through its non-standard enable_response_messages extension, but we still need to prove a supported Responses request path can consume cached IDs on the next turn.
• Token IDs are not portable. Their validity depends on the effective backend, model, tokenizer revision, chat template, tools, special-token policy, and potentially adapter configuration.
• Filters after rehydrate can modify history, including file resolution and document extraction. Any such transformation invalidates tokens created for the earlier representation.
• Persisting the complete accumulated token sequence on every ResponseRecord could produce quadratic storage growth across a long response chain.
• vLLM Automatic Prefix Caching already targets repeated prefixes in multi-round conversations and reuses KV-cache computation, which may deliver more value than caching token IDs in Praxis.

References:

• vLLM Responses protocol
• vLLM Automatic Prefix Caching

Spike scope

1. Benchmark long multi-turn Responses workloads and isolate tokenization cost from model prefill and decoding.
2. Compare the baseline with vLLM Automatic Prefix Caching enabled.
3. Document the exact vLLM extension and version that produces reusable token IDs.
4. Prove an end-to-end request path that feeds cached IDs back into vLLM for a subsequent Responses turn, including streaming and tool-call history where supported.
5. Define cache identity and invalidation rules covering:
   • effective backend and model
   • tokenizer and chat-template fingerprints
   • tools, instructions, and special-token behavior
   • model rewrite, routing, failover, and adapters
   • downstream history mutations and compaction
6. Evaluate storage shape and bounds without duplicating the full accumulated token sequence in every response record.
7. Define a safe fallback to ordinary JSON rehydration and backend tokenization for missing, stale, unsupported, or mismatched cache entries.

Exit criteria

• Reproducible measurements show whether tokenization is a material bottleneck after Automatic Prefix Caching is considered.
• A supported backend API for consuming cached token IDs is demonstrated, or the spike records that no suitable API exists.
• Any proposed cache format is opaque and backend-specific, versioned, size-bounded, and tied to a complete validity fingerprint.
• The recommendation clearly chooses one of:
  • rely on backend prefix caching and close this issue;
  • open a narrowly scoped vLLM-specific implementation issue;
  • defer until the inference backend exposes a stable reusable-token contract.

Out of scope for the spike

• Adding cached_tokens directly to generic ResponseRecord or ConversationRecord types.
• Database schema migrations before the representation, producer, consumer, and invalidation contract are proven.
• Treating token IDs as provider-independent data.

Origin

Suggested in praxis-proxy/praxis#491 review by @franciscojavierarceo.

Parent

Part of #93. Related completed foundations: #21 (rehydrate) and #19 (response store).

[franciscojavierarceo]

I approve of this ticket 👍

[leseb]

moved parent to be #93 since this is optimization realm and i want to mark this filter as completed.