[MRV2][Spec Decode] Adaptive Speculative Decoding - Initial Support

[benchislett]

Purpose

This PR implements initial support for Adaptive Speculative Decoding using DSpark's confidence head: we enable fully variable-length per request speculative decoding with FULL CUDA Graph support. This PR does not implement online dynamic scheduling of the verification budget; instead, we rely on user-provided num_speculative_tokens_per_batch_size to determine what size pool of draft tokens to verify at each batch size.

This PR fleshes out a number of fundamental aspects of variable-length speculation, such as:

• Attention metadata flags to declare compatibility with variable length decoding
• FULL CUDA Graph compatibility
• Structured outputs support
• Logprobs support
• Proper integration with statistics capture
• Mixed prefill/decode batch compatibility (preliminary; falls back to uniform batch with same dynamic-SD selected budget)

Design and Implementation Details

Coming soon 😴

Key areas for reviewers are:

• Scheduler/ModelRunner responsibility assignment: I tried a few times to implement with an allocation-blind scheduler and have ModelRunner do all the real "scheduling" work, but it seemed even more hacky once structured outputs, mixed prefill/decode, and non-uniform graph support entered the picture. Still, it could go either way.
• VARLEN_BATCH flag on attention cudagraph compatibility: is this a sufficient way to check that a backend is compatible with variable-length queries?

Testing

Since only FLASH_ATTENTION is currently functional, I run with Qwen3-8B-FP8 and the corresponding DeepSeek-trained DSpark. Since verification is so cheap on an 8B model, no significant speedup is measurable; that is not the objective of this PR. However, I do observe an increase in acceptance rate under a fixed verification budget:

Acceptance Rate Study: SPEED-Bench, All Categories, Concurrency 128, Shuffled

SPEED-Bench, All Categories, Concurrency 128, Shuffled

Category	Baseline (K=5)	Baseline (K=7)	Adaptive (Max K=7, Effective K=5 Per Request)
coding	4.6267	5.5640	5.0972
humanities	3.2844	3.6392	3.5036
math	4.2535	5.0100	4.5879
multilingual	3.0081	3.4400	3.3648
qa	2.9236	3.2078	3.0896
rag	3.7169	4.2810	4.0475
reasoning	3.9697	4.5843	4.2913
roleplay	2.3373	2.4564	2.4243
stem	3.5963	4.1574	3.9192
summarization	2.9987	3.3062	3.1559
writing	3.1799	3.4470	3.2988
Overall Average	3.4450	3.9176	3.7073

Measured using NVIDIA SPEED-Bench harness, thinking off, vLLM config:

engine_args:
  speculative_config:
    adaptive_verification: true
    num_speculative_tokens_per_batch_size:
      - [1, 128, 5]
    attention_backend: FLASH_ATTN
  gpu_memory_utilization: 0.75
  attention_backend: FLASH_ATTN
  max_num_batched_tokens: 32768
sampling_kwargs:
  temperature: 0.7
  top_p: 0.8
  top_k: 20
dataset_kwargs:
  shuffle_seed: 42

I also implement various unit tests, including E2E coverage of acceptance length, GSM8k correctness, and an assert that we're observing almost-exactly N*K draft tokens at each step.

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[mergify]

Documentation preview: https://vllm--48692.org.readthedocs.build/en/48692/