Skip to content

[Bug]: kv_cache_utils warns that prefix-cache reuse is disabled even when disable_eagle_block_drop keeps it working #55518

Description

@twu3202

Your current environment

The output of python collect_env.py
Collecting environment information...
==============================
        System Info
==============================
OS                           : Ubuntu 22.04 LTS (x86_64)
GCC version                  : (Ubuntu 12.3.0-1ubuntu1~22.04.3) 12.3.0
Clang version                : Could not collect
CMake version                : Could not collect
Libc version                 : glibc-2.35

==============================
       PyTorch Info
==============================
PyTorch version              : 2.13.0+cu130
Is debug build               : False
CUDA used to build PyTorch   : 13.0
ROCM used to build PyTorch   : N/A
XPU used to build PyTorch    : N/A

==============================
      Python Environment
==============================
Python version               : 3.10.12 (main, Jun 22 2026, 18:55:27) [GCC 11.4.0] (64-bit runtime)
Python platform              : Linux-6.8.0-138-generic-x86_64-with-glibc2.35

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : Could not collect
CUDA_MODULE_LOADING set to   :
GPU models and configuration : GPU 0: NVIDIA RTX 6000 Ada Generation
Nvidia driver version        : 595.84
cuDNN version                : Could not collect
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : False

==============================
          CPU Info
==============================
Architecture:                            x86_64
CPU(s):                                  48
Vendor ID:                               AuthenticAMD
Model name:                              AMD Ryzen Threadripper PRO 7965WX 24-Cores
NUMA node(s):                            1

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.6.18
[pip3] nccl4py==0.5.0
[pip3] numpy==2.2.6
[pip3] nvidia-cublas==13.1.1.3
[pip3] nvidia-cuda-runtime==13.0.96
[pip3] nvidia-cudnn-cu13==9.20.0.48
[pip3] nvidia-ml-py==13.610.43
[pip3] nvidia-nccl-cu13==2.29.7
[pip3] pyzmq==27.2.0
[pip3] torch==2.13.0
[pip3] torchaudio==2.11.0
[pip3] torchvision==0.28.0
[pip3] transformers==5.16.1
[pip3] triton==3.7.1
[conda] Could not collect

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.28.1rc1.dev383+g8a0a7ee40 (git sha: 8a0a7ee40)
vLLM Build Flags:
  CUDA Archs: Not Set; ROCm: Disabled; XPU: Disabled
GPU Topology:
  GPU0  CPU Affinity  NUMA Affinity  GPU NUMA ID
GPU0   X   0-47  0    N/A

==============================
     Environment Variables
==============================
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_<user>

Describe the bug

_warn_if_unannotated_eagle_mamba (vllm/v1/core/kv_cache_utils.py) is gated on use_eagle(), but the flag-all fallbacks it describes — KVCacheCoordinator.__init__, SchedulerOffloadConfig.from_spec, MooncakeStoreCoordinator._verify_and_split_kv_cache_groups — are gated on use_eagle_block_drop(). With --speculative-config '{..., "disable_eagle_block_drop": true}' (#53388), _annotate_eagle_groups returns before flagging anything, so the "already flagged" short-circuit never fires and the warning is emitted for every model with a Mamba KV group, saying "prefix-cache reuse across requests will be disabled" while nothing is flagged and reuse is untouched.

Observed on main @ 8a0a7ee, Qwen3.6-35B-A3B W4A16 (GDN + full attention, resolved block 1056), synthetic prompts, single run: MTP k=1 with the default block drop → 0 prefix-cache hits (the expected trailing-block drop, #38182); MTP + disable_eagle_block_drop → 22,176 hits / 72,992 queried tokens, identical to the no-speculation baseline, acceptance unchanged (59.4% → 59.4%). The warning text is identical in both runs.

Related: #55390 annotates the MTP draft group positionally so the default-drop case stops warning, but keeps _annotate_eagle_groups gated on use_eagle_block_drop(), so the disable_eagle_block_drop case above still warns after it; #52771 removes the offloading fallback; #54163 narrows use_eagle_block_drop() for DFlash/DSpark and would share the fix's gate. #55357 (a different problem) also records this warning next to a 96% prefix-cache hit rate on a qwen4_exp hybrid.

Expected

Gate the warning on the same predicate as the fallbacks (use_eagle_block_drop()), and let the scheduler's existing "block dropping is disabled" warning say what is still true (the trailing block stays eligible for cross-request hits and external offload). Log text only; no behaviour change. I have the patch and a parametrized test ready.

(Investigation and the runs above were done with Claude's assistance; I reviewed the analysis and the numbers.)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions