KV cache connector hangs with Eagle speculative decoding when rewind crosses block boundary

[michaelfeil]

Description

When using a KV cache connector with Eagle speculative decoding, the executor can hang during get_finished. The hang occurs when rejected draft tokens cause rewindKVCache to free a block that crosses a block boundary.

Root cause

KvCacheConnectorSchedulerOutputRequest.block_ids only grows monotonically — it is never trimmed after rewind. When rewindKVCache calls releaseLastBlock (which happens when the rewind crosses a block boundary), the connector retains stale references to the now-freed block ids.

This causes get_finished's cross-rank mpi_allgather + set.intersection to never complete, because the freed block ids are still present in the connector's per-request bookkeeping.

Repro

• Backend: PyTorch
• Spec decoding: Eagle (non-linear tree)
• KV cache connector: enabled (e.g. KVBM)
• Observe: hang after the first iteration where Eagle rejects draft tokens and the rewind crosses a block boundary

Suggested fix

Add an on_rewind hook to the connector that trims block_ids (and related per-request state) to match the post-rewind live cache indices after each rewindKVCache call.