[Bug]: Async chunk requests can stay in WAITING_FOR_CHUNK indefinitely if upstream chunk never arrives

[Ronnie-Rui]

Your current environment

The output of python collect_env.py

OS: Ubuntu 22.04.5 LTS (x86_64)
Python: 3.12.11
PyTorch: 2.11.0+cu130
CUDA available: True
GPU: NVIDIA GeForce RTX 4090

Your code version

Details

vLLM Version: 0.20.0

Describe the bug

In async chunk mode, downstream requests can remain in RequestStatus.WAITING_FOR_CHUNK indefinitely if the expected upstream chunk is never produced or never becomes visible through the connector.

The relevant path is OmniChunkTransferAdapter.process_pending_chunks():

• downstream stage calls load_async(request);
• request status is changed to WAITING_FOR_CHUNK;
• request is temporarily removed from the scheduler queue;
• restore_queues() puts it back after scheduling;
• if _poll_single_request() never observes the expected chunk, the request keeps cycling in WAITING_FOR_CHUNK.

Currently I could not find a timeout or cleanup path for this state. This means failures in the upstream stage, connector write failures, dropped terminal chunks, or a custom payload extraction path returning no payload can turn into a hanging streaming request instead of a visible request failure.

Expected behavior:

If a request waits too long for an async chunk, vllm-omni should eventually fail/cleanup the request instead of leaving it in WAITING_FOR_CHUNK forever.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[amy-why-3459]

Yes, we have an RFC. If you're interested, could you help implement this feature? #1297

[Ronnie-Rui]

Yes, we have an RFC. If you're interested, could you help implement this feature? #1297

I'd be happy to help! I'll implement this feature as soon as possible.

[Ronnie-Rui]

@hsliuustc0106 @amy-why-3459 I plan to address this issue incrementally: start with a low-risk mitigation to prevent hangs, then gradually improve error propagation and reliability.

Short term: add a configurable timeout while keeping current behavior as default. If explicitly set, the request will fail when no chunk is received within the timeout. The downside is that it only prevents indefinite hangs and may still kill slow requests.

Mid term: address the limitation by adding terminal error / cancel / heartbeat signals, so downstream can know whether upstream failed, was cancelled, is still processing, or has made no progress.

Long term: build cross-stage reliability with request graph, coordinator liveness, connector ACK/lease/TTL, and outcome metrics, so failures can be propagated, diagnosed, and observed.

Could you take a look and share your thoughts on this direction?