Motivation.
Continuous video understanding - a model that watches a live camera and emits
a short reply per frame for hours (captioning, alerting, counting objects over
time) - has no good shape in vLLM today.
Each frame's reply has to be produced in the context of the frames before it,
and with request-at-a-time serving the only way to supply that context is to
resend it. That leaves two options, and both fail:
- One request per frame carrying the history. Every frame re-uploads and
re-processes everything the model has already seen. Work per frame grows with
the age of the stream; the feed falls behind within minutes. Bounding the
resent window caps the growth but pays a full window prefill on every frame.
- One long-lived request that never ends. Two resources then grow without
bound: GPU memory, since everything the model has seen stays resident for the
life of the request (video is token-hungry - an hour at 2 fps is on the order
of a million tokens of context); and sequence length, since the stream crosses
the model's trained horizon within minutes to hours and output quality
degrades past it.
vLLM already has the mechanism the second option needs. Streaming input
(#28973), introduced to power the realtime audio API and described in
Streaming Requests & Realtime API in vLLM,
keeps one long-lived request alive while input arrives in chunks: context stays
live in GPU memory between chunks, so appending a chunk costs only that chunk.
That solves cost-per-frame.
What it does not solve is what happens when a session runs for hours. The
session's KV grows with every chunk and is released only when the session ends,
and its position counter keeps climbing. The blog post says as much:
"additional care must be taken to avoid holding sessions open as they will be
blocking the corresponding memory from being used by other requests…
Currently, vLLM will not preempt "idle" streaming input sessions - this
behaviour will be improved in a future update."
For audio-conversation lengths that is tolerable. For an always-on camera it is
the whole problem. Sessions exist; memory management for sessions does not.
This is not hypothetical demand. #50570 (RTSP live-stream captioning via a
DeepStream/NVDEC backend) brings continuous streams into vLLM at the ingest
layer and emits one caption per decoded segment; how context is carried across
segments is left open. Ingest and session memory are the two halves of the same
feature: that PR gets frames in, this proposal lets the model keep watching
without unbounded memory. We would like to converge with it rather than build a
second path.
The design is a serving-side adaptation of
StreamingVLM (bounded KV, windowed
inference over an unbounded stream) that fits vLLM's existing scheduler and KV
manager - no attention-kernel changes and no model changes.
Proposed Change.
Give a streaming-input session a constant memory footprint no matter how long
it runs. The design mirrors how a person watches a feed: keep the instructions
and the recent past sharp, let old frames fade, occasionally consolidate.
- A sliding window over the video history. Each session keeps its task
instructions (pinned, never dropped), the most recent N frames, and the text
the model has produced. When the window is full, the KV of the oldest frame is
freed in place while the session keeps running - no restart, and nothing
that survives is recomputed.
- Survivors keep their positions. Dropping old frames leaves gaps in the
position sequence, and we deliberately do not renumber what remains:
renumbering means rewriting cached attention state for every surviving token,
which is expensive and (as discussed in other threads) not obviously correct.
Gaps are benign for the multimodal-RoPE model class we target, and we measured
it (below).
- A fresh start before the horizon. Position numbers still creep upward. As a
session approaches the model's trained sequence limit it consolidates: the
surviving window is replayed once as a compact prompt starting again at
position 0, and the session continues. This is rare (hours apart) and cheap -
surviving frames' image features come from the encoder cache, so the vision
tower is not re-run.
- A minimal session API to drive it: create a session → push a frame (returns
that frame's reply) → close.
What this touches, in engine terms:
- a KV-manager primitive to free a block-aligned token range of a live
request (today blocks are freed only when a request ends), with the freed
blocks removed from the prefix-cache index so their content can never be
served to anyone else;
- scheduler-side retention/eviction/consolidation driven by per-session
parameters, plus the position bookkeeping that keeps the worker's view in
lockstep with the block table;
- a thin frontend for the session lifecycle.
Requests that do not set the per-session parameters take none of these paths,
and the feature is off unless enabled at server start.
Measured: constant, lower per-frame cost. Continuous dashcam captioning,
one short JSON reply (caption + alert label) per frame, Cosmos3 Nano (bf16),
single H100-80GB, N = 8-frame window, identical sampling/pixel budget/clip (128 frames @ 2 fps). The only
variable is the access pattern: a streaming session versus the best a
request-at-a-time server can do (resend the last N frames plus prior replies to
/v1/chat/completions every frame).
| Per frame (steady state) |
Streaming session |
Re-send baseline |
| Time-to-first-token (prefill) |
32 ms |
57 ms (1.8× / +78%) |
| Reply tokens decoded |
~36 |
~39 |
| Total latency |
0.315 s |
0.322 s |
TTFT is the metric that isolates what the access pattern changes: the session
prefills only the new frame (~25 tokens) instead of the whole resent window
(862 tokens). That cost is flat for the session while it grows for the
baseline with window size, resolution and retained history.
Measured: eviction does not cost accuracy. We measured accuracy on synthetic generated counting task (generate a sequence of numbers and requiring the model to remember it previous counts) and it is identical to a model that receives it's own history in full at every step. Furthermore, on a captioning task of car crash footage we did not detect a measurable drop in quality. However, there is no public / open-source benchmark aiming at exactly this task and it's a known limitation.
Feedback Period.
Two weeks.
CC List.
@njhill @ywang96 @DarkLight1337 @joshuadeng @patrickvonplaten @ViranjanPagar @Isotr0py
Any Other Things.
All of this is implemented and has been running for a while. We have had single sessions run continuously for 24 hours at a steady frame rate with flat memory: hundreds of thousands of frames through one
session, many consolidations along the way, no restarts and no drift in latency.
We have used it on a few different kinds of footage: dashcam, traffic and
intersection monitoring, indoor scenes; and for different tasks: per-frame
captioning, event alerting, and counting that has to keep state across the
window boundary, which is the case naive re-sending gets wrong. It runs on more
than one multimodal-RoPE model family, at bf16 and fp8.
Before submitting a new issue...
Motivation.
Continuous video understanding - a model that watches a live camera and emits
a short reply per frame for hours (captioning, alerting, counting objects over
time) - has no good shape in vLLM today.
Each frame's reply has to be produced in the context of the frames before it,
and with request-at-a-time serving the only way to supply that context is to
resend it. That leaves two options, and both fail:
re-processes everything the model has already seen. Work per frame grows with
the age of the stream; the feed falls behind within minutes. Bounding the
resent window caps the growth but pays a full window prefill on every frame.
bound: GPU memory, since everything the model has seen stays resident for the
life of the request (video is token-hungry - an hour at 2 fps is on the order
of a million tokens of context); and sequence length, since the stream crosses
the model's trained horizon within minutes to hours and output quality
degrades past it.
vLLM already has the mechanism the second option needs. Streaming input
(#28973), introduced to power the realtime audio API and described in
Streaming Requests & Realtime API in vLLM,
keeps one long-lived request alive while input arrives in chunks: context stays
live in GPU memory between chunks, so appending a chunk costs only that chunk.
That solves cost-per-frame.
What it does not solve is what happens when a session runs for hours. The
session's KV grows with every chunk and is released only when the session ends,
and its position counter keeps climbing. The blog post says as much:
For audio-conversation lengths that is tolerable. For an always-on camera it is
the whole problem. Sessions exist; memory management for sessions does not.
This is not hypothetical demand. #50570 (RTSP live-stream captioning via a
DeepStream/NVDEC backend) brings continuous streams into vLLM at the ingest
layer and emits one caption per decoded segment; how context is carried across
segments is left open. Ingest and session memory are the two halves of the same
feature: that PR gets frames in, this proposal lets the model keep watching
without unbounded memory. We would like to converge with it rather than build a
second path.
The design is a serving-side adaptation of
StreamingVLM (bounded KV, windowed
inference over an unbounded stream) that fits vLLM's existing scheduler and KV
manager - no attention-kernel changes and no model changes.
Proposed Change.
Give a streaming-input session a constant memory footprint no matter how long
it runs. The design mirrors how a person watches a feed: keep the instructions
and the recent past sharp, let old frames fade, occasionally consolidate.
instructions (pinned, never dropped), the most recent N frames, and the text
the model has produced. When the window is full, the KV of the oldest frame is
freed in place while the session keeps running - no restart, and nothing
that survives is recomputed.
position sequence, and we deliberately do not renumber what remains:
renumbering means rewriting cached attention state for every surviving token,
which is expensive and (as discussed in other threads) not obviously correct.
Gaps are benign for the multimodal-RoPE model class we target, and we measured
it (below).
session approaches the model's trained sequence limit it consolidates: the
surviving window is replayed once as a compact prompt starting again at
position 0, and the session continues. This is rare (hours apart) and cheap -
surviving frames' image features come from the encoder cache, so the vision
tower is not re-run.
that frame's reply) → close.
What this touches, in engine terms:
request (today blocks are freed only when a request ends), with the freed
blocks removed from the prefix-cache index so their content can never be
served to anyone else;
parameters, plus the position bookkeeping that keeps the worker's view in
lockstep with the block table;
Requests that do not set the per-session parameters take none of these paths,
and the feature is off unless enabled at server start.
Measured: constant, lower per-frame cost. Continuous dashcam captioning,
one short JSON reply (caption + alert label) per frame, Cosmos3 Nano (bf16),
single H100-80GB, N = 8-frame window, identical sampling/pixel budget/clip (128 frames @ 2 fps). The only
variable is the access pattern: a streaming session versus the best a
request-at-a-time server can do (resend the last N frames plus prior replies to
/v1/chat/completionsevery frame).TTFT is the metric that isolates what the access pattern changes: the session
prefills only the new frame (~25 tokens) instead of the whole resent window
(862 tokens). That cost is flat for the session while it grows for the
baseline with window size, resolution and retained history.
Measured: eviction does not cost accuracy. We measured accuracy on synthetic generated counting task (generate a sequence of numbers and requiring the model to remember it previous counts) and it is identical to a model that receives it's own history in full at every step. Furthermore, on a captioning task of car crash footage we did not detect a measurable drop in quality. However, there is no public / open-source benchmark aiming at exactly this task and it's a known limitation.
Feedback Period.
Two weeks.
CC List.
@njhill @ywang96 @DarkLight1337 @joshuadeng @patrickvonplaten @ViranjanPagar @Isotr0py
Any Other Things.
All of this is implemented and has been running for a while. We have had single sessions run continuously for 24 hours at a steady frame rate with flat memory: hundreds of thousands of frames through one
session, many consolidations along the way, no restarts and no drift in latency.
We have used it on a few different kinds of footage: dashcam, traffic and
intersection monitoring, indoor scenes; and for different tasks: per-frame
captioning, event alerting, and counting that has to keep state across the
window boundary, which is the case naive re-sending gets wrong. It runs on more
than one multimodal-RoPE model family, at bf16 and fp8.
Before submitting a new issue...