Skip to content

Fix websocket emit-queue starvation when emit() returns None - #430

Open
dr-regier wants to merge 1 commit into
gradio-app:mainfrom
dr-regier:fix-emit-queue-starvation
Open

Fix websocket emit-queue starvation when emit() returns None#430
dr-regier wants to merge 1 commit into
gradio-app:mainfrom
dr-regier:fix-emit-queue-starvation

Conversation

@dr-regier

Copy link
Copy Markdown

Problem

WebSocketHandler._emit_to_queue enqueues whatever emit() returns with no guard:

while not self.quit.is_set():
    output = await self.stream_handler.emit()
    self.queue.put_nowait(output)

When a handler is idle, emit() returns None instantly on every call. With an unbounded queue and no yield on the idle path, this spins thousands of times/sec enqueuing None, floods the queue (observed ~164k depth), and starves the consumer _emit_loop so it isn't scheduled to send the frame already at the head of the queue.

Real-world impact (telephony): up to ~18s of intermittent dead air before an already-generated greeting reached the caller. Related: #203, where the same starvation is worked around by adding await asyncio.sleep(0.01) inside the user's emit().

Fix

On the idle path (output is None), back off 20ms and skip the enqueue:

if output is None:
    await asyncio.sleep(0.02)
    continue
self.queue.put_nowait(output)

Why this is safe

_emit_loop already skips None outputs, so enqueuing None never produced a frame - it only created work and event-loop contention. This removes wasted work and yields the loop to the consumer on idle; no downstream behavior changes.

Validation

On the live phone path, first-frame send dropped from ~18.6s to ~0.1-0.4s with queue depth ~0 on every call. Tested on 0.0.33; the affected code is identical on main.

Notes / alternatives

  • 20ms is a conservative idle backoff (negligible added latency vs the starvation it prevents). Open to tuning.
  • Alternatives considered: bounding the queue, or documenting the emit() await requirement instead of handling it here. Went with the loop-side guard because it's minimal and fixes the existing AsyncStreamHandler breaks when no await is used in emit() #203 workaround at the source. Happy to reshape.

`WebSocketHandler._emit_to_queue` enqueued whatever `emit()` returned with
no guard. When a handler is idle, `emit()` returns `None` instantly on every
call, so with an unbounded queue and no yield on the idle path the loop spins
thousands of times/sec enqueuing `None`. This floods the queue and starves the
consumer `_emit_loop`, delaying frames already at the head of the queue.

Real-world impact (telephony): up to ~18s of intermittent dead air before an
already-generated greeting reached the caller.

On the idle path, back off 20ms and skip the enqueue. `_emit_loop` already
skips `None` outputs, so enqueuing `None` never produced a frame - this only
removes wasted work and yields the loop to the consumer; no downstream
behavior changes.

Related: gradio-app#203
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant