fix: add asyncio.sleep(0) to SSE generator for real-time streaming

[lorry01]

Problem

The SSE endpoint /api/vanna/v2/chat_sse buffers responses instead of delivering them in real-time. When testing with browser DevTools, all chunks arrive in 2-3 batches rather than streaming continuously.

Expected behavior: Chunks should arrive one-by-one as they are generated.

Actual behavior: Chunks are batched together and delivered in 2-3 large groups, losing the real-time streaming effect.

Root Cause

In src/vanna/servers/fastapi/routes.py, the generate() async generator yields data but doesn't give the event loop a chance to process network I/O between yields. This causes data to accumulate in the buffer until the generator pauses (waiting for LLM response) or completes.

Solution

Add await asyncio.sleep(0) after each yield statement. This yields control back to the event loop, allowing it to process pending network I/O and send data to the client immediately.

async def generate() -> AsyncGenerator[str, None]:
"""Generate SSE stream."""
try:
async for chunk in chat_handler.handle_stream(chat_request):
chunk_json = chunk.model_dump_json()
yield f"data: {chunk_json}\n\n"
await asyncio.sleep(0) # Yield control to event loop for immediate transmission
yield "data: [DONE]\n\n"
except Exception as e:
# ...## Why await asyncio.sleep(0) works

In Python's asyncio, await asyncio.sleep(0) is a common idiom to:

1. Yield control to the event loop
2. Allow other pending tasks (including network I/O) to execute
3. Return control to the coroutine immediately

Without this, the generator runs synchronously between yield statements, and the event loop doesn't get a chance to flush the network buffer until the generator blocks on an actual I/O operation.

Testing

Environment:

• Python 3.10
• FastAPI + Uvicorn (no reverse proxy)
• Vanna 2.0.1

Before fix:

• Browser DevTools shows all SSE messages arriving in 2-3 batches at the same timestamp

After fix:

• SSE messages arrive continuously with distinct timestamps, achieving true real-time streaming

Related

This is a common issue with FastAPI StreamingResponse. Similar solutions are documented in:

• Starlette StreamingResponse discussion
• FastAPI streaming best practices