Skip to content

Cancelled room.connect() (asyncio.wait_for timeout in Python bindings) leaks UDP sockets; accumulation ends in FFI panic #1340

Description

@natkingsize2

Summary

When a Python caller wraps room.connect() in asyncio.wait_for(..., timeout=X) and the timeout fires
before the connection handshake completes, room.disconnect() called afterward in a finally block has
no effect on the in-flight connection attempt. The Rust FFI side keeps a room handle alive and continues
attempting the handshake on a background thread, unaware that the Python future was cancelled. Each such
cancelled attempt leaks UDP (ICE) sockets that are never reclaimed. Repeated cancellations without an
intervening successful connect accumulate leaked sockets linearly, and after enough of them the FFI layer
panics with:

livekit_ffi::server::room - timed out waiting for ReadyForRoomEventRequest after ConnectCallback (room_handle=N)
FFI Panic: invalid request: timed out waiting for ReadyForRoomEventRequest after ConnectCallback (room_handle=N)

This panic crashes the entire Python process hosting the FFI (not just the room/connection).

Environment

  • SDK: livekit (Python bindings) 1.1.13 · livekit-api 1.2.0 · livekit-protocol 1.1.21
  • OS: macOS (Darwin) — reproduced on a dev machine, separate from any production room
  • Reproduced against a locally-run livekit-server (isolated dev instance, ports 17880/17881, bind
    127.0.0.1 only) — not a hosted/cloud LiveKit instance

Repro steps (100% reproducible)

  1. Start a local livekit-server on an isolated port/bind (see attached dev-livekit.yaml).
  2. In a loop, call:
    room = rtc.Room()
    try:
        await asyncio.wait_for(room.connect(url, token, options=...), timeout=0.05)  # deliberately shorter than ICE handshake
    except asyncio.TimeoutError:
        pass
    finally:
        await room.disconnect()
    with a timeout short enough that the handshake cannot possibly complete in time (e.g. 0.05s).
  3. Measure open UDP file descriptors before/after each cycle (e.g. lsof -p <pid> | grep UDP | wc -l).

Observed

  • Single cancelled connect: 12 UDP sockets remain open immediately after disconnect() returns (count
    scales with local network interface count × peer-connection count; on the repro machine: 3 interfaces ×
    ~2 peer connections × ~2 candidates ≈ 12).

  • 6 cancelled connects back-to-back (no successful connect in between): sockets accumulate roughly
    linearly with no sign of self-recovery in the observed window:

    after timeout # udp_fds
    1 12
    2 28
    3 44
    4 44
    5 56
    6 84
  • On the 6th burst the process then crashes with the FFI panic quoted above.

  • Control test: normal connect→disconnect cycles that are allowed to complete (not cancelled) do
    not leak — UDP fd count returns to baseline every time, across both delayed and rapid-fire cycles.
    This isolates the leak specifically to the cancel-mid-handshake path, not reconnection frequency in
    general.

Client-side code audited (for context, not part of the ask)

Our calling code already wraps room.connect() in try/finally with room.disconnect() unconditionally
called, and has no other unclosed track/stream handles. From the client API surface, there does not appear
to be a way to tell the FFI layer "abandon this specific in-flight connect attempt" — disconnect() only
tears down a room that finished connecting.

Ask

  • Is a cancelled/timed-out connect() expected to leak the sockets it opened, or is this a bug in socket
    cleanup on the abandoned handshake path?
  • Is there a supported way to cancel an in-flight connect() attempt from the Python bindings that also
    tears down the FFI-side room handle and its sockets?
  • Is the eventual panic (timed out waiting for ReadyForRoomEventRequest after ConnectCallback) a known
    failure mode of leaving an abandoned room handle running in the background, and is it expected to crash
    the whole host process rather than fail contained to that handle?

Attachments available on request

Minimal repro scripts (exp1exp4) covering: clean reconnect loops (control, no leak), single
cancelled-connect leak, and the burst-accumulation + panic repro with captured stderr.


หมายเหตุภายใน (ไม่ใช่ส่วนของ issue)

  • Mitigation ที่ทำได้จากฝั่งเราโดยไม่รอ SDK patch (ตัวเลือก ไม่ใช่คำสั่ง): ยืด CONNECT_TIMEOUT_S,
    เลิกครอบ wait_for ถ้า SDK มี timeout ในตัวอยู่แล้ว (ยังไม่ยืนยัน), ใช้ fd-count เป็นสัญญาณเตือนเชิงรุกแล้ว
    self-restart แบบควบคุม (ผูกกับใบ watcher-exit-on-N-failures ที่ iris กำลังทำอยู่พอดี — เป็นชั้นป้องกัน
    ที่ตรงจุดที่สุดเพราะไม่ต้องรอ SDK ตอบเลย)
  • ค้นเว็บสั้นๆ (2026-08-19) ไม่พบ public report ที่ตรงกับข้อความ panic นี้เป๊ะ — อาจเป็นเคสใหม่

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions