Skip to content

Working WebRTC Streaming for OmniGibson (Replaces Deprecated Streamclient)#1916

Open
yerbymatey wants to merge 8 commits intoStanfordVL:mainfrom
yerbymatey:feat/webrtc-streaming
Open

Working WebRTC Streaming for OmniGibson (Replaces Deprecated Streamclient)#1916
yerbymatey wants to merge 8 commits intoStanfordVL:mainfrom
yerbymatey:feat/webrtc-streaming

Conversation

@yerbymatey
Copy link

WebRTC streaming supporting ICE/SDP + TCP only WS streaming to cover the now deprecated omni.services.streamclient.webrtc and their built in js viewer for local machines and cloud instances. Setup is mainly env driven (grab and set public ip if you have one, any mapped ports, pick between streaming modes). There's a small setup script to grab Nvidia's WebRTC streaming library since we can't ship it directly :/ Setup guide is here.

Kit directly serves a static site with some custom extensions to help out with session coordination and some stopgaps for Nvidia's event plumbing.

Tested on two Ubuntu 22.04 boxes under v different network setups:

  • local + Tailscale
  • local + external IP + port-forwarding
  • local + firewalled UDP (to test TCP only websocket)
  • provider full VM (Vast.ai) with mapped UDP/TCP

Haven’t tested an unprivileged Runpod-style Docker image yet with TCP only and proxied HTTPS, but the design should survive it since the backend URL can be pulled from origin headers and the SPA routing lives in the web extension. TCP only streams are view-only rn, so keyboard or mouse events don't get registered.

Note on extension load + enabling: The custom exts use a lazy ExtensionManager to set the extension path during kit runtime instead of the app dir copy pattern y'all had. Ext loading is kind of in this weird halfway place right now where isaacsim.core.utils.extension.enable_extension() calls will work for ext mgmt, but will sometimes fall short for recursive dep solving.

Tracking Nvidia's doc drift to swap out and extend functionality for exts that still worked with Isaac Sim 4.5 + Kit 106.5 was absolute dogwater but streaming should be way more straightforward once y'all bump to Isaac Sim 5.0 lol

@hang-yin hang-yin requested a review from cgokmen December 16, 2025 18:54
@Seyeon321
Copy link

Thanks for this PR!
I tried it while setting up remote visualization, and it clarified a lot of the issues I was running into with the deprecated streaming path.
Really appreciate the work :) !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants