Working WebRTC Streaming for OmniGibson (Replaces Deprecated Streamclient)#1916
Open
yerbymatey wants to merge 8 commits intoStanfordVL:mainfrom
Open
Working WebRTC Streaming for OmniGibson (Replaces Deprecated Streamclient)#1916yerbymatey wants to merge 8 commits intoStanfordVL:mainfrom
yerbymatey wants to merge 8 commits intoStanfordVL:mainfrom
Conversation
for more information, see https://pre-commit.ci
|
Thanks for this PR! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WebRTC streaming supporting ICE/SDP + TCP only WS streaming to cover the now deprecated
omni.services.streamclient.webrtcand their built in js viewer for local machines and cloud instances. Setup is mainly env driven (grab and set public ip if you have one, any mapped ports, pick between streaming modes). There's a small setup script to grab Nvidia's WebRTC streaming library since we can't ship it directly :/ Setup guide is here.Kit directly serves a static site with some custom extensions to help out with session coordination and some stopgaps for Nvidia's event plumbing.
Tested on two Ubuntu 22.04 boxes under v different network setups:
Haven’t tested an unprivileged Runpod-style Docker image yet with TCP only and proxied HTTPS, but the design should survive it since the backend URL can be pulled from origin headers and the SPA routing lives in the web extension. TCP only streams are view-only rn, so keyboard or mouse events don't get registered.
Note on extension load + enabling: The custom exts use a lazy ExtensionManager to set the extension path during kit runtime instead of the app dir copy pattern y'all had. Ext loading is kind of in this weird halfway place right now where
isaacsim.core.utils.extension.enable_extension()calls will work for ext mgmt, but will sometimes fall short for recursive dep solving.Tracking Nvidia's doc drift to swap out and extend functionality for exts that still worked with Isaac Sim 4.5 + Kit 106.5 was absolute dogwater but streaming should be way more straightforward once y'all bump to Isaac Sim 5.0 lol