Adding Unique IDs to WebRTC Frames in OME for Client Synchronization

[XiphosF]

Hi, my OME server is receiving an RTMP video stream and publishing it normally via WebRTC at ws://OME_SERVER_IP:3333/app/stream. I'd like to add a unique ID to each frame of the WebRTC stream to synchronize multiple clients. I considered using the WebRTC Data Channel if the synchronization with the video is accurate enough (not sure), or embedding SEI messages in the H.264 stream. Are there any ways to use the WebRTC Data Channel or integrate custom SEI messages with OME?

[getroot]

Is the player a native player, not a browser? What I don't understand is how to get client sync by putting a unique id in the sei. You can send a stream with the sei inserted from the RTMP encoder to OME. Since OME doesn't truncate the sei, you can just take it out of the player and use it.

It's very hard to get frame-by-frame client sync. Even if you do it in absolute time, the wall clock of each device is slightly different, in the hundreds of ms range.

If you do webrtc ultra-low latency streaming with a latency of 4-6 frames, you can roughly get client sync.

[getroot]

So, should OvenPlayer delay WebRTC playback a bit until it gets the result from Cloud Computing Server? There is no way to do that. Are you trying to use SEI for this? I don't see the whole picture yet.

[XiphosF]

Yes, that was the idea. The latency introduced by the computing server should be relatively short (~200–300 ms). So, I wanted to buffer a few frames in the player to ensure the received frames have their corresponding analysis results before displaying. The idea to have exact correspondance between analysis results and frames in the player was by using an ID for each frame in the stream with SEI at the OME level.

[getroot]

That's what I don't understand. As far as I know, it's impossible to buffer a few frames in a WebRTC player. The browser doesn't give the user permission to buffer WebRTC. Do you know of an API that I don't know about?

[bchah]

If the delay introduced by the computing server is more or less "fixed" at 300ms, what if we can leverage OME's existing features to create the result you need?

My thinking is:

• Send in your original stream to Application "A", which is read by your computing server for analysis
• Use the Push Publisher to push the stream to Application "B" on the same OME server
• In Application "B", use the secret-but-maybe-works Publisher flag of <DelayBufferTimeMs>300</DelayBufferTimeMs> to delay the outbound stream
• Send your metadata to the browser viewers (who will view the stream on Application "B", not "A")

(The "DelayBufferTimeMs" is not officially supported in OME open-source by the way - for official support consider a purchase of OvenMediaEngine Enterprise)

[getroot]

@bchah
Good idea. Ideally, if a fixed delay is sufficient, instead of using push publish, you can create two apps and replicate the stream using a mux provider in one of the apps.

[getroot]

@XiphosF
I realized that there is a possibility using SEI and the WebRTC API below. I now know that the API below is blocking. If you develop a special player using the API below, you can achieve your goal.

https://developer.mozilla.org/en-US/docs/Web/API/WebRTC_API/Using_Encoded_Transforms

And you probably need to insert SEI NAL Units per frame into the original video at the encoder level. (OME OpenSource does not provide such API.)