FFmpeg: Add Whip Muxer support for subsecond latency streaming

[winlinvip]

This PR is updated by #18.

This PR has been squashed into a single commit on the patch/whip/v1 branch for easier submission to the FFmpeg community for code review.

Note: Due to the high level of interest in this patch, which has resulted in numerous reviews and discussions, including feedback from ffmpeg developers via email, we will be tracking all of these interactions. To facilitate this process, we will be using GitHub Discussions or Discord: Real-time Broadcast or Discord: SRS. Please feel free to use this platform to discuss the patch or any other questions or suggestions you may have.

WHIP stands for the WebRTC-HTTP ingestion protocol, which is a sub-second streaming protocol designed by encoders and publishers. It is widely supported by various tools and media servers, allowing it to interact with other WebRTC clients, ingest streams to media servers, and is compatible with all modern browsers.

Unfortunately, most WHIP implementations are highly complex and require modern C++11 or C++14, or RUST. This complexity makes it impossible to integrate with FFmpeg, which requires C.

However, if FFmpeg were to incorporate WHIP support, it could be a game-changer for both FFmpeg and the WebRTC ecosystem, particularly for certain IoT or small devices that are too small to run modern languages but can run FFmpeg.

To meet FFmpeg's requirements, this PR contains just C code. And we have rewritten the WHIP and WebRTC protocol stack using only around 3k lines of C code.

Content

• Usage How do I use this patch?
• Usage: FFmpeg + SRS How do I use SRS as a WHIP server for FFmpeg?
• Usage: FFmpeg + Janus How do I use Janus as a WHIP server for FFmpeg?
• Usage: FFmpeg + Pion How do I use Pion as a WHIP server for FFmpeg?
• Usage: FFmpeg + Millicast How do I use Millicast as a WHIP server for FFmpeg?
• Usage: FFmpeg + TRTC How do I use TRTC's WHIP cloud service for FFmpeg?
• Usage: FFmpeg + Cloudflare How do I use Cloudflare's WHIP cloud service for FFmpeg?
• Known Issues Regarding known issues and what has been fixed.
• Latency What's the process for measuring latency? Typically, it's around 150ms.
• OpenSSL What's the supported version of OpenSSL, and how do I build it?
• Load Certificate File How to import a DTLS certificate and private key from a file?
• Authorization How can I authorize using an HTTP header?
• Contributors About the authors, contributors, and reviewers for this patch.
• Links Useful links about this patch.

Usage

Please select a open-source WHIP server to work with FFmpeg.

• Usage: FFmpeg + Janus
• Usage: FFmpeg + Pion
• Usage: FFmpeg + Millicast
• Usage: FFmpeg + SRS

If you encounter any issues or get stuck, please leave us a message on Discord.

Note: We've included Janus, Pion, Millicast, and SRS as examples of WHIP servers you can use. However, there are many other options available, such as Galene, Deadsfu, and more. For more information, please see the ietf112-hackathon-whip.pdf.

You also have the option to select a WHIP cloud service, which is typically provided by video cloud service providers.

• Usage: FFmpeg + TRTC
• Usage: FFmpeg + Cloudflare

If you encounter any issues or get stuck, please leave us a message on Discord.

Usage: FFmpeg + SRS

To enable FFmpeg to publish a stream, SRS can be used as the WHIP server. It is recommended to use docker.

ip="192.168.3.85" 
docker run --rm -it -p 1935:1935 -p 1985:1985 -p 8080:8080 \
    --env CANDIDATE=$ip -p 8000:8000/udp \
    ossrs/srs:5 ./objs/srs -c conf/rtc2rtmp.conf

Alternatively, you can build SRS from its source code.

cd ~/git
git clone -b develop https://github.com/ossrs/srs.git
cd srs/trunk
./configure
make

# After building SRS, you may run it using a configuration file.
cd ~/git/srs/trunk
./objs/srs -c conf/rtc2rtmp.conf

Note: Please upgrade to SRS version 5.0.153 or higher, or 6.0.43 or higher.

To download the code and build FFmpeg, you can use the following command.

cd ~/git 
git clone -b feature/rtc-muxer https://github.com/winlinvip/ffmpeg-webrtc.git
cd ffmpeg-webrtc
./configure --enable-muxer=whip --enable-openssl --enable-version3 \
    --enable-libx264 --enable-gpl --enable-libopus
make -j10

Note: To enable DTLS handshake, OpenSSL is mandatory. Please install OpenSSL, for instance, brew install openssl, and then configure the environment by running export PKG_CONFIG_PATH="/usr/local/opt/openssl@3/lib/pkgconfig".

Note: For demonstration purposes, you can install libx264 by running brew install x264 and libopus by running brew install opus.

Although WebRTC has the capability to support x264 main and high profiles without B frames, it is advisable to use the baseline profile for better compatibility. If your stream doesn't have these codecs, you can transcode it using FFmpeg.

~/git/ffmpeg-webrtc/ffmpeg -re -f lavfi -i testsrc=size=1280x720 -f lavfi -i sine=frequency=440 -pix_fmt yuv420p \
    -vcodec libx264 -profile:v baseline -r 25 -g 50 -acodec libopus -ar 48000 -ac 2 \
    -f whip "http://localhost:1985/rtc/v1/whip/?app=live&stream=livestream"

# Or you can also capture your screen, and measure the end-to-end latency.
~/git/ffmpeg-webrtc/ffmpeg -f avfoundation -framerate 25 -pixel_format yuyv422 -i "2:0" \
    -vcodec libx264 -pix_fmt yuv420p -profile:v baseline -preset:v ultrafast \
    -b:v 800k -s 1024x576 -r 25 -g 50 -tune zerolatency -threads 1 -bf 0 \
    -acodec libopus -ar 48000 -ac 2 \
    -f whip "http://localhost:1985/rtc/v1/whip/?app=live&stream=livestream"

After publishing stream to SRS, you can play the WHIP stream in web browser such as Chrome, using srs-player.

• WHEP: http://localhost:1985/rtc/v1/whip-play/?app=live&stream=livestream

The image below shows that the latency is around 150ms.

The RTMP, HTTP-FLV, or HTTP-TS stream remuxed by SRS can be played using ffplay, VLC, or srs-player.

Usage: FFmpeg + Janus

We referred to WISH, WHIP and Janus: Part II to make it possible to use FFmpeg for publishing a stream to Janus via WHIP. We have also created a demo docker image janus-docker for quick testing.

git clone https://github.com/winlinvip/janus-docker.git
cd janus-docker

Initially, run the demo Docker that includes Janus server using the following command:

cd ~/git/janus-docker

ip="192.168.3.85" && sed -i '' "s/nat_1_1_mapping.*/nat_1_1_mapping=\"$ip\"/g" janus.jcfg

docker run --rm -it -p 8081:8080 -p 8188:8188 -p 8443:8443 -p 20000-20010:20000-20010/udp \
    -v $(pwd)/janus.jcfg:/usr/local/etc/janus/janus.jcfg \
    -v $(pwd)/janus.plugin.videoroom.jcfg:/usr/local/etc/janus/janus.plugin.videoroom.jcfg \
    -v $(pwd)/janus.transport.http.jcfg:/usr/local/etc/janus/janus.transport.http.jcfg \
    -v $(pwd)/janus.transport.websockets.jcfg:/usr/local/etc/janus/janus.transport.websockets.jcfg \
    -v $(pwd)/videoroomtest.js:/usr/local/share/janus/demos/videoroomtest.js \
    ossrs/janus:v1.0.12

Note: Kindly modify the IP address to your own IP address. You can use ifconfig or ipconfig to determine it.

After that, access the URL http://localhost:8081/videoroomtest.html?room=2345 in your browser to join the Janus room.

Next, download and run the Simple WHIP Server for Janus using the following command:

git clone https://github.com/meetecho/simple-whip-server.git
cd simple-whip-server
npm install
npm run build
npm run start

Generate a WHIP handler using curl, which will enable ffmpeg to join the same Janus room through WHIP:

curl -H 'Content-Type: application/json' -d '{"id": "abc123", "room": 2345}' \
    http://localhost:7080/whip/create

To download the code and build FFmpeg, you can use the following command.

cd ~/git 
git clone -b feature/rtc-muxer https://github.com/winlinvip/ffmpeg-webrtc.git
cd ffmpeg-webrtc
./configure --enable-muxer=whip --enable-openssl --enable-version3 \
    --enable-libx264 --enable-gpl --enable-libopus
make -j10

Note: To enable DTLS handshake, OpenSSL is mandatory. Please install OpenSSL, for instance, brew install openssl, and then configure the environment by running export PKG_CONFIG_PATH="/usr/local/opt/openssl@3/lib/pkgconfig".

Note: For demonstration purposes, you can install libx264 by running brew install x264 and libopus by running brew install opus.

Remark: Please use OpenSSL 1.0.2 and newer, because Janus requires DTLS 1.2.

Although WebRTC has the capability to support x264 main and high profiles without B frames, it is advisable to use the baseline profile for better compatibility. If your stream doesn't have these codecs, you can transcode it using FFmpeg.

~/git/ffmpeg-webrtc/ffmpeg -re -f lavfi -i testsrc=size=1280x720 -f lavfi -i sine=frequency=440 -pix_fmt yuv420p \
    -vcodec libx264 -profile:v baseline -r 25 -g 50 -acodec libopus -ar 48000 -ac 2 \
    -f whip 'http://localhost:7080/whip/endpoint/abc123'

# Or you can also capture your screen, and measure the end-to-end latency.
~/git/ffmpeg-webrtc/ffmpeg -f avfoundation -framerate 25 -pixel_format yuyv422 -i "2:0" \
    -vcodec libx264 -pix_fmt yuv420p -profile:v baseline -preset:v ultrafast \
    -b:v 800k -s 1024x576 -r 25 -g 50 -tune zerolatency -threads 1 -bf 0 \
    -acodec libopus -ar 48000 -ac 2 \
    -f whip 'http://localhost:7080/whip/endpoint/abc123'

After publishing the stream to the Janus room, you will be able to view it on the previously opened webpage. The image below shows that the latency is around 120ms.

Note: Sometimes, the latency might exceed 120ms, which could be due to my environment. We have reported an issue at here.

Usage: FFmpeg + Pion

To download the code and build FFmpeg, you can use the following command.

cd ~/git 
git clone -b feature/rtc-muxer https://github.com/winlinvip/ffmpeg-webrtc.git
cd ffmpeg-webrtc
./configure --enable-muxer=whip --enable-openssl --enable-version3 \
    --enable-libx264 --enable-gpl --enable-libopus
make -j10

Note: To enable DTLS handshake, OpenSSL is mandatory. Please install OpenSSL, for instance, brew install openssl, and then configure the environment by running export PKG_CONFIG_PATH="/usr/local/opt/openssl@3/lib/pkgconfig".

Note: For demonstration purposes, you can install libx264 by running brew install x264 and libopus by running brew install opus.

Although WebRTC has the capability to support x264 main and high profiles without B frames, it is advisable to use the baseline profile for better compatibility. If your stream doesn't have these codecs, you can transcode it using FFmpeg.

~/git/ffmpeg-webrtc/ffmpeg -re -f lavfi -i testsrc=size=1280x720 -f lavfi -i sine=frequency=440 -pix_fmt yuv420p \
    -vcodec libx264 -profile:v baseline -r 25 -g 50 -acodec libopus -ar 48000 -ac 2 \
    -f whip -authorization "seanTest" "https://b.siobud.com/api/whip"

# Or you can also capture your screen, and measure the end-to-end latency.
~/git/ffmpeg-webrtc/ffmpeg -f avfoundation -framerate 25 -pixel_format yuyv422 -i "2:0" \
    -vcodec libx264 -pix_fmt yuv420p -profile:v baseline -preset:v ultrafast \
    -b:v 800k -s 1024x576 -r 25 -g 50 -tune zerolatency -threads 1 -bf 0 \
    -acodec libopus -ar 48000 -ac 2 \
    -f whip -authorization "seanTest" "https://b.siobud.com/api/whip"

After publishing stream to pion, you can play the WebRTC stream in web browser such as Chrome

• WebRTC: https://b.siobud.com/seanTest

Usage: FFmpeg + Millicast

On the way.

Usage: FFmpeg + TRTC

On the way...

Usage: FFmpeg + Cloudflare

Here's a WHIP URL for testing: https://customer-wi9sckcs7uxt7lh4.cloudflarestream.com/67805a89d86d263b95f7056e5b212a0ek99a1686b042e85c3a5f86c38af9d9dad/webRTC/publish

And WHEP to play it back:
https://customer-wi9sckcs7uxt7lh4.cloudflarestream.com/99a1686b042e85c3a5f86c38af9d9dad/webRTC/play

You can use this player: https://wish.chens.link/watch

Known Issues

The current version has some known issues that we plan to fix in the future. You are welcome to help us fix them by sending a pull request.

The current known issues include:

• RTCP SR is needed to sync audio and video streams when converting RTC stream to RTMP or HLS. It helps recreate the timestamp. See WHIP: Convert OBS WHIP to RTMP failed for no RTCP SR packets. srs#3582
• Long First Screen Wait Time: Respond to the PLI request for quick decoding and rendering; otherwise, it takes a GOP's duration to display the first decoded image. A RTC encoder ought to have the capability to refresh the IDR frame when a new player joins or there is packet loss, as requested by the player through a PLI. When working within the ffmpeg framework, it's not possible to control the encoder in muxer. This means we cannot force an encoder to generate an IDR frame right away. One possible solution could be to cache the IDR frame or a GOP of frames in the RTC muxer and send it to the server upon receiving a PLI request. The player can then handle any latency issues that may arise.
• Multiple Slices Issue: If the -tune zerolatency option is enabled and threads>1, FFmpeg will encode the frame in multiple slices. This process can cause stuttering in Chrome's decoding, with only the IDR being able to be decoded. Both the IDR and P frames may be encoded to multiple slices and sent by multiple RTP packets with the same timestamp. RTC players, such as Chrome, may have issues decoding frames that have been encoded using multiple slices. As a result, Chrome may only decode some of the slices of the IDR frame and drop the P frames. If this issue occurs, it may appear as if the video player is stuttering and the decoded framerate will be equivalent to the GOP size.
• There is an issue with the timestamp of OPUS audio, especially when the input is an MP4 file. The timestamp deviates from the expected value of 960, causing Chrome to play the audio stream with noise. This problem can be replicated by transcoding a specific file into MP4 format and publishing it using the WHIP muxer. However, when directly transcoding and publishing through the WHIP muxer, the issue is not present, and the audio timestamp remains consistent. See comment for detail.
• The SDP functionality already exists in FFmpeg, so it is advisable to utilize it instead of generating SDP manually. The av_sdp_create function is employed to produce SDP for RTSP. Upon testing av_sdp_create and comparing its output with the WHIP muxer, we observed a considerable difference. It is strongly recommended to enhance the SDP functionality in the future once this patch is integrated. See here for detail.
• In addition to OpenSSL, it is necessary to provide support for alternative cryptographic libraries like GnuTLS, mbedtls, and others. To accomplish this, a dtls.c file should be created, analogous to tls.c, to improve utilization. Presently, only OpenSSL is supported, but after merging this patch, the goal is to extend support to other SSL libraries. This approach is taken because incorporating support for all SSL libraries in a single patch would result in excessive complexity and hinder comprehension for others. See here for detail.
• Currently, only the DTLS server role is supported for DTLS, as the DTLS server is much simpler than the DTLS client, and there is no need to handle the ARQ since the DTLS client will take care of it. All WHIP servers will support both DTLS server and client roles, making it safe and acceptable for FFmpeg to use the DTLS server role.
• Features such as congestion control, NACK, and FEC are not yet implemented. While NACK is essential, FEC remains optional.
• Currently, only the first candidate (UDP and host type) is utilized. A mechanism to determine the fastest candidate, select the best one, or switch during publishing is needed.
• While capturing the screen using FFmpeg, the h.264 profile may be negative (e.g. -99), in which case we set the profile to 0x42 (baseline). However, in such a situation, it may be necessary to parse from SPS/PPS.

Below are the issues that have already been fixed:

• Need to improve the end-to-end latency to 150ms from 800ms. Fixed by @T-bagwell @mypopydev @winlinvip
• For SRS to convert RTC to RTMP using time information, it is essential to send an RTCP SR report. Fixed by @duiniuluantanqin
• Only send binding requests to the server and cannot receive binding requests from the server. Fixed by @cloudwebrtc. In addition, he assists in enabling ICE use-candidate for Janus or Pion to finalize the ICE handshake.
• WHIP currently supports only HTTP, with HTTPS yet to be implemented. However, incorporating HTTPS should be straightforward, as ffmpeg already contains the necessary code. Note: The ffmpeg HTTP library has the functionality to support both the HTTP and HTTPS protocols.
• Support baseline/main/high profile without B frames. Even though WebRTC has the capability to support x264 main and high profiles without B frames, it is advisable to use the baseline profile for better compatibility. See bd9f7d1 @duiniuluantanqin
• During the DTLS handshake, when FFmpeg receives a ServerHello from the server, it generates a ClientHello for retransmission with a Certificate. This causes the server to retransmit the ServerHello, which still works but is not efficient. We should try to eliminate the unnecessary retransmission of the ClientHello. See 3b3b17a @winlinvip
• Compatibility is limited to OpenSSL 1.0.1k or newer versions. To prevent build failures, it may be necessary to ensure compatibility with older versions. See d68c259 @winlinvip
• When remuxing WebRTC to RTMP or HTTP-FLV, stuttering can occur every N seconds, which may be the same as the GOP size. Note that this bug occurs during screen streaming, but there is no issues when converting a file to WHIP streaming. After using the h264_mp4toannexb BSF, the RTC2RTMP works better, without stuttering issues anymore. See winlinvip@6598ffc @winlinvip
• Use h264_mp4toannexb to convert MP4/ISOM to annexb. Since the h264_mp4toannexb filter only processes the MP4 ISOM format and bypasses the annexb format, it is necessary to manually insert encoder metadata before each IDR when dealing with annexb format packets. For instance, in the case of H.264, we must insert SPS and PPS before the IDR frame. See here for detail. @winlinvip
• When duplicating a stream, the demuxer already sets the extradata, profile, and level of the par, so this function won't be invoked. When using an encoder like libx264, the extradata in par->extradata contains the SPS, including profile and level information, but the par's profile and level are not specified. It is essential to extract the profile and level data from the extradata and assign it to the par's profile and level. Remember to enable AVFMT_GLOBALHEADER, or the extradata will be empty. See eb0d0c0 @mypopydev @winlinvip

Latency

To test the end-to-end latency of a WIHP stream published using FFmpeg, you can capture your desktop using FFmpeg, open a stopwatch or miaobiao in the browser, and compare the player with the original stopwatch.

~/git/ffmpeg-webrtc/ffmpeg -f avfoundation -framerate 25 -pixel_format yuyv422 -i "2:0" \
    -vcodec libx264 -pix_fmt yuv420p -profile:v baseline -preset:v ultrafast \
    -b:v 800k -s 1024x576 -r 25 -g 50 -tune zerolatency -threads 1 -bf 0 \
    -acodec libopus -ar 48000 -ac 2 \
    -f whip 'http://localhost:1985/rtc/v1/whip/?app=live&stream=livestream'

Note: The parameter -i "2:0" is formatted as a video:audio device ID. Please use the ~/git/ffmpeg-webrtc/ffmpeg -f avfoundation -list_devices true -i "" command to list device ID and information.

Note: It is recommended to keep the threads=1 setting to prevent the occurrence of the Multiple Slices Issue. Please refer to Known Issues for more information.

The test results are incredible! The image below shows that the latency is around 150ms.

OpenSSL

The following OpenSSL versions are supported. A GitHub action here is available to automatically test all major OpenSSL versions for compatibility with FFmpeg WHP.

• OpenSSL 1.0.1k and newer. Note that OpenSSL 1.0.1j does not work, please refer to this link.
• OpenSSL 1.0.2 and newer.
• OpenSSL 1.1.0h and newer. Note that the compilation of OpenSSL 1.1.0g failed, but the API should work if you can fix the compilation bug.
• OpenSSl 3.0.0 and newer.
• OpenSSl 3.1.0 and newer.

In short, OpenSSL 1.0.1k and newer versions should work. However, OpenSSL 1.1.0h and newer verions are highly recommended.

Execute the command below to compile OpenSSL.

./config && make && make install_sw

Note: For macOS, please use command KERNEL_BITS=64 ./config instead.

Note: You can specify the target directory by --prefix=$HOME/.release/openssl and set the PKG_CONFIG_PATH="$HOME/.release/openssl/lib/pkgconfig" for FFmpeg.

Note: For OpenSSL 1.0, if OpenSSL not found even after confirming that the PKG_CONFIG_PATH is correctly set, then use --extra-libs="-ldl" while configuring FFmpeg.

Load Certificate File

To import a DTLS certificate and private key from a file, you should first generate an SSL certificate and private key file or obtain them from a Certificate Authority (CA) service.

openssl genrsa -out dtls.key 2048
openssl req -new -x509 -key dtls.key -out dtls.crt -days 3650 \
    -subj "/C=CA/ST=Toronto/L=Toronto/O=Me/OU=Me/CN=ffmpeg.org"

Then use -cert_file and -key_file to load it:

~/git/ffmpeg-webrtc/ffmpeg -re -f lavfi -i testsrc=size=1280x720 -f lavfi -i sine=frequency=440 -pix_fmt yuv420p \
    -vcodec libx264 -profile:v baseline -r 25 -g 50 -acodec libopus -ar 48000 -ac 2 \
    -f whip -cert_file dtls.crt -key_file dtls.key \
    "http://localhost:1985/rtc/v1/whip/?app=live&stream=livestream"

It works.

Authorization

Set option -authorization token to use the authorization of WHIP, please refer to the Authentication and authorization.

~/git/ffmpeg-webrtc/ffmpeg -re -f lavfi -i testsrc=size=1280x720 -f lavfi -i sine=frequency=440 -pix_fmt yuv420p \
    -vcodec libx264 -profile:v baseline -r 25 -g 50 -acodec libopus -ar 48000 -ac 2 \
    -f whip -authorization "mF_9.B5f-4.1JqM" \
    "http://localhost:1985/rtc/v1/whip/?app=live&stream=livestream"

Note: The token is mF_9.B5f-4.1JqM, and the HTTP header is set to Authorization: Bearer mF_9.B5f-4.1JqM.

Contributors

This patch has been created and is maintained by the developers below.

• Steven Liu [email protected]
• winlin [email protected]
• yangrtc [email protected]
• cloudwebrtc [email protected]
• Haibo Chen [email protected]

This patch has been reviewed by the developers listed below, and we extend our gratitude to them.

Note: Given that updating and discussing this patch may take several months or even a year, we have decided not to use the Pull Request comments for code review. Instead, we will be utilizing discussions.

The following review comments pertain to the patch/whip/v0 branch.

• Steven Liu [email protected]
  • CR #1: Rountrip0: Comments from Steven Liu
• Haibo Chen [email protected]
  • CR #1: Rountrip0: Comments from Haibo Chen
• Michael Niedermayer [email protected]
  • CR #1: Roundtrip1: Comments from Michael Niedermayer
• Kieran Kunhya [email protected]
  • CR #1: Roundtrip2: Comments from Kieran Kunhya
• Derek Buitenhuis [email protected]
  • CR #1: Roundtrip3: Comments from Derek Buitenhuis

The following review comments pertain to the patch/whip/v1 branch.

• TBD.

Links

• WHIP: https://datatracker.ietf.org/doc/draft-ietf-wish-whip/
• SRS WHIP: WebRTC: Support WHIP(WebRTC-HTTP ingestion protocol) srs#3170
• OBS WHIP: Add WebRTC (WHIP) output support obsproject/obs-studio#7926
• metaRTC: feature/metaRTC
• For RTP plaintext mode: Add RTP Plaintext Mode for Debugging with Wireshark

[winlinvip]

Updated by #18: Feature: refactor DTLS to merge it into tls_openssl.c. This is a big step toward merging the WHIP patch into FFmpeg. I initialized this patch but was unable to finish it. However, after the efforts of FFmpeg developers and maintainers, especially the work from Jack Lau, Steven Liu, Sean Der, Mypopy Zhao, a new patch has been created and completed the last improvements for FFmpeg WHIP. With this patch, we are now ready to submit it to FFmpeg, and we will have a rudimentary WHIP implementation in FFmpeg. Regarding to the new patch, there will still be lots of work to do. We need your help to test and review this patch.