-
Notifications
You must be signed in to change notification settings - Fork 18
FFmpeg: Add Whip Muxer support for subsecond latency streaming #1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
598aecc
to
cf69b01
Compare
48d93eb
to
92c4187
Compare
b4da552
to
2ad58fe
Compare
…cessary ClientHello.
1. Fix OpenSSL build error. 2. Support OpenSSL 1.0.1k and newer versions. 3. Support WHIP authorization via Bearer HTTP header. 4. Change the option default value from 1500 to 1200, to make Pion work. 5. Detect the minimum required OpenSSL version, should be 1.0.1k and newer. 6. Quickly check the SDP answer by taking a glance at the first few bytes.
1. Merge ICE and DTLS ARQ max retry options into a single handshake timeout. 2. Utilize DTLS server role to prevent ARQ, as the peer DTLS client will handle ARQ. 3. Replace IO from DTLSContext with a callback function. 4. Measure and analyze the time cost for each step in the process. 5. Implement DTLS BIO callback for packet fragmentation using BIO_set_callback. 6. Generate private key and certificate prior to ICE for faster handshake. 7. Refine DTLS MTU settings using SSL_set_mtu and DTLS_set_link_mtu. 8. Provide callback for DTLS state, returning errors when DTLS encounters issues or closes. 9. Consolidate ICE request/response handling and DTLS handshake into a single function.
1. Refine WHIP muxer name. 1. Refine SRTP key macros. 1. Refine logging context. 1. Refine SSL error messages. 1. Refine DTLS error messages. 1. Refine RTC error messages. 1. Use AV_RB8 to read integer from memory. 1. Update DTLS curve list to X25519:P-256:P-384:P-521. 1. Refine SRTP profile name for FFmpeg and OpenSSL. 1. Replace magic numbers with macros and extract to functions. 1. Alter log levels from INFO to VERBOSE, except for final results. 1. Use typedef SRTPContext. 1. Refine the ICE STUN magic number. 1. Reposition the on_rtp_write_packet function. 1. Refer to Chrome definition of RTP payload types. 1. Replace magic numbers with macros for RTP and RTCP payload types. 1. Rename to WHIP muxer. 1. Add TODO for OPUS timestamp issue. 1. Refine comments, do not hardcode H.264. 1. Define SDP session id and creator IP as macros. 1. Refine fixed frame size 960 to rtc->audio_par->frame_size. 1. Use h264_mp4toannexb to convert MP4/ISOM to annexb. 1. Address occasional inaccuracies in OPUS audio timestamps. 1. Correct marker setting after utilizing BSF. 1. Remove dependency on avc.h after using BSF.
…64_mp4toannexb filter only processing MP4 ISOM format.
1. Change the CommonName from ffmpeg.org to lavf. 2. Rename rtcenc.c to whip.c, rtc to whip. 3. Replace av_get_random_seed by AVLFG. 4. Add TODO to support libtls, mbedtls, and gnutls.
df62b77
to
63e3a55
Compare
Updated by #18: Feature: refactor DTLS to merge it into tls_openssl.c. This is a big step toward merging the WHIP patch into FFmpeg. I initialized this patch but was unable to finish it. However, after the efforts of FFmpeg developers and maintainers, especially the work from Jack Lau, Steven Liu, Sean Der, Mypopy Zhao, a new patch has been created and completed the last improvements for FFmpeg WHIP. With this patch, we are now ready to submit it to FFmpeg, and we will have a rudimentary WHIP implementation in FFmpeg. Regarding to the new patch, there will still be lots of work to do. We need your help to test and review this patch. |
This PR is updated by #18.
This PR has been squashed into a single commit on the patch/whip/v1 branch for easier submission to the FFmpeg community for code review.
WHIP stands for the WebRTC-HTTP ingestion protocol, which is a sub-second streaming protocol designed by encoders and publishers. It is widely supported by various tools and media servers, allowing it to interact with other WebRTC clients, ingest streams to media servers, and is compatible with all modern browsers.
Unfortunately, most WHIP implementations are highly complex and require modern C++11 or C++14, or RUST. This complexity makes it impossible to integrate with FFmpeg, which requires C.
However, if FFmpeg were to incorporate WHIP support, it could be a game-changer for both FFmpeg and the WebRTC ecosystem, particularly for certain IoT or small devices that are too small to run modern languages but can run FFmpeg.
To meet FFmpeg's requirements, this PR contains just C code. And we have rewritten the WHIP and WebRTC protocol stack using only around 3k lines of C code.
Content
Usage
Please select a open-source WHIP server to work with FFmpeg.
If you encounter any issues or get stuck, please leave us a message on Discord.
You also have the option to select a WHIP cloud service, which is typically provided by video cloud service providers.
If you encounter any issues or get stuck, please leave us a message on Discord.
Usage: FFmpeg + SRS
To enable FFmpeg to publish a stream, SRS can be used as the WHIP server. It is recommended to use docker.
Alternatively, you can build SRS from its source code.
To download the code and build FFmpeg, you can use the following command.
Although WebRTC has the capability to support x264 main and high profiles without B frames, it is advisable to use the baseline profile for better compatibility. If your stream doesn't have these codecs, you can transcode it using FFmpeg.
After publishing stream to SRS, you can play the WHIP stream in web browser such as Chrome, using srs-player.
The image below shows that the latency is around 150ms.
The RTMP, HTTP-FLV, or HTTP-TS stream remuxed by SRS can be played using ffplay, VLC, or srs-player.
Usage: FFmpeg + Janus
We referred to WISH, WHIP and Janus: Part II to make it possible to use FFmpeg for publishing a stream to Janus via WHIP. We have also created a demo docker image janus-docker for quick testing.
git clone https://github.com/winlinvip/janus-docker.git cd janus-docker
Initially, run the demo Docker that includes Janus server using the following command:
After that, access the URL http://localhost:8081/videoroomtest.html?room=2345 in your browser to join the Janus room.
Next, download and run the Simple WHIP Server for Janus using the following command:
git clone https://github.com/meetecho/simple-whip-server.git cd simple-whip-server npm install npm run build npm run start
Generate a WHIP handler using curl, which will enable ffmpeg to join the same Janus room through WHIP:
To download the code and build FFmpeg, you can use the following command.
Although WebRTC has the capability to support x264 main and high profiles without B frames, it is advisable to use the baseline profile for better compatibility. If your stream doesn't have these codecs, you can transcode it using FFmpeg.
After publishing the stream to the Janus room, you will be able to view it on the previously opened webpage. The image below shows that the latency is around 120ms.
Usage: FFmpeg + Pion
To download the code and build FFmpeg, you can use the following command.
Although WebRTC has the capability to support x264 main and high profiles without B frames, it is advisable to use the baseline profile for better compatibility. If your stream doesn't have these codecs, you can transcode it using FFmpeg.
After publishing stream to pion, you can play the WebRTC stream in web browser such as Chrome
Usage: FFmpeg + Millicast
On the way.
Usage: FFmpeg + TRTC
On the way...
Usage: FFmpeg + Cloudflare
Here's a WHIP URL for testing: https://customer-wi9sckcs7uxt7lh4.cloudflarestream.com/67805a89d86d263b95f7056e5b212a0ek99a1686b042e85c3a5f86c38af9d9dad/webRTC/publish
And WHEP to play it back:
https://customer-wi9sckcs7uxt7lh4.cloudflarestream.com/99a1686b042e85c3a5f86c38af9d9dad/webRTC/play
You can use this player: https://wish.chens.link/watch
Known Issues
The current version has some known issues that we plan to fix in the future. You are welcome to help us fix them by sending a pull request.
The current known issues include:
-tune zerolatency
option is enabled andthreads>1
, FFmpeg will encode the frame in multiple slices. This process can cause stuttering in Chrome's decoding, with only the IDR being able to be decoded. Both the IDR and P frames may be encoded to multiple slices and sent by multiple RTP packets with the same timestamp. RTC players, such as Chrome, may have issues decoding frames that have been encoded using multiple slices. As a result, Chrome may only decode some of the slices of the IDR frame and drop the P frames. If this issue occurs, it may appear as if the video player is stuttering and the decoded framerate will be equivalent to the GOP size.Below are the issues that have already been fixed:
Latency
To test the end-to-end latency of a WIHP stream published using FFmpeg, you can capture your desktop using FFmpeg, open a stopwatch or miaobiao in the browser, and compare the player with the original stopwatch.
The test results are incredible! The image below shows that the latency is around 150ms.
OpenSSL
The following OpenSSL versions are supported. A GitHub action here is available to automatically test all major OpenSSL versions for compatibility with FFmpeg WHP.
In short, OpenSSL
1.0.1k
and newer versions should work. However, OpenSSL 1.1.0h and newer verions are highly recommended.Execute the command below to compile OpenSSL.
Load Certificate File
To import a DTLS certificate and private key from a file, you should first generate an SSL certificate and private key file or obtain them from a Certificate Authority (CA) service.
openssl genrsa -out dtls.key 2048 openssl req -new -x509 -key dtls.key -out dtls.crt -days 3650 \ -subj "/C=CA/ST=Toronto/L=Toronto/O=Me/OU=Me/CN=ffmpeg.org"
Then use
-cert_file
and-key_file
to load it:It works.
Authorization
Set option
-authorization token
to use the authorization of WHIP, please refer to the Authentication and authorization.Contributors
This patch has been created and is maintained by the developers below.
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
This patch has been reviewed by the developers listed below, and we extend our gratitude to them.
The following review comments pertain to the patch/whip/v0 branch.
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
The following review comments pertain to the patch/whip/v1 branch.
Links