Next steps for ffmpeg.wasm

[jeromewu]

Hi,

First of all, thanks for people who use and support ffmpeg.wasm for the past two years. 😄 Without you, it is impossible for ffmpeg.wasm to reach 8.4k+ stars and be used by 1.9k+ projects / packages! And now we also have 32 contributors support this open source project and that's why we have 30 releases already. 🎉

I have to admit I didn't actively maintain this ffmpeg.wasm for the past 1 year, but thanks to this one year I also had time to rethink what is the next for ffmpeg.wasm and I would like to share it here and it would be great if you can provide any feedbacks / suggestions.

There are 4 aspects I would like to share: community, developer experience (DX), performance and features.

Community

As an open source project, it relies on the support from the community to grow. But most of the time the community won't grow itself, it requires mechanisms to achieve. That's why I would like to add / enhance following mecahnisms here:

• ffmpeg.wasm discord channel
• Document roadmap with Github Projects

Developer Experience

As one of the main contributor of this project, I really suffered from the not-so-good DX when developing ffmpeg.wasm. Mainly there are 5 issues I really want to do to make it easier for contributors:

• Adopt monorepo using lerna to keep source code in one place
• Speed up core build time (it takes around 1 hour to build a new ffmpeg.wasm-core) with cache and parameters tuning
• Adopt typescript to achieve one module system and provide types
• CI for building and testing assets.
• Documentation for how to build and contribute to this project

Performance

If you check Issues here, many issues actually relate to performance issue of ffmpeg.wasm. That is one issue I found it extremely critical and hard to resolve. But if we don't do anything about it, ffmpeg.wasm will never reach a state of usable.

The primary attempt was to use multi-threading to speed up, it works but also suffers from comapatibility issues as not all browsers support SharedArrayBuffer and makes it messy to use. Also multi-thread using massive memory (like 2 GB) that might not be affordable for low-end devices.

To improve performance without multi-threading, right now the most promising way is to use SIMD intrinsics, this is not easy but key to success. As a beginning, I would like to try it on x264 as mp4 is one of the most popular formats as it is.

• Speed up x264 with SIMD intrinsics
• Speed up ffmpeg with SIMD intrinsics
• Speed up other codecs if possible

Features

I know there are lots of features requested by the community, and some of them (ex. RTSP) are queried many times. But the fact is some features actually limited by the current status of WebAssembly which makes it almost technically impossible at the moment. Thus it requires extra amount of efforts to figure out a workaround or even changes in WebAssembly itself. I think features like this should be discussed in an issue or thread for a long time, and requires everyone's contribution to make it true.

• Accumulate long term discussions in Github Discussion

---

I believe there are much more we can and should improve here, but for now I would like to focus on these aspects / tasks and see how it goes. I cannot guarantee how long it takes for issues like performance, but definitely I am happy to share with everyone when there is any progress. At the same time it is always welcome to submit PRs or just provide your ideas. 😄