Issue with Raw Image Streaming to FFmpeg Process via Pipe on React Native

[uripont]

Title: Challenges with Raw Image Streaming to FFmpeg in React Native Application

Description:

Goal

We aim to implement a solution for capturing and encoding video directly within a React Native application. The objective is to stream raw image data directly into an FFmpeg process for continuous video encoding, ensuring minimal storage overhead and efficient processing.

Problem

1. Pipe Auto Closes with PNGs: When attempting to stream PNG images to the FFmpeg process, the pipe auto-closes after each image is passed. This behavior disrupts the continuous encoding process, resulting in incomplete video streams.

2. Inability to Pass Base64 Raw Images: In our current implementation we attempted to pass as base64 encoded raw images. However, the FFmpeg process expects a file path rather than raw data when passing it with FFmpegKitConfig.writeToPipe(), making it impossible to feed the images directly into the encoder. This limits our ability to keep the encoding process active until explicitly closed.

The ideal solution involves continuously streaming raw image data directly to FFmpeg without the need for intermediate storage. This would involve:

• Configuring FFmpeg to accept raw video input via a pipe.
• Ensuring the pipe remains open for continuous data feed until manually closed.
• The encoder keeps taking in the raw inputs and using hardware-accelerated libraries to encode them into a video.

Challenges

• FFmpeg Configuration: Properly configuring FFmpeg to handle raw image data streams, including setting appropriate pixel formats and frame dimensions, is causing issues.
• Data Feed: Developing a mechanism to pass raw image data (in base64 or other formats) directly into FFmpeg without requiring file paths (and thus without requiring them to be stored on the user's file system).

We seek guidance on:

• Configuring FFmpeg to handle continuous raw image data streams.
• Efficiently passing base64 raw images or similar data formats directly to FFmpeg.
• Ensuring the encoding process remains active until manually closed, without auto-closing the pipe.

Environment:

• React Native
• ffmpeg-kit-react-native

Expected Behavior:
Continuous streaming of raw image data to FFmpeg, allowing seamless video encoding until the process is manually terminated.

Actual Behavior:
Pipes auto-close after each PNG frame, and raw image data cannot be directly passed due to file path requirements.

Any advice or solutions to address these issues would be greatly appreciated.
Thank you!

[Indra-poudel]

@uripont did you able to come with solution. I have similar usecase :D

[uripont]

Unfortunately no, we haven't found any solution to this issue.

But we have found an alternative way to fulfill our needs: instead of encoding a video ourselves frame by frame, we end up re-encoding an already recorded video to achieve the same result.

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[tanersener]

The current FFmpegKitConfig.writeToPipe() method is designed to work with only one input file. It closes the pipe as soon as that input is fully read, so it doesn’t support multiple files as you mentioned. While it’s technically possible to extend this functionality, it's outside the project’s focus. Additionally, there are already other ways to write data into pipes, so adding limited pipe features in FFmpegKit isn’t very beneficial.

If you need to use pipes with multiple files, we suggest extending the existing iOS and Android implementations. You can find the source code here for iOS and here for Android. Although these aren’t written for React Native, the approach is similar, you can replace Android/iOS specific methods with React Native equivalents.

As for passing base64 data to FFmpeg, we don’t have a solution yet. I’m experimenting with some options when I can, but nothing is ready at the moment.