progress event support

[astroza]

Hi,

I was working on a transcoding platform and I found your module to wrapping ffmpeg. I'm interested to make improvements that would be useful to everyone involved in this niche.

I'm planning to replace Popen.communicate call to achieve the goals:

• Make ffmpy efficient in memory usage
• Give it progress reporting support

Popen.communicate() is inefficient because of stdout and stderr buffering. Let's see some cases:

Use case 1:

you want to send transcoded data over network as fast as available. If communicate() is used, you must to accumulate the entire transcoded movie on a memory buffer to send it.

Use case 2:

you want to know the transcoding state (output size, time, bitrate) in real time. If communicate() is used, you can't do it because this method is consuming the stderr pipe

I have a experimental branch with a on_progress callback implemented here:
https://github.com/astroza/ffmpy/tree/progress_support

An usage example is:

def on_progress(state):
        print state.time

ff = ffmpy.FFmpeg(
        global_options="-hide_banner -nostdin -y -stats",
        inputs={local_input_path: None},
        outputs={local_output_path: None},
        on_progress=on_progress
)
ff.run(stderr=subprocess.PIPE)

Now I'm working to make it backwards compatible. The branch break your API

UPDATE

https://github.com/astroza/ffmpy/tree/progress_support_v2 keeps backward compatibility

Usage:

def on_progress(state):
        print state.time

ff = ffmpy.FFmpeg(
        global_options="-hide_banner -nostdin -y -stats",
        inputs={local_input_path: None},
        outputs={local_output_path: None}
)
print(ff.run(on_progress=on_progress)[1])

[yarodevuci]

@astroza is there any way to get progress of remaining time or something like that ? or like 50% complete

[horamarques]

Hi
any plans to add error messages on the progress output?
any ideia on how to do it?

thanks

[Benny739]

Any news on this?

[astroza]

Hi @Benny739, I have had no response from the repo owner. Nevertheless, you can find an working branch (it's used on a company) here https://github.com/astroza/ffmpy/tree/progress_support_v2

movie_description = ....

def on_progress(state):
       progress = (float(state.time or state.start_time) / movie_description.duration) * 100
       print(progress)

ff = ffmpy.FFmpeg(
        global_options="-hide_banner -nostdin -y -stats",
        inputs={local_input_path: None},
        outputs={local_output_path: None}
)
print(ff.run(on_progress=on_progress)[1])

state object has all what you need

[Benny739]

Thanks a lot, this is exactly what I was looking for.

@Ch00k merging this into the package would be wonderful.

[Benny739]

Hi @astroza,
I'm having unclosed files now, any idea how I can close them?

/Users/benny/angular/video/python/integ_test/tests/integ_test_convert.py:31: ResourceWarning: unclosed file <_io.BufferedWriter name=4>
  converter.generate_thumbnails()
Object allocated at (most recent call last):
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", lineno 778
    self.stdin = io.open(p2cwrite, 'wb', bufsize)
/Users/benny/angular/video/python/integ_test/tests/integ_test_convert.py:31: ResourceWarning: unclosed file <_io.BufferedReader name=5>
  converter.generate_thumbnails()
Object allocated at (most recent call last):
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", lineno 789
    self.stderr = io.open(errread, 'rb', bufsize)

I'm basically using your code.

[astroza]

Sorry I can’t help you, it seems unrelated to the library (according you stack trace) and I can’t see your code

[Benny739]

I found the bug, it's caused by piping and not closing stdin and stderr. In the wait function after the while loop I close stdin and stderr now, and the warning is gone.

    def wait(self, on_progress=None, stderr_ring_size=30):
        stderr_ring = []
        is_running = True
        stderr_fileno = self.process.stderr.fileno()
        ff_state = FFstate()
        while is_running:
            latest_update = os.read(stderr_fileno, self.update_size)
            if ff_state.consume(latest_update) and on_progress is not None:
                on_progress(ff_state)
            stderr_ring.append(latest_update.decode())
            if len(stderr_ring) > stderr_ring_size:
                del stderr_ring[0]
            is_running = self.process.poll() is None

        stderr_out = str.join("", stderr_ring)

        self.process.stdin.close()
        self.process.stderr.close()

        if self.process.returncode != 0:
            raise FFRuntimeError(self.cmd, self.process.returncode, stderr_out)

        return stderr_out