Disable idle/background throttling when using a CanvasCaptureMediaStream

[aarongustafson]

Today, it's easily possible to build full live video mixers using existing APIs such as the CanvasCaptureMediaStream. However, the capability is crippled by the browser attempting to optimize canvas drawing and requestAnimationFrame timers. If the tab/window is backgrounded, drawing to the canvas stops and timers only run once per second. This happens in all common user agents.

When a canvas is being captured, all of this throttling should be disabled.

If necessary, perhaps a permission/request can be made, similar to WakeLock.

The only workaround for this today is to use pop-up windows for the UI. However, this has been proven to be less reliable, as browsers are starting to implement optimizations when the window is covered but not minimized. Additionally, if the user minimizes the window, everything stops. This is preventing the deployment of several video-related applications.

[guest271314]

This is primarily a Chrome issue https://bugs.chromium.org/p/chromium/issues/detail?id=1100746#c32, which has been fixed to an appreciable degree https://bugs.chromium.org/p/chromium/issues/detail?id=1145203. This is the specification issue re mute and unmute events being fired w3c/mediacapture-screen-share#141, test stephenlindauer/MediaRecorder-shareTab-issue#1, we do not lose any audio or video when recording at the same time the window where capture is occurring does not have focus. We can manually adjust how many frames are captured using frameRate constraint, when we want more frame per secod or an exact amount of frames for other use cases https://github.com/guest271314/screenshot.

[guest271314]

Some tests for

If the tab/window is backgrounded, drawing to the canvas stops and timers only run once per second. This happens in all common user agents.

• https://github.com/guest271314/MediaFragmentRecorder/tree/canvas-webaudio
• https://github.com/guest271314/MediaFragmentRecorder/tree/imagecapture-audiocontext-readablestream-writablestream
• https://github.com/guest271314/MediaFragmentRecorder/tree/webrtc-replacetrack-canvas-webaudio

Am interested in the results when the tab or window is not focused (note, flags or preferences can also impact media playback and capture, e.g., --disable-gesture-requirement-for-media-playback; --autoplay-policy=no-user-gesture-required).

[gregwhitworth]

Hey @guest271314 while this is a valid ask but is leading with a solution. It would be good to simply provide the usecases rather than a solution itself as certain areas this won't be possible or the way in which we'd be able to unlock the solution.

[guest271314]

It would be good to simply provide the usecase

If you read the Chromium bug linked in this issue and speification issue you will see my use case.

Which is how I encountered the bug: The specification does not state MediaStreamTrack should or must be muted when no user activity occurs.