Make it possible to play unsupported video codecs

[szaimen]

Is your feature request related to a problem? Please describe.
Some video containers are not supported by webbrowsers and probably will not be in the near future.

Describe the solution you'd like
Find a solution that makes it possible to play almost all video containers in browsers.
One solution would be using wasm and ffmpwg to transcode videos on the fly directly in the browser. See e.g. this article: https://blog.scottlogic.com/2020/11/23/ffmpeg-webassembly.html#:~:text=creating%20a%20streaming%20transcoder

Describe alternatives you've considered
Not doing this?

Additional context
Add any other context or screenshots about the feature request here.

[labor4]

I see what you mean ;-)

A word of caution though.
Ffmpeg as a child of a php thread is probably a limited experience, prone to hanging the whole system. Even rewrapping could be too much.

My first thought appart from this, was to offload the task to a mature standalone server, also capable of using GPUs, and providing an IN/OUT API for nextcloud. The vast amount of combinations in containers and codecs regarding web delivery also seems to me a very standalone project idea.

[szaimen]

I think the memories app has the pssibility to transcode videos on the fly. So maybe we could have a look at the app how they are doing it which sounds like the best approach to me.

[labor4]

IIRC ffmpeg as a subprocess of php would limit to 1 thread.
Encoding ProRes to h264 can easily eat 500% cpu on 2.2Ghz.
That's why I recommend ffmpeg to be an instance that is independent and can take care of itself.

[szaimen]

The memories app features on-the-fly transcoding now

[skjnldsv]

The memories app features on-the-fly transcoding now

Done via php and ffmpeg btw
https://github.com/pulsejet/memories/wiki/Configuration

[szaimen]

Done via php and ffmpeg btw
https://github.com/pulsejet/memories/wiki/Configuration

No, the transcoder is written in go IIRC:
https://github.com/pulsejet/go-vod

[ohthehugemanatee]

IMO this is about much more than supporting other video formats. Note that modern cameraphones don't encode their videos with HLS for streaming, and can produce very large files. Here are some of the other reasons I have in mind:

• Devices, browsers, and connections vary greatly in their capabilities
• All browsers suck at skipping around in a 400MB remote 4k video file without HLS and chunking.
• Present implementation seems to start a new process reading the entire video file for every reload (or maybe every time the user skips around in the file), which is very IO heavy. And since the whole experience is dead slow for a big video file, users do a lot of reloading. (This has brought my storage cluster to its' knees several times)
• modern users expect adaptive resolution adjustment for bandwidth and browser capabilities..
• Other parts of Nextcloud (Memories, namely) transcode, so the user experience of video watching changes drastically depending on where they launched the file from.

Recommend:

1. changing the Issue title to "Implement transcoding to playback video on variety of devices, bandwidths, and file formats"
2. making it easier for 3p apps to replace parts of the interface for video/photo viewing, so e.g. memories doesn't need to implement it's own whole modal window to get their improvements to the video experience... they can just replace part of Viewer's video modal interface and use that.

[labor4]

for reference i recommend to also open the research looking at bigger scale applications:

• i'm in awe how much optimization Jellyfin is doing to get as far as possible without encoding (basically rewrapping). Every browser has its own limits in combination of stream+container. Their Progress is slow and as such very impressive to find respect in.
• Streaming servers such as the Mistserver (now open source) built their whole businessmodel on the fact that no app wants to do what they do, if not necessary. Mistserver doesn't even do encoding primarily. It primarily works on finding the best possible combination for the specific browser. It is heavy on API implementation, which is impressive. Built to be invisible.
• There are simpler give/take platforms that do the handling/encoding externally that may be more measured for NC plugging (think in terms of what NC did for external office servers).
• I work in postproduction and tried a lot of things to automate our facility. I believe we should keep a good look at hw limits. Because if something works, doesn't mean it works always. The devil is in the details. Sometimes a higher bitrate means game over. Or a different codec. When you reach the need for a GPU, you also dont want to have this inside a NC instance since it's very bulky hardware-whise.

I think this could be something to burn our fingers on.
It may be worthwhile to check on an external encoder. Maybe something smaller, a docker instance even. I think at some point the problems that video online delivery holds, should be contained in a separate instance.

I found multiple projects at first glance, but not the promising one right away, i earlier found trustworthy and mature, Sorry. Mistserver doesn't do GPU, so i would not recommend it as a sole solution (i deliver preencoded multichannel mpeg_ts to mistserver, but thats streaming not VOD)

My remarks seem overkill,
but if there are fitting solutions out there, this may well overlap with the desire for NC to be a video delivery platform, which for files it already is.

best wishes
m

[ohthehugemanatee]

That's actually how Memories does it - the app downloads go-vod at runtime. In settings you can choose to use an external go-vod over HTTP, specifically so you can separate hardware requirements as necessary.

While I agree that there are lots of potential solutions to investigate, given that there's already a Nextcloud-based implementation that offers the all the desired capabilities using Vue, we should focus on leveraging that work. I also created a ticket on Memories to add the option to override the video/photo viewers globally.

[ohthehugemanatee]

related #330 nextcloud/server#39560 and nextcloud/server#19567