Multimedia support for fyne

[spatocode]

Providing multimedia support for fyne such as playing back audios and videos which may take advantage of the available codecs in the platform. This requires a media engine or similar.
May require #225

[MarcoVdE]

I suggest relying on packages that already exist:
Video: https://awesome-go.com/#video I'd suggest https://github.com/3d0c/gmf as it uses the ffmpeg libraries.
Audio: https://awesome-go.com/#audio-and-music specifically I'd probably go with: https://github.com/gen2brain/malgo as it is a (unofficial) port of the miniaudio project: https://github.com/dr-soft/miniaudio

EDIT: Note https://github.com/fyne-io/fyne/projects/4#card-14426093 , think we should add this issue to the card @andydotxyz

[c1ngular]

Preferably with support of streaming protocols , such as http/rtmp/rtsp...

[Bluebugs]

Video support is problematic for license reason. Every dependencies that provide video/audio decoding is under lgpl. If we want to continue providing one binary with no dependencies, the resulting application can only be gpl. We might want to have an extra repository that provide this feature independently.
Also the simplest and most efficient API to use for just a player would be libmpv in my opinion (mpv.io).

[c1ngular]

@Bluebugs true , media player support require lots of consideration and work indeed .
though libmvp stated on its github page :

mpv's focus is not on power-efficient playback on embedded or integrated GPUs (for example, hardware decoding is not even enabled by default). Low power GPUs may cause issues like tearing, stutter, etc. The main video output uses shaders for video rendering and scaling, rather than GPU fixed function hardware.

i believe fyne has support for embedded hardware/systems such as Raspberry Pi ? i am really interested to try this combination out on Raspberry Pi 4B whenever available , hardware accelerated and not .

[MarcoVdE]

Video support is problematic for license reason. Every dependencies that provide video/audio decoding is under lgpl. If we want to continue providing one binary with no dependencies, the resulting application can only be gpl. We might want to have an extra repository that provide this feature independently.
Also the simplest and most efficient API to use for just a player would be libmpv in my opinion (mpv.io).

I'd go the dependency route on another project rather than building a custom solution, using e.g. gmf would allow for this repo not having to manage keeping up besides adding the latest ffmpeg library.

[Bluebugs]

I am not sure gmf answer or need after a quick look at it (might be wrong). I see two problems with it.

First it depends on ffmpeg being installed on the system as specified with the pkg-config of ffmpeg when the application has been build. This might be a problem in term of portability and will require some work to be addressed.

The second problem I see is that it doesn't seems to be designed to integrate efficiently with a canvas, but more designed for video manipulation. This is not too surprising as there isn't many UI framework in go. This might mean an important amount of work.

Also it isn't clear to me how portable a solution it is.

I think the way to go at it, is to list our requirement first and see if we can find something that help. So what I believe we need:

• portable solution (Linux, Windows, Mac, iOS and Android)
• single binary possible if the application accept to be GPL, relocatable dependencies if not a single binary for the application
• integration into an open gl canvas easily
• would be great if we can get some hardware acceleration on some platform (looking at how hard it is for Firefox to do so, I don't put that as a hard requirement)

What do you think? Any think I missed?

[c1ngular]

@Bluebugs is it a doable option to develop a new opengl integrated/optimized media player based on ffmpeg libraries ?
ffmpeg has a very solid infrastructures with a strong community/plugins/patches , it enables hardware acceleration on many platforms (even on Raspberry Pi ), so you/we don't have to start everything from scratch . IMHO, ingredients needed for such a media player is already out there , main work would be integration to fyne ?

[Bluebugs]

The difficulty is in the pipeline to manage having it setup the right way everywhere. Ffmpeg is the building block for it, but using it is non trivial. That's why things like libmpv (vlc, gstreamer, upipe, ...) exist and encapsulate it. The warning on their website is something that I do not think would be a problem for us as fyne already use shader and not a fixed pipeline. Also I would strongly recommend to use an existing abstracting layer on top of ffmpeg as it is a really hard job to get right.

[c1ngular]

@Bluebugs i am not an expert , and your point makes sense , it would be a huge boost if a solid "existing abstracting layer on top of ffmpeg" could be found and integrated .

is this work already in progress ?

[Uldiniad]

are animated gifs currently supported or would these need to be handled as well?

[andydotxyz]

Animated gifs are not fully supported, but they can be shown in Fyne with a little application code. This is possible because we can draw a single GIF frame in the normal image way.
We will be adding them in fully as a result of the animation code I’m working on I hope. I’m just not sure how to expose the Loop option for them.