Is it possible to apply AMD frame generation to MPV?

[laichiaheng]

The frame generation is widely used in gaming, there are much less artifacts with AI than older frame generating tools like SVP.
Is it possible for us to use the frame generation in video playing?

[Jules-A]

AMD does have it as part of AMF specifically for video: https://github.com/GPUOpen-LibrariesAndSDKs/AMF/blob/master/amf/doc/AMF_FRC_API.md#22-configuring-the-frc , however, unlike AMD's video upscaler (https://ffmpeg.org/ffmpeg-filters.html#sr_005famf-1) it hasn't been implemented into FFPMEG yet so MPV devs would have to figure out a way to implement it for it to work in MPV.

EDIT: FRC apparently only works in a DX12 or OpenCL context. MPV doesn't have a DX12 context option and OpenCL from what I recall reading was too messy/difficult so it's unlikely they'll end up adding it. AMF_FRC_ENGINE_TYPE does have an option for DX11 but I'm confused...

[laichiaheng]

So it's possible for FFMPEG developer to support AMF?

[42147]

Nope for mpv. Yep for some other video player. Here's a video of someone comparing the original AMD Fluid Motion Video with the new AMD Fluid Motion Frames: https://www.youtube.com/watch?v=sESxDwqBN2k

There's Bluesky Frame Rate Converter if you're on Windows and want to test it yourself. Don't have Windows anymore, but have been holding on to old GCN cards just for the small chance they end up releasing the sources for the original AFMV, but there's zero chance for that since it was developed in collaboration with CyberLink Corp.. What a waste of an ASIC.

EDIT: Niklas Haas floated this idea a few years ago, which could be useful for implementing frame interpolation plugins in mpv: New grand scripting API overhaul

In the mean time I just use RIFE with Vapoursynth: VapourSynth-RIFE-ncnn-Vulkan

[laichiaheng]

It works perfectly below 720P@30fps.
--vo=dmabuf_wayland seems to be the most power efficient way to play video under Wayland, it doesn't load any of my shaders and scale algorithm in the config.

Maybe the driver of 9070XT is not fully optimized yet? Or maybe it's because Nvidia uses the Tensor core.

By the way, do you suggest to go above 4 threads for RIFE? In the comments, it says that it suggest to use 1-4 threads.
What does the :4:16 do behind vapoursynth?

[42147]

Ok. So dmabuf-wayland is similar to --vo=vaapi. I'll switch to Wayland as soon as Xfce fully supports it.

As I interpreted the table in the SVP wiki, the RTX 3080 without TensorRT can do 1080p. With TensorRT even an RTX 3060 could do 1080p. The culprit in this situation could be any of the parts in the process chain, Vapoursynth, ncnn, Vulkan, Mesa or AMDGPU. There's always something missing in hardware capabilities or not yet implemented in software. An RX 480 can process RIFE @ 960x540. So a Vega 56 which even supports half-precision floating-point format FP16 should be able to do 1280x720, but yet it starts dropping frames on playback at 1280x540. Apparently Vapoursynth has some image processing functions that don't support FP16 yet, so if it feeds FP32 format data to the RIFE plugin, does it make it slower and if so, how much.

Try 3 or 4 GPU threads. I haven't had significant gains from going over 2 GPU threads myself but I did see someone post a comment to add more threads if the GPU is "hungry". So if the GPU load seems low add more threads.

vapoursynth=file:buffered-frames:concurrent-frames

EDIT: Apparently at some point I benchmarked that 4:16 and had a lot less dropped frames than using the defaults. https://mpv.io/manual/stable/#video-filters-vapoursynth

EDIT: If you haven't already, set hwdec=no. Using hwdec just slows down the processing since it has to send the decoded frames from the GPU to the CPU for Vapoursynth to process and then back again. This is also where a native libplacebo plugin could be beneficial if all processing is done on the GPU. AFMV benefited from a zero-copy mode where the GPU decoded images were processed with the ASIC without sending them through the CPU in between.

[42147]

You might be right about the drivers not being fully optimized yet. Saw a comment by a 9070 XT owner saying they got significant gains from the latest Windows drivers on games. Also just realized the 9070 XT came out less than 6 months ago. Mesa 25.2 also just got released...

[laichiaheng]

You might be right about the drivers not being fully optimized yet. Saw a comment by a 9070 XT owner saying they got significant gains from the latest Windows drivers on games. Also just realized the 9070 XT came out less than 6 months ago. Mesa 25.2 also just got released...

https://github.com/AmusementClub/vs-mlrt
Is it possible to replace the NCNN with vsmigx: MIGraphX-based GPU Runtime for RIFE?
It says that it is optimized for AMD GPU, and the drawback of NCNN is that it's slower.
There also are vapoursynth-plugin-mlrt-migx-runtime-git and vapoursynth-plugin-mlrt-ext-models-rife in AUR.
Can we apply this plugin to the vpy that you gave us before?

[42147]

I've never used vs-mlrt myself but have browsed the repo a few times before.

So apparently TensorRT can work on AMD GPUs:

AMD has ported the TensorRT inference engine to MIGraphX for use with AMD GPUs.

https://www.amd.com/content/dam/amd/en/documents/products/embedded/solutions/apollo-baidu-white-paper.pdf

Glad to see you already got your questions answered by a developer over at the vs-mlrt Discussion page. Unfortunate to hear MIGraphX isn't a viable option at this time. AMD just seems very unlucky regarding motion interpolation for video. There was also a case with SVP not working on Linux with AMD GPUs using OpenCL.

[philipl]

If you're willing to spend a few dollars, Lossless Scaling's Frame Gen works with mpv on both Linux and Windows (I've only tested on Linux).

[laichiaheng]

I really hope that I can do it without Steam.

[DefectiveByDesign]

VLC 3.0.22 will implement AMD frame generation. I would hope MPV can copy this over, as well as natively supporting FSR1 for image scaling. FSR doesn't just help upscaling, it seems to better clarify details out of compressed video.

It's been pretty annoying that frame gen and upscaling is locked behind paywalls like SVP or lossless scaling. Especially on android. I've been using MPV on PC, SVP on android, but VLC might just have provided a reason to switch.

[DefectiveByDesign]

Yes, but this isn't easy to setup, especially on Linux. I've tried using it on android, and I'm pretty sure it doesn't work there unless you have a special setup under a blue moon. At least on windows, you can run MPV.net, but everywhere else....

So I'm not a fan. The method for writing the config file is rather vague. I've seen completely different methods / syntax used for examples. Then there's that whole api output thing. Does it not work at all with specific render output methods? vo=gpu? vo=gpu-next? IDK.

Like here's a question: If the shader is for OpenGL, how does this work in Vulkan? Is there issues with wayland? There's definitely android issues.

The general impression I get of how this is dealt with, is:
1: Here's a feature that exists.
2: We're not going to tell you at all how to use it properly.
3: We're not going to document the help in easy language.
4: The examples can be contradictory, and usually don't explain what can cause it to not work. There's years in-between examples. Search engine results don't prioritize the best examples.
5: There's multiple shaders, the common one is rather old, there could be a better version elsewhere. You probably need a custom version for android. Search results are terrible. Good luck.
6: It just works for me, IDK what your problem is. Get good scrub, learn to code.

Yeah. Then VLC "just works" for people.

I believe I got it working in windows when it first came out, but then I switched to linux, and had issues with it. Never got android working, afaik. Maybe it works fine now, but I kind of stopped caring, especially since I don't remember the process.

I'm using Haruna, and it doesn't have a stat page, nor does a custom bind show page 2, it's just blank.
config is:
[Playback]
UseHWDecoding=true
glsl-shader="~~/shaders/FSR.glsl"
gpu-api=vulkan
profile=gpu-hq
vo=gpu-next

No idea if it's working. Maybe it is, maybe it isn't. That's why I'd rather it be directly implemented. Especially if frame generation is directly implemented.

edit: Tried MPC-QT, maybe it works better. IDK. Still can't find any stat that says it's working... Nope, it's a known issue that the stat page doesn't work.

[ghost]

I don't see how FSR1 being a built-in shader would solve your underlying issue. You would still need to add a single line to your config regardless.

[Jules-A]

You should be able to use AMD's FSR (the one specifically for video and not games) from ffmpeg as a video filter (https://ffmpeg.org/ffmpeg-filters.html#sr_005famf-1) but I can't seem to get it working in MPV:

If that was working, along with the shader method, there's no reason to have it built in to the video player.
Frame gen is a different story though...

[DefectiveByDesign]

Jules, that link does not seem to specifically mention FSR. It mentions AMF, and some upscalers, like bicubic. They mention "sr1", but it says nothing about what this is, which is typical of these feature manuals. Maybe it's FSR, maybe it's not. IDK.
1: Here's a feature that exists.
2: We're not going to tell you how to use it properly. Vague description, job done.

"I can't seem to get it working in MPV" This is the typical user experience of MPV. It's a video player catering to command line using programmers.

"If that was working, along with the shader method, there's no reason to have it built in to the video player."
Except it's not working, it's poorly documented, and hasn't worked properly or been documented properly ever since they "supported" the feature.

There's probably some strange undocumented activation method, like needing to run Ubuntu LTS under X11 Gnome, on some specific Nvidia GTX video card, using a specific driver, with OpenGL, and nothing else works. We don't know why it doesn't work. It's not explained. They've left it like this for years, and there's no attempt to improve it. It's a linux app that for whatever reason works better under windows, and instead of using high quality by default, uses the most generic settings. It also runs slower than everything else on Android for whatever reason, and drops frames on older arm hardware.

I've seen some people recommend using Gamescope with FSR to upscale video, and lossless scaling for windows. That's how bad the support is. We shouldn't have to do this, when MPV "supports" shaders directly, but that's the typical user experience.

[Jules-A]

It's AMD's video super resolution, essentially FSR but optimized for video. Unfortunately I haven't managed to get it to work yet so I can't tell how it compares with the shader.

According to AMF, the default sr1.0 is just a relabeled FSR:

VideoSR1.0 scaling algorithm. This algorithm is based on FSR 1.0.

VideoSR1.1 scaling algorithm. This algorithm is intended for specific internal integrations and is exposed purely for experimental use. VideoSR1.1 is only supported when "AMF_HQ_SCALER_ENGINE_TYPE" is set to "AMF_MEMORY_DX11" or "AMF_MEMORY_DX12" and the input and output formats are not NV12 or P010.

As for your other rambles, it's extremely simple on Windows to get shaders working but I haven't tried on Linux though I'd imagine if you are using Linux then a few command line stuff shouldn't be a problem???

[kasper93]

Is it possible to apply AMD frame generation to MPV?

Fixed by: #17148