Request: Whisper.cpp with ffmpeg 8.0

[lededev]

configure --enable-whisper to enable whisper, need whisper.cpp libraray and header file to compile with ffmpeg.

[Jamaika1]

Not everything works out for me. I don't know if the ffmpeg plugin whisper supports OpenCl, Vulkan, or Cuda.
https://forum.doom9.org/showthread.php?p=2021507#post2021507

Edit: You can use ready-made libraries (.a). I have no idea what CUDA/OPENCL/VULKAN drivers were used in ggml.
https://packages.msys2.org/packages/mingw-w64-ucrt-x86_64-llama.cpp
https://packages.msys2.org/packages/mingw-w64-ucrt-x86_64-ggml
https://packages.msys2.org/packages/mingw-w64-ucrt-x86_64-whisper.cpp

[waldonnis]

Looks like the required library supports Vulkan and CUDA, and the main ffmpeg commit includes some ability to specify GPUs (see af_whisper.c), but I haven't read/built it or tried the binaries on doom9 yet to see how the ffmpeg filter actually works.

I'm definitely interested in using this if it works well and will probably attempt to build it locally in the next couple days.

[waldonnis]

After wrestling with this over the past couple of days...

Building this with MSVC was easy and worked well (at least with both the CPU-only and CUDA backends; didn't try Vulkan with MSVC). Building it with msys/mingw, though...not so much.

Honestly, the cmake-based config needs some work in general, and especially if you want to use msys/mingw. Without writing a novel and because I didn't spend more than a couple of non-contiguous hours on it:

• CPU-only builds work fairly well when using the included binaries/tools, although downloading/installing the models is a bit clunky and isn't handled in the build/install process
• CUDA backend won't even build by default purely using msys/mingw (uses nvcc on all Windows "platforms", meaning MSVC is required; didn't try hacking in llvm/clang)
• Vulkan backend required some cmake modifications to even build since it doesn't look at Vulkan's pkgconfig....then still segfaulted (may not have been whisper.cpp's fault; the suite's Vulkan build has never worked reliably for me)
• ggml libraries' naming needed to be fixed (easy - just deleting or fixing the conditionals on a few lines)
• Linking ffmpeg's config testcase against even a cpu-only whisper.cpp build coughed up a bunch of missing symbol errors - some of which were due to included pkgconfig file's link order

I just ran out of time to keep banging on it for now, so there are probably other issues at play that I didn't see.