[Feature Request] image sequences, merging video sources, logo overlay, custom `filter_complex/vf`

[jo-chemla]

Hi there,
Want to say thanks for this great utility, I've sincerely been exploring dozens of ffmpeg GUIs/wrappers (handbrake, videomass, shutter-encoder, one-person utilities etc) over the past few years, and your shot clearly targets both novice and experts with classic/reproducible needs. Custom presets is also a great way to share some configs, thanks for having this feature already baked-in!

Here are a few missing feature requests that could be worth mentioning - just saw you're targeting working back at it in a few weeks, hope some of these make sense:

• image sequences as input (or output) - as mentioned in Support for images and audio manipulation and conversion #31 - would indeed be a game changer for compositing 3D render frames sequences, usually requires the user to tell the sequence framerate, eventually start/end frame numbers.
• adding multiple videos sources and merging/appending them into a single video file. Maybe simply add an option to concatenate all videos of the sources list?
• burning/overlaying logo would also be useful, at a given position
• custom filter_complex could be great for expert users to customize the ffmpeg command.
• 360 metadata burning via Spatial Media Metadata Injector would also be great, might require an external dependency, not sure if that's the mindset of Frame

It's possible the custom filter_complex command could fulfill all of these needs for experts. Sharing below some of the filters we use at @Iconem in case it's helpful, and again thanks for the amazing work on Frame!

-vf "v360=equirect:c3x2:out_forder=frblud,untile=3x2"
-vf "v360=e:rectilinear:h_fov=90:v_fov=90"
-filter_complex "[0] extractplanes=r+g[r][g] ; [1][r][g]remap "
-vf "movie=logo_white_h80px.png [watermark]; [in][watermark] overlay=main_w-overlay_w-50:main_h-overlay_h-50:enable='between(t,1,2)' [out]"

[jo-chemla]

By the way, I just published FrameGUI to winget-pkgs repository, which means it's now easy to install on any windows via winget install 66HEX.Frame command see here microsoft/winget-pkgs#352184

It might be useful to add a github action workflow to this repo following publish, which would automatically publish the updated winget package.

The standard way to do it is to use winget-releaser (which uses Komac). It requires a Classic Github Personal Access Token with public_repo scope is created, following this link, then the Token can be added to the repo as a secret named WINGET_ACC_TOKEN. See below, that user also will have to fork the winget-pkgs repository. Can do a quick PR if you want me to, with this workflow

name: Publish release to WinGet
on:
  release:
    types: [released]
jobs:
  publish:
    runs-on: windows-latest
    steps:
      - uses: vedantmgoyal9/winget-releaser@latest
        with:
          identifier: 66HEX.Frame
          installers-regex: 'x64\-setup\.exe'
          token: ${{ secrets.WINGET_ACC_TOKEN }}

Notes:
You will need to create a classic Personal Access Token (PAT) with public_repo scope. New fine-grained PATs aren't supported by the action.
Fork microsoft/winget-pkgs under the same account/organization as the project's repository. If you are forking winget-pkgs on a different account (e.g. bot/personal account), you can use the fork-user input to specify the username of the account where the fork is present.

[66HEX]

Thanks for that Winget thing!

As for logo overlays - it should be added in a couple of days - I already migrated preview from HTMLVideoElement to PixiJS based infinite canvas + added zoom controls so the whole positioning flow will be smooth!

[jo-chemla]

Nice to see winget-pkgs now always includes latest Frame release since you've deployed the github action!

Thanks also for getting back regarding the logo overlay burning, great to hear it's coming thanks to PixiJS - hopefully it won't be too hard to map the placement setup from pixijs to ffmpeg video filter. This PixiJS based preview could also open the pandora box to more niche use-cases like burning the webcam bubble - the way oss screen-recorders let you do it like OpenScreen, cap.so or recordly - note some do have a nice UX for handling simple timelines.

Also, you always show a very neat attention to details, really great seeing how the added controls (zooms/transforms) bend smoothly within the existing UI.

Curious how your timeline experiment went along and whether you would be pushing forward multiple video concatenation as well as image sequence inputs at some point!

[66HEX]

The timelines experiment probably won’t make it into production. I’ve come to the conclusion that it would be too big a shift in the nature of the entire app, and I think most people actually like Frame as a converter rather than a full-fledged editor. Plus, I don’t have any idea how to sensibly integrate this feature into the app’s current workflow. If you have any suggestions, I’m of course open to them.

As for Pixi.js - yes, it opens up a lot of possibilities. In the latest release, HTMLVideoElement will still handle the decoding itself, though it will be hidden, and the frames will be sent to Sprites in pixi.

In the future, I plan to decode/demux media on the Rust side and send the data from there to the front end, likely via WebSocket. Thanks to this, I hope to fix all the GStreamer/Tauri-related issues on Linux once and for all.

[66HEX]

This is what the overlay setup process looks like right now.

GitHub.mp4

Mapping to the canvas => ffmpeg is already working correctly. I’ll probably have to remove those unnecessary tooltips and thoroughly test the renders in various settings, but everything is almost ready ;)

[jo-chemla]

That logo overlay flow looks great, thanks for sharing.

Regarding HTMLVideoElement, pixijs and especially the rust demuxing plan, I just wanted to mention MediaBunny (repo) which has proven great on our side to play (encode/decode) videos of any encoding format hw-accelerated, enabling frame-perfect navigation etc.

Remotion is built on top to make videos programmatically (probably could handle some features like logo-overlay etc). It is react-based initially, but also can be integrated into a svelte project, with a player and studio.

These might be too heavy for your use-case - mediabunny probably is the closest to what could be useful - just thought they were worth mentioning!

[66HEX]

I know mediabunny, but issue with that solution is that it depends on WebCodecs API, so I would still have to bundle Frame with Gstreamer. I had various issues with that on Linux distros AppImages. I already tried many fixes, yet still issue persists.

I see two ways to fix this:
-wait till Tauri CEF becomes stable and production ready
-move to canvas preview with backend demuxing/decoding based on how Cap Screen Recorder handles it

While Linux isn't my priority, I kept it in back of my mind during the refactoring. Maybe one day the time will come when I'll really get down to it