Upgrading ffmpeg to v8 breaks integration tests' hash checks [NixOS]

[miangraham]

Nixpkgs recently bumped the default version of ffmpeg from 7 to 8 and broke ytdl-sub with the change. It's easily fixed by pinning to ffmpeg 7 so no big deal for now, but rolling release distros will start to trip on it if they haven't already, and eventually it'll make its way to Ubuntu and blow up your CI tests too. So I guess this is an early warning. :)

Here's the actual log with test failures on nixpkgs' build infra: https://hydra.nixos.org/build/313562159/nixlog/2

=========================== short test summary info ============================
FAILED tests/integration/cli/test_dl.py::TestCliDl::test_cli_dl_command[False-True] - AssertionError: MD5 hash for JMC/Mock Entry 20-1.mp4 does not match: 0558be...
FAILED tests/integration/plugins/test_chapters.py::TestChapters::test_chapters_from_comments[False] - AssertionError: MD5 hash for JMC/Mock Entry 20-1.mp4 does not match: de48fe...
FAILED tests/integration/plugins/test_output_options.py::TestOutputOptions::test_download_archive_migration - AssertionError: MD5 hash for Season 2020/s2020.e000001 - Mock Entry 20-3.mp...
FAILED tests/integration/plugins/test_output_options.py::TestOutputOptions::test_empty_info_json_and_thumb[False] - AssertionError: MD5 hash for Season 2020/s2020.e000001 - Mock Entry 20-3.mp...
FAILED tests/integration/plugins/test_output_options.py::TestOutputOptions::test_missing_thumbnail - AssertionError: MD5 hash for Season 2020/s2020.e000001 - Mock Entry 20-3.mp...
FAILED tests/integration/prebuilt_presets/test_tv_show_by_date.py::TestPrebuiltTVShowPresets::test_tv_show_presets[Kodi TV Show by Date] - AssertionError: MD5 hash for Best Prebuilt TV Show by Date/Season 2020/s202...
FAILED tests/integration/prebuilt_presets/test_tv_show_by_date.py::TestPrebuiltTVShowPresets::test_tv_show_presets[Jellyfin TV Show by Date] - AssertionError: MD5 hash for Best Prebuilt TV Show by Date/Season 2020/s202...
FAILED tests/integration/prebuilt_presets/test_tv_show_by_date.py::TestPrebuiltTVShowPresets::test_tv_show_presets[Plex TV Show by Date] - AssertionError: MD5 hash for Best Prebuilt TV Show by Date/Season 2020/s202...
FAILED tests/integration/prebuilt_presets/test_tv_show_by_date.py::TestPrebuiltTVShowPresets::test_tv_show_presets[Emby TV Show by Date] - AssertionError: MD5 hash for Best Prebuilt TV Show by Date/Season 2020/s202...
FAILED tests/integration/prebuilt_presets/test_tv_show_by_date.py::TestPrebuiltTVShowPresets::test_episode_ordering_presets[upload-year-upload-month-day] - AssertionError: MD5 hash for Best Prebuilt TV Show by Date/Season 2020/s202...
FAILED tests/integration/prebuilt_presets/test_tv_show_by_date.py::TestPrebuiltTVShowPresets::test_episode_ordering_presets[upload-year-upload-month-day-reversed] - AssertionError: MD5 hash for Best Prebuilt TV Show by Date/Season 2020/s202...
FAILED tests/integration/prebuilt_presets/test_tv_show_by_date.py::TestPrebuiltTVShowPresets::test_episode_ordering_presets[upload-year-download-index] - AssertionError: MD5 hash for Best Prebuilt TV Show by Date/Season 2020/s202...
FAILED tests/integration/prebuilt_presets/test_tv_show_by_date.py::TestPrebuiltTVShowPresets::test_episode_ordering_presets[upload-year-month-upload-day] - AssertionError: MD5 hash for Best Prebuilt TV Show by Date/Season 202008/s2...
FAILED tests/integration/prebuilt_presets/test_tv_show_by_date.py::TestPrebuiltTVShowPresets::test_episode_ordering_presets[release-year-release-month-day] - AssertionError: MD5 hash for Best Prebuilt TV Show by Date/Season 2000/s200...
FAILED tests/integration/prebuilt_presets/test_tv_show_by_date.py::TestPrebuiltTVShowPresets::test_episode_ordering_presets[release-year-release-month-day-reversed] - AssertionError: MD5 hash for Best Prebuilt TV Show by Date/Season 2000/s200...
FAILED tests/integration/prebuilt_presets/test_tv_show_by_date.py::TestPrebuiltTVShowPresets::test_episode_ordering_presets[release-year-download-index] - AssertionError: MD5 hash for Best Prebuilt TV Show by Date/Season 2000/s200...
FAILED tests/integration/prebuilt_presets/test_tv_show_by_date.py::TestPrebuiltTVShowPresets::test_episode_ordering_presets[release-year-month-release-day] - AssertionError: MD5 hash for Best Prebuilt TV Show by Date/Season 200008/s2...
FAILED tests/integration/prebuilt_presets/test_tv_show_collection.py::TestPrebuiltTvShowCollectionPresets::test_tv_show_collection_presets[season_indices0-Jellyfin TV Show Collection] - AssertionError: MD5 hash for Best Prebuilt TV Show Collection/Season 01/s01...
FAILED tests/integration/prebuilt_presets/test_tv_show_collection.py::TestPrebuiltTvShowCollectionPresets::test_tv_show_collection_presets[season_indices0-Plex TV Show Collection] - AssertionError: MD5 hash for Best Prebuilt TV Show Collection/Season 01/s01...
FAILED tests/integration/prebuilt_presets/test_tv_show_collection.py::TestPrebuiltTvShowCollectionPresets::test_tv_show_collection_presets[season_indices0-Emby TV Show Collection] - AssertionError: MD5 hash for Best Prebuilt TV Show Collection/Season 01/s01...
FAILED tests/integration/prebuilt_presets/test_tv_show_collection.py::TestPrebuiltTvShowCollectionPresets::test_tv_show_collection_presets[season_indices0-Kodi TV Show Collection] - AssertionError: MD5 hash for Best Prebuilt TV Show Collection/Season 01/s01...
FAILED tests/integration/prebuilt_presets/test_tv_show_collection.py::TestPrebuiltTvShowCollectionPresets::test_tv_show_collection_presets[season_indices1-Jellyfin TV Show Collection] - AssertionError: MD5 hash for Best Prebuilt TV Show Collection/Season 01/s01...
FAILED tests/integration/prebuilt_presets/test_tv_show_collection.py::TestPrebuiltTvShowCollectionPresets::test_tv_show_collection_presets[season_indices1-Plex TV Show Collection] - AssertionError: MD5 hash for Best Prebuilt TV Show Collection/Season 01/s01...
FAILED tests/integration/prebuilt_presets/test_tv_show_collection.py::TestPrebuiltTvShowCollectionPresets::test_tv_show_collection_presets[season_indices1-Emby TV Show Collection] - AssertionError: MD5 hash for Best Prebuilt TV Show Collection/Season 01/s01...
FAILED tests/integration/prebuilt_presets/test_tv_show_collection.py::TestPrebuiltTvShowCollectionPresets::test_tv_show_collection_presets[season_indices1-Kodi TV Show Collection] - AssertionError: MD5 hash for Best Prebuilt TV Show Collection/Season 01/s01...
FAILED tests/integration/prebuilt_presets/test_tv_show_collection.py::TestPrebuiltTvShowCollectionPresets::test_episode_ordering_presets[season_indices0-upload-year-month-day] - AssertionError: MD5 hash for Best Prebuilt TV Show Collection/Season 01/s01...
FAILED tests/integration/prebuilt_presets/test_tv_show_collection.py::TestPrebuiltTvShowCollectionPresets::test_episode_ordering_presets[season_indices0-upload-year-month-day-reversed] - AssertionError: MD5 hash for Best Prebuilt TV Show Collection/Season 01/s01...
FAILED tests/integration/prebuilt_presets/test_tv_show_collection.py::TestPrebuiltTvShowCollectionPresets::test_episode_ordering_presets[season_indices0-release-year-month-day] - AssertionError: MD5 hash for Best Prebuilt TV Show Collection/Season 01/s01...
FAILED tests/integration/prebuilt_presets/test_tv_show_collection.py::TestPrebuiltTvShowCollectionPresets::test_episode_ordering_presets[season_indices0-release-year-month-day-reversed] - AssertionError: MD5 hash for Best Prebuilt TV Show Collection/Season 01/s01...
FAILED tests/integration/prebuilt_presets/test_tv_show_collection.py::TestPrebuiltTvShowCollectionPresets::test_episode_ordering_presets[season_indices0-playlist-index] - AssertionError: MD5 hash for Best Prebuilt TV Show Collection/Season 01/s01...
FAILED tests/integration/prebuilt_presets/test_tv_show_collection.py::TestPrebuiltTvShowCollectionPresets::test_episode_ordering_presets[season_indices0-playlist-index-reversed] - AssertionError: MD5 hash for Best Prebuilt TV Show Collection/Season 01/s01...
FAILED tests/integration/prebuilt_presets/test_tv_show_collection.py::TestPrebuiltTvShowCollectionPresets::test_episode_ordering_presets[season_indices1-upload-year-month-day] - AssertionError: MD5 hash for Best Prebuilt TV Show Collection/Season 01/s01...
FAILED tests/integration/prebuilt_presets/test_tv_show_collection.py::TestPrebuiltTvShowCollectionPresets::test_episode_ordering_presets[season_indices1-upload-year-month-day-reversed] - AssertionError: MD5 hash for Best Prebuilt TV Show Collection/Season 01/s01...
FAILED tests/integration/prebuilt_presets/test_tv_show_collection.py::TestPrebuiltTvShowCollectionPresets::test_episode_ordering_presets[season_indices1-release-year-month-day] - AssertionError: MD5 hash for Best Prebuilt TV Show Collection/Season 01/s01...
FAILED tests/integration/prebuilt_presets/test_tv_show_collection.py::TestPrebuiltTvShowCollectionPresets::test_episode_ordering_presets[season_indices1-release-year-month-day-reversed] - AssertionError: MD5 hash for Best Prebuilt TV Show Collection/Season 01/s01...
FAILED tests/integration/prebuilt_presets/test_tv_show_collection.py::TestPrebuiltTvShowCollectionPresets::test_episode_ordering_presets[season_indices1-playlist-index] - AssertionError: MD5 hash for Best Prebuilt TV Show Collection/Season 01/s01...
FAILED tests/integration/prebuilt_presets/test_tv_show_collection.py::TestPrebuiltTvShowCollectionPresets::test_episode_ordering_presets[season_indices1-playlist-index-reversed] - AssertionError: MD5 hash for Best Prebuilt TV Show Collection/Season 01/s01...
=== 37 failed, 916 passed, 23 deselected, 762 warnings in 543.68s (0:09:03) ====

All I see is hash failures, so my guess is things are mostly functional and you've just got small changes in the args or output between ffmpeg v7 and v8 that need accounting for. But I was focused on getting the packaging fixed so I haven't looked closely for the underlying cause.

[jmbannon]

Yeah hashing has always been tricky dealing with ffmpeg versions, different OSs, etc. Honestly I'm not familiar with the nixOS ecosystem and am quite surprised to see CI tests run for it! But you're right that there could be some better consistency with ffmpeg versions across all the builds.

Ideally we should be using yt-dlp's fork of ffmpeg across all docker images, CI, etc: https://github.com/yt-dlp/FFmpeg-Builds

[miangraham]

Yep, for nixpkgs builds all python tests get run by default and can be turned off piecemeal if they're needlessly blocking things. Which would probably be fine to do here, but it seemed good to keep them while it's easy.

Completely agreed that it would be better to standardize on the yt-dlp fork's builds. Nixpkgs' yt-dlp setup seems to ignore those at present and I'm not sure if there's a reason for it. Not too familiar with how it got put together, just visiting the build because my server was unhappy. :)

Do you know anyone running ytdl-sub natively on Arch? I think they're affected as well judging by the ffmpeg version, but maybe there are some workarounds I'm not seeing. Nevermind, they don't block packaging on tests so it's probably not an issue for anyone not choosing to run them, assuming the mp4 output is functional despite the byte diff.

Just to be clear: There's no issue here for anyone willing and able to hang out on ffmpeg 7, and I don't think nixpkgs is likely to drop support for several years. They've still got stuff building with v4 and v6 in there. If you're comfortable letting CI pick your version, I think you're at least half a year away from the GHA Ubuntu version switching to a new LTS that might cause trouble.