Add keep_max_files_sort_by option for playlist-index-based retention - #1484
Conversation
…es_sort_by config option - Add playlist_index field to DownloadMapping dataclass with backward-compatible deserialisation - Create KeepMaxFilesSortByValidator accepting "upload_date" or "playlist_index" - Register keep_max_files_sort_by in OutputOptions with default "upload_date" - Add unit tests for serialisation, deserialisation, and validator behaviour
…move_stale_files - Add sort_by parameter to remove_stale_files() with playlist_index ascending sort - None playlist indices sort last (pruned first) when mixed with populated values - Fall back to upload_date with warning when all entries have None playlist index - Wire sort_by resolution from OutputOptions through subscription_download - Add only_recent_sort_by override to "Only Recent" prebuilt preset
- Verify default sort_by="upload_date" keeps most recent N entries by date descending - Verify old archives without playlist_index sort identically to upload_date mode - Verify keep_max_files=0 skips pruning entirely
|
Most of the failing tests are due to hash mismatch of the download archive file. We will need to update all of them via setting the REGENERATE_FIXTURES global variable to True, rerunning, and committing. I can help with this and the integration test if you'd like. |
|
Also, I feel we should make the order support both playlist_index ascending and descending. Right now from the doc:
It seems its playlist_index ASC. I see both cases where authors add new vids to playlists at the front or back. We should support the DESC case too and make it explicit in the name. |
…t orders - Rename 'playlist_index' to 'playlist_index_asc' and add 'playlist_index_desc' for explicit ascending/descending playlist index sorting - Regenerate test fixtures to include new keep_max_files_sort_by field - Regenerate plugins.rst from docstrings - Add descending sort tests and update existing tests
|
Thanks @jmbannon - regenerated the hashes and added support for both directions for |
…x ruff formatting
…rm-specific media hashes
ffmpeg on Windows does not produce bit-identical output even with -bitexact, causing unchanged media files to appear as 'modified' in the transaction log during reformat steps. This matches the existing pattern in expected_download.py which skips all MD5 hash checks on Windows. Also improves the assertion error message to show actual vs expected line counts instead of just 'True'.
|
sorry about that @jmbannon - broke it. again in a similar but different way. have tested it out in my fork and made sure all tests are passing this time - https://github.com/michaeldyrynda/ytdl-sub/pull/1/checks i also added a fix for the windows failures in ecba1d3 - same fix as is used in looks like ffmpeg isn't deterministic on Windows, which causes the ordering to change. if you'd prefer those failing tests stay in place, let me know and i'll drop the commit. |
|
Ran a few of the disabled e2e tests manually, notably |
2 participants
Adds a
keep_max_files_sort_byconfig option that controls how entries are ordered when pruning withkeep_max_files.Previously, pruning always sorted by
upload_date(keeping the most recent). This adds support for sorting byplaylist_index(keeping the lowest indices), which is useful for playlists where position matters more than upload date e.g. keeping the first N episodes of a series.Changes
keep_max_files_sort_by- newoutput_optionsfield accepting"upload_date"(default, preserving existing behaviour) or"playlist_index"playlist_indexonDownloadMapping- persisted in the archive JSON so it's available at prune time; old archives without the field gracefully default toNonesort_byis"playlist_index"but no entries have an index (e.g. old archives), logs a warning and falls back toupload_date(current behaviour)only_recent_sort_byoverride wired into thedownload_deletion_optionshelperTest plan
keep_max_files_sort_by: "playlist_index"Closes #1461