Feature Request: Automatic deletion of previously mirrored files excluded by filtering configuration

[naveen1583]

Summary

Currently, when filtering configuration is changed to exclude certain files (e.g., using exclude_platform to filter out old Python versions), bandersnatch does not automatically delete previously mirrored files that are now excluded by the active filters. This results in wasted storage space and creates a mismatch between user expectations and actual mirror contents.

Problem

When users update their filtering configuration to be more restrictive (for example, excluding packages for Python 2.7 or old platform-specific wheels), they expect that subsequent mirror synchronizations would remove files that no longer match the active filter criteria. However, bandersnatch currently only prevents new downloads of filtered content but leaves previously downloaded files in place.

This leads to:

1. Wasted storage space - Old files that are no longer needed continue to consume disk space
2. Inconsistent mirror state - The mirror contains files that would not be downloaded if starting fresh with the current configuration
3. Manual cleanup burden - Users must manually identify and delete files, which is error-prone and time-consuming

Proposed Solution

Add functionality to automatically delete previously mirrored files that are now excluded by the active filtering configuration. This could be implemented as:

1. Automatic deletion during sync - An optional configuration flag (e.g., cleanup_filtered_files = true) that enables automatic removal of files excluded by current filters during the mirror synchronization process

2. Separate cleanup command - A dedicated command (e.g., bandersnatch cleanup-filtered) that scans the mirror and removes files that don't match the current filter configuration

Use Cases

• Filtering old Python versions (e.g., Python 2.7 packages) to save storage after they're no longer needed
• Excluding specific platforms using exclude_platform filters
• Adjusting package allowlists/blocklists and wanting the mirror to reflect those changes
• Applying regex or metadata filters and needing historical files to be cleaned up

Benefits

• Automatic storage optimization
• Mirror state consistent with current configuration
• Reduced manual maintenance overhead
• User expectations met when changing filter settings

[cooperlees]

Howdy,

Happy to support/enhance this, as long as a contributor is open to decent test coverage and adding to our integration testing (here)[https://github.com/pypa/bandersnatch/blob/main/test_runner.py]. The separate command partially exists, but I don't know how well it's done to cover new plugins (so I'd love testing help) via:

• bandersnatch verify --delete
  • This is slow as it loads all metadata + trawls the file system - So is super resource heavy ...
  • We could batch this more or something too - totally happen for that

The automatic deleteion, I would like this to be config controlled like you said, as it will add time to sync runs and I'd like people to opt into that. We need to talk about how we'd do it, and I can't think off the top of my head an efficient way (and thus why I've only done it via verify today) without PyPI metadata help or keeping a local cache or what's installed - e.g. sqlite database etc. etc.

So any ideas on how this would be implemented?