[ENH]: add operator for garbage collection to list all files used by collection version

[codetheweb]

Description of changes

Does what the title says. Given a version file and a version, the operator constructs a complete list of files used by the collection version (including block files by traversing all sparse indices).

Test plan

How are these changes tested?

• Tests pass locally with pytest for python, yarn test for js, cargo test for rust

Tested in the next PR.

Documentation Changes

Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the docs section?

n/a

[github-actions]

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

• Can you think of any use case in which the code does not behave as intended? Have they been tested?
• Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
• If appropriate, are there adequate property based tests?
• If appropriate, are there adequate unit tests?
• Should any logging, debugging, tracing information be added or removed?
• Are error messages user-friendly?
• Have all documentation changes needed been made?
• Have all non-obvious changes been commented?

System Compatibility

• Are there any potential impacts on other parts of the system or backward compatibility?
• Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

• Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)

[codetheweb]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• [ENH]: garbage collector v2 orchestrator (supports forking) #4468
• [ENH]: add operator for garbage collection to list all files used by collection version #4466 👈 (View in Graphite)
• [ENH]: add operator to compute versions to garbage collect from version graph #4464
• [ENH]: add orchestrator to construct version graph for garbage collection #4463
• [ENH]: add batch get version file paths method to Sysdb #4432 : 1 other dependent PR (#4447 )
• [ENH]: SysDb should return lineage, version file paths and root collection ID on collections #4557
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[propel-code-bot]

Add Garbage Collection Operator for File Listing

This PR adds a new ListFilesAtVersionsOperator to enable garbage collection by constructing a complete list of files used by a specific collection version. The operator traverses sparse indices to identify all related block files, supporting the cleanup of unused files in the storage system.

Key Changes:
• Created new ListFilesAtVersionsOperator that inventories all files used by a collection version
• Implemented logic to traverse sparse indices and collect block file references
• Added Debug implementation for RootManager to support the new operator

Affected Areas:
• rust/garbage_collector/src/operators/list_files_at_version.rs
• rust/blockstore/src/arrow/provider.rs
• rust/garbage_collector/src/operators/mod.rs

This summary was automatically generated by @propel-code-bot