[ENH]: add operator to compute versions to garbage collect from version graph

[codetheweb]

Description of changes

Adds an operator to compute which versions to garbage collect from a version graph. Intended to replace rust/garbage_collector/src/operators/compute_versions_to_delete.rs.

Test plan

How are these changes tested?

• Tests pass locally with pytest for python, yarn test for js, cargo test for rust

Added new tests for operator.

Documentation Changes

Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the docs section?

n/a

[github-actions]

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

• Can you think of any use case in which the code does not behave as intended? Have they been tested?
• Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
• If appropriate, are there adequate property based tests?
• If appropriate, are there adequate unit tests?
• Should any logging, debugging, tracing information be added or removed?
• Are error messages user-friendly?
• Have all documentation changes needed been made?
• Have all non-obvious changes been commented?

System Compatibility

• Are there any potential impacts on other parts of the system or backward compatibility?
• Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

• Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)

[codetheweb]

• [ENH]: wire GC v2 to new cleanup modes & call FinishDatabaseDeletion from garbage collector #4671
• [ENH]: soft delete databases, add FinishDatabaseDeletion gRPC method to hard delete databases #4627
• [ENH]: perform collection hard deletes from garbage collector #4605
• [ENH]: sysdb changes to support moving collection hard deletes to garbage collector #4607
• [ENH]: add validation during garbage collection for empty file paths #4586
• [ENH]: garbage collector v2 orchestrator (supports forking) #4468
• [ENH]: allow deleting v0 from version file & correctly propagate errors on DeleteCollectionVersion #4579
• [ENH]: add operator for garbage collection to list all files used by collection version #4466
• [ENH]: add operator to compute versions to garbage collect from version graph #4464 👈 (View in Graphite)
• [ENH]: add orchestrator to construct version graph for garbage collection #4463
• [ENH]: add batch get version file paths method to Sysdb #4432 : 1 other dependent PR (#4447 )
• [ENH]: SysDb should return lineage, version file paths and root collection ID on collections #4557
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[Sicheng-Pan]

this implies we never delete the first version of the new collection after fork, and thus we can never gc the data of the source collection at the forking version?

[codetheweb]

correct
I did this to preserve the existing behavior of GC but I'm fine changing it to allow deleting the first version

[propel-code-bot]

Add Operator for Computing Versions to Delete from Version Graph

This PR introduces a new operator, ComputeVersionsToDeleteOperator, designed to determine which collection versions can be garbage collected by analyzing a version graph. It encapsulates logic for selecting deletable versions based on configurable parameters like cutoff time and a minimum number of versions to retain, and includes comprehensive tests for different graph scenarios (linear, forked trees) to validate correct behavior.

Key Changes:
• Introduced compute_versions_to_delete_from_graph.rs implementing the ComputeVersionsToDeleteOperator and related data structures.
• Logic to traverse the version graph, evaluating and marking alive collection versions for deletion or retention based on cutoff_time and min_versions_to_keep.
• Ensured that version 0 is always retained, preserving current GC behavior after forks.
• Added error handling with a custom error enum (ComputeVersionsToDeleteError).
• Implemented and registered the new operator module in mod.rs.
• Included thorough unit tests covering linear and forked version graph topologies.

Affected Areas:
• rust/garbage_collector/src/operators/compute_versions_to_delete_from_graph.rs
• rust/garbage_collector/src/operators/mod.rs

This summary was automatically generated by @propel-code-bot

[codetheweb]

Merge activity

• May 23, 1:25 AM UTC: @codetheweb merged this pull request with Graphite.