[ENH]: add orchestrator to construct version graph for garbage collection

[codetheweb]

Description of changes

Adds an orchestrator to construct the version graph for all collections in a fork tree to be used by garbage collection.

Test plan

How are these changes tested?

• Tests pass locally with pytest for python, yarn test for js, cargo test for rust

Added tests for new orchestrator.

Documentation Changes

Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the docs section?

n/a

[codetheweb]

• [ENH]: wire GC v2 to new cleanup modes & call FinishDatabaseDeletion from garbage collector #4671
• [ENH]: soft delete databases, add FinishDatabaseDeletion gRPC method to hard delete databases #4627
• [ENH]: perform collection hard deletes from garbage collector #4605
• [ENH]: sysdb changes to support moving collection hard deletes to garbage collector #4607
• [ENH]: add validation during garbage collection for empty file paths #4586
• [ENH]: garbage collector v2 orchestrator (supports forking) #4468
• [ENH]: allow deleting v0 from version file & correctly propagate errors on DeleteCollectionVersion #4579
• [ENH]: add operator for garbage collection to list all files used by collection version #4466
• [ENH]: add operator to compute versions to garbage collect from version graph #4464
• [ENH]: add orchestrator to construct version graph for garbage collection #4463 👈 (View in Graphite)
• [ENH]: add batch get version file paths method to Sysdb #4432 : 1 other dependent PR (#4447 )
• [ENH]: SysDb should return lineage, version file paths and root collection ID on collections #4557
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[github-actions]

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

• Can you think of any use case in which the code does not behave as intended? Have they been tested?
• Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
• If appropriate, are there adequate property based tests?
• If appropriate, are there adequate unit tests?
• Should any logging, debugging, tracing information be added or removed?
• Are error messages user-friendly?
• Have all documentation changes needed been made?
• Have all non-obvious changes been commented?

System Compatibility

• Are there any potential impacts on other parts of the system or backward compatibility?
• Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

• Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)

[codetheweb]

logged graph repr can be pasted into any graphviz-compatible viewer for debugging (e.g. https://graph.flyte.org)

[propel-code-bot]

Add Orchestrator for Building Collection Version Graphs for GC

This PR introduces a new orchestrator (ConstructVersionGraphOrchestrator) for constructing a version dependency graph across all collections in a fork tree, to support garbage collection workflows. Supporting Rust operators to fetch lineage files, fetch version files, and batch-fetch version file paths are added and integrated, alongside updates to relevant orchestrator logic and supporting changes to the garbage collector pipeline, storage API, and dependencies. Extensive test coverage for the graph construction is included, validating both simple and complex collection/version lineage cases.

Key Changes:
• Added rust/garbage_collector/src/construct_version_graph_orchestrator.rs, implementing the version graph orchestrator.
• Introduced FetchLineageFileOperator, FetchVersionFileOperator (refactored), and GetVersionFilePathsOperator, each with input/output structs and error handling.
• Updated orchestrator wiring to fetch and use lineage files and version file paths for graph construction.
• Altered FetchVersionFileOutput to return parsed CollectionVersionFile objects (not just raw bytes).
• Integrations and minor refactoring in garbage_collector_orchestrator.rs and storage/lib.rs for new interfaces.
• Augmented Cargo manifests with petgraph and base64 dependencies for graph construction and debugging.
• Implemented comprehensive tests for constructing version graphs covering simple and forked histories.

Affected Areas:
• Rust garbage_collector orchestrator logic
• Rust operators for storage/sysdb fetching
• Rust storage interface
• Graph construction, lineage and version file management
• Cargo.toml, Cargo.lock (dependency management)

Potential Impact:

Functionality: Enables garbage collection logic to operate on entire version/fork trees rather than a single collection. Improves ability to trace dependencies and perform accurate collection/variant cleanup.

Performance: Slight increase in orchestrator complexity; batched fetching of version files may help performance. Additional in-memory graph processing is limited by number of collections in a fork.

Security: No new security risks introduced; new code inherits data access/authorization from existing storage and sysdb layers.

Scalability: Graph-based approach scales to arbitrary fork trees; performance may need tuning for very large trees but core approach is scalable.

Review Focus:
• Correctness of graph assembly logic and edge construction in ConstructVersionGraphOrchestrator.
• Error and edge case handling, especially with missing lineage/version files.
• Operator input/output API clarity and error management.
• Pre-existing orchestrator integrations after storage API changes.
• Test coverage adequacy and edge case completeness.

Testing Needed

• Run all garbage_collector unit/integration tests (including newly added tests for version graphs).
• Test garbage collection on collections with simple and complex ancestry (with/without lineages, forks, joins).
• Check graph serialization/debugging output for correctness and usability.
• Smoke test operator error conditions (missing files, sysdb errors, unexpected node splits, etc).

Code Quality Assessment

rust/garbage_collector/src/operators/get_version_file_paths.rs: Simple, direct batch fetching and error propagation.

rust/garbage_collector/src/garbage_collector_orchestrator.rs: Updated to match new FetchVersionFileOutput interface; residual commented-out code was removed.

rust/storage/src/lib.rs: Added Debug implementation for Storage; otherwise minimal change.

Cargo files: Dependency updates are precise and necessary for new features.

rust/garbage_collector/src/construct_version_graph_orchestrator.rs: Well-structured; uses async patterns, clear error enums, and trait-based orchestrator integration. Large, so future decomposition may help maintainability.

rust/garbage_collector/src/operators/fetch_version_file.rs: Refactored for new output types; improved error reporting. Debug implementations and API patterns follow Rust conventions.

rust/garbage_collector/src/operators/fetch_lineage_file.rs: Clean, idiomatic, covers code and decode paths. Good use of error enums.

Best Practices

Dependency Management:
• Cargo additions scoped to new features

Documentation:
• Rustdoc, struct and method comments present; some areas could be further documented (e.g. on graph invariants)

Error Handling:
• Extensive error enums and propagation
• Clear conversion between operator/task/orchestrator errors

Code Modularity:
• New operators isolated per responsibility
• Separation of orchestrator and execution logic

Testing:
• Broad, scenario-driven, and edge-case-heavy tests, including graph assertions

Potential Issues

• If collection or version lineage is partially missing, logic may terminate with error or skip nodes; correctness under partial data should be monitored.
• Performance for very large version graphs could be constrained by in-memory graph construction.
• Care should be taken if extending the graph types (e.g., status fields) to ensure consistency.

This summary was automatically generated by @propel-code-bot

[sanketkedia]

nit: should also tracing::error?

[codetheweb]

ok_or_terminate() will log any error

[codetheweb]

Merge activity

• May 23, 12:46 AM UTC: @codetheweb merged this pull request with Graphite.