Path Relativization: Metafile and Locator Write Paths #502
+370
−4
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ge workloads and multiple operation types per operation
…g for large workloads and multiple operation types per transaction
pdames
reviewed
Mar 10, 2025
pdames
requested changes
Mar 11, 2025
Co-authored-by: Patrick Ames <[email protected]> Signed-off-by: David Martinez <[email protected]>
Co-authored-by: Patrick Ames <[email protected]> Signed-off-by: David Martinez <[email protected]>
Co-authored-by: Patrick Ames <[email protected]> Signed-off-by: David Martinez <[email protected]>
…st_transaction.py file
pdames
approved these changes
Mar 14, 2025
There was a problem hiding this comment.
LGTM - thanks! Please address your newly added TODO in a fast-follow PR. If some of our existing code paths are already incorrectly using relative locator and/or metafile paths before calling your path relativization method, then they'll probably need to be updated to only use absolute paths.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Implemented path relativization for paths embedded within the metafile and locator metadata of individual transaction operations. This occurs prior to serializing the transaction object to ensure relative paths are used throughout.
Rationale
The introduction of path relativization improves system portability by enabling dynamic resolution of paths relative to the catalog root. This simplifies user-specific configurations and enhances flexibility in permission management across different environments.
Changes
Modified
deltacat/storage/model/transaction.pyby implementing the relativization method. This includes the addition of operation-level logic and comprehensive error handling. Theto_serializable()method has been refactored to accept the catalog root path, providing the necessary context for the relativization process.Added a new test file
deltacat/tests/storage/model/test_abs_to_relative.py, to verify the correctness of the relativization logic. The tests include both unit tests for the helper functions and integration tests that simulate real-world transactions and operations.Impact
All tests have passed successfully, indicating that the changes do not break existing functionality. Further testing is recommended to ensure compatibility with all potential use cases and that no unintended regressions have been introduced.
Testing
deltacat/tests/storage/model/test_abs_to_relative.pywas added to validate the new relativization logic. The tests cover both the functionality of individual helper methods and full integration via mock transactions and operations.