Path Relativization: Metafile and Locator Write Paths #502
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ge workloads and multiple operation types per operation
…g for large workloads and multiple operation types per transaction
pdames
reviewed
Mar 10, 2025
pdames
requested changes
Mar 11, 2025
Co-authored-by: Patrick Ames <[email protected]> Signed-off-by: David Martinez <[email protected]>
Co-authored-by: Patrick Ames <[email protected]> Signed-off-by: David Martinez <[email protected]>
Co-authored-by: Patrick Ames <[email protected]> Signed-off-by: David Martinez <[email protected]>
…st_transaction.py file
pdames
approved these changes
Mar 14, 2025
There was a problem hiding this comment.
LGTM - thanks! Please address your newly added TODO in a fast-follow PR. If some of our existing code paths are already incorrectly using relative locator and/or metafile paths before calling your path relativization method, then they'll probably need to be updated to only use absolute paths.
pdames
added a commit
to pdames/deltacat
that referenced
this pull request
Mar 24, 2025
* path relativization algo, helpers, new signature for to_serializable IMPLEMENTED * relativizatio done and tested. Need to debug failing test_metafile_io.py tests * relativization done with tests and logging.debug calls * path relativization completed. Tests added, debug output removed * fixed formatting and ensured pylint passes * refactoed relativization code added additional tests to test with large workloads and multiple operation types per operation * refactored relativization code, cleaned up setter logic, added testing for large workloads and multiple operation types per transaction * Update deltacat/storage/model/transaction.py Co-authored-by: Patrick Ames <[email protected]> Signed-off-by: David Martinez <[email protected]> * Update deltacat/storage/model/transaction.py Co-authored-by: Patrick Ames <[email protected]> Signed-off-by: David Martinez <[email protected]> * Update deltacat/storage/model/transaction.py Co-authored-by: Patrick Ames <[email protected]> Signed-off-by: David Martinez <[email protected]> * Added PR review changes, updated test_abs_to_relative.py to be new test_transaction.py file * fixed delta file issue with absolute paths * cleaned code, lint --------- Signed-off-by: David Martinez <[email protected]> Co-authored-by: Patrick Ames <[email protected]>
KatiaJacoby
pushed a commit
to nsaadhvi/codebase-deltacat
that referenced
this pull request
Apr 26, 2025
* path relativization algo, helpers, new signature for to_serializable IMPLEMENTED * relativizatio done and tested. Need to debug failing test_metafile_io.py tests * relativization done with tests and logging.debug calls * path relativization completed. Tests added, debug output removed * fixed formatting and ensured pylint passes * refactoed relativization code added additional tests to test with large workloads and multiple operation types per operation * refactored relativization code, cleaned up setter logic, added testing for large workloads and multiple operation types per transaction * Update deltacat/storage/model/transaction.py Co-authored-by: Patrick Ames <[email protected]> Signed-off-by: David Martinez <[email protected]> * Update deltacat/storage/model/transaction.py Co-authored-by: Patrick Ames <[email protected]> Signed-off-by: David Martinez <[email protected]> * Update deltacat/storage/model/transaction.py Co-authored-by: Patrick Ames <[email protected]> Signed-off-by: David Martinez <[email protected]> * Added PR review changes, updated test_abs_to_relative.py to be new test_transaction.py file * fixed delta file issue with absolute paths * cleaned code, lint --------- Signed-off-by: David Martinez <[email protected]> Co-authored-by: Patrick Ames <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Implemented path relativization for paths embedded within the metafile and locator metadata of individual transaction operations. This occurs prior to serializing the transaction object to ensure relative paths are used throughout.
Rationale
The introduction of path relativization improves system portability by enabling dynamic resolution of paths relative to the catalog root. This simplifies user-specific configurations and enhances flexibility in permission management across different environments.
Changes
Modified
deltacat/storage/model/transaction.pyby implementing the relativization method. This includes the addition of operation-level logic and comprehensive error handling. Theto_serializable()method has been refactored to accept the catalog root path, providing the necessary context for the relativization process.Added a new test file
deltacat/tests/storage/model/test_abs_to_relative.py, to verify the correctness of the relativization logic. The tests include both unit tests for the helper functions and integration tests that simulate real-world transactions and operations.Impact
All tests have passed successfully, indicating that the changes do not break existing functionality. Further testing is recommended to ensure compatibility with all potential use cases and that no unintended regressions have been introduced.
Testing
deltacat/tests/storage/model/test_abs_to_relative.pywas added to validate the new relativization logic. The tests cover both the functionality of individual helper methods and full integration via mock transactions and operations.