feat: generate an Symlink Manifest for External Engines #3889
Conversation
github-actions
bot
added
binding/python
|
ACTION NEEDED delta-rs follows the Conventional Commits specification for release automation. The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification. |
rtyler
changed the title
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #3889 +/- ##
==========================================
- Coverage 73.76% 73.75% -0.02%
==========================================
Files 151 152 +1
Lines 39396 39447 +51
Branches 39396 39447 +51
==========================================
+ Hits 29061 29094 +33
- Misses 9023 9036 +13
- Partials 1312 1317 +5 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Signed-off-by: JustinRush80 <[email protected]>
JustinRush80
force-pushed
the
feat/generate_manifest
branch
from
d87bdc0 to
8542271
Compare
Signed-off-by: JustinRush80 <[email protected]>
Signed-off-by: JustinRush80 <[email protected]>
Signed-off-by: JustinRush80 <[email protected]>
I do not have a great test environment to validate that the manifests are _correct_ so that they can be used by an engine like Presto. As such we will have to depend on users to validate this Signed-off-by: R. Tyler Croy <[email protected]>
rtyler
force-pushed
the
feat/generate_manifest
branch
from
6064ea0 to
5269e97
Compare
rtyler
requested review from
hntd187,
ion-elgreco,
roeap and
rtyler
as code owners
|
@JustinRush80 I have finished this functionality but would request that you test the generated manifest with an engine that requires it. @roeap this is ready to merge once it's green in CI |
Sure thing! Is this functionality able to incrementally update the manifest? I am still learning the code base but how does it handle partitions? |
|
@JustinRush80 I have no idea what partitions are supposed to look like, if there is a specification of what this manifest is supposed to look like, I'm happy to help make this implementation compliant |
Pull request was converted to draft
…h80/delta-rs into feat/generate_manifest
…ustinRush80/delta-rs into feat/generate_manifest" This reverts commit d972b2a, reversing changes made to 5269e97.
2 participants
Description
Add the GENERATE command that create an symlink manifest.
I have done an very dumb solution for my day job but it definitely not scalable. So creating this draft to first understand the expected behavior of this command and then turn my initial solution to fit the code standard.
Related Issue(s)
Documentation
https://books.japila.pl/delta-lake-internals/post-commit-hooks/GenerateSymlinkManifest/