[sdlf-datalakelibrary] simplified datalake-library #540
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dlpzx
force-pushed
the
fix/datalake-library-import
branch
3 times, most recently
from
a0ab49f to
f3141ad
Compare
dlpzx
force-pushed
the
fix/datalake-library-import
branch
from
f3141ad to
5758a7d
Compare
kukushking
reviewed
Aug 14, 2025
| @@ -1,3 +1,4 @@ | |||
| # [**DEPRECATED**] | |||
kukushking
reviewed
Aug 14, 2025
| sqs_interface = SQSInterface(sqs_config.get_stage_dlq_name) | ||
|
|
||
| client = DataLakeClient( | ||
| team=event["body"]["team"], pipeline=event["body"]["pipeline"], stage=event["body"]["pipeline_stage"] |
There was a problem hiding this comment.
fw these keys might not exist in the event
Contributor
Author
There was a problem hiding this comment.
how? we send the event from the routing lambda with those inputs in the event.body
kukushking
reviewed
Aug 14, 2025
| local_path = client.s3.download_object(bucket, key) | ||
|
|
||
| # Apply business business logic: | ||
| # Apply business logic: |
There was a problem hiding this comment.
qq - previously datalake library also contained transformation business logic. is the user expected to extend the library or logic has completely moved here?
Contributor
Author
There was a problem hiding this comment.
the transformation logic was removed in previous PRs. I am assessing the best way to allow customers to add their custom transformation logic
kukushking
approved these changes
Aug 14, 2025
dlpzx
merged commit 3ba3fb4
into
aws-solutions-library-samples:main
2 of 3 checks passed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue #, if available:
Running the workshop following the steps in https://catalog.us-east-1.prod.workshops.aws/workshops/501cb14c-91b3-455c-a2a9-d0a21ce68114/en-US/20-production results in the stageA and stageB failure (in the routing Lambdas). The initial error is described here: #538
Taking a closer look at the issue the
datalake_libraryand the Lambdas is larger than just some imports:Those are bugs, on top of it the architecture of the datalake_library is not clean, there are unused methods, mix of interfaces, configs.
Next steps:
Description of changes:
datalake_library--> read description of the layer in datalake_library/READMEDataLakeClientTesting
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.