Serverless Data Lake Framework 2.0.0

[cnfait]

Serverless Data Lake Framework 2.0.0 is now available.

The workshop has been updated.

For users of SDLF 1.x, version 1 is still available on the master branch. Development of newer versions of SDLF (2.x) happens on branch main. The workshop still contains sections for version 1 as well.

GPG signature files are included for archives in this release's assets files. You can use our public key to verify that the downloaded archive file is original and unmodified.

We welcome your feedback!

What’s New

SDLF 2.0 is very much in the same spirit as SDLF 1.0 - the constructs are the same and CloudFormation is still used as the language for provisioning the infrastructure. 2.0 intends to fix long-standing issues with SDLF, extend its usage to more data architecture patterns and bring commonly-asked features to the framework.

• SDLF components are now CloudFormation modules
  • there is one module per component: foundations, team, pipeline, stageA, stageB, dataset.
  • datalakeLibrary is used to build a Lambda layer, it is not a CloudFormation module.
  • deploy.sh takes care of deploying the CICD infrastructure used to build these modules, and register them in the private CloudFormation registry of each account. Modules are updated whenever there is a deployment that requires them.
• SDLF CICD pipelines now live in the Shared DevOps account
  • CloudFormation stacks are created in child accounts through crossaccount IAM roles.
• SDLF can deploy an arbitrary number of child accounts driven from a single devops account.
  • pDomain (which defaults to datalake) can be provided when deploying foundations.
  • each domain can have the usual three environments (dev, test, prod).
• Deploying foundations and teams is now done from a new repository called sdlf-main.
  • this repository is created during the initial setup with deploy.sh.
  • foundations and teams deployment happens in datadomain-{domain}-{env}.yaml.
  • sdlf-main works the same way everything works in SDLF - main, test and dev branches are expected.
  • it is easier to know which teams have been created, and to remove them as they don’t share the same set of parameters in parameters-{env}.json.
• Deploying pipelines and datasets is now done from a new repository called sdlf-main-{domain}-{team}.
  • this repository is created when a new team is created.
  • pipelines deployment happens in pipelines.yaml and datasets in datasets.yaml.
  • sdlf-main-{domain}-{team} works the same way everything works in SDLF - main, test and dev branches are expected.
  • it is easier to know which pipelines and datasets have been created, and to remove them as they don’t share the same set of parameters in parameters-{env}.json.
• Mappings between datasets and transforms in stageB is done directly when defining a dataset.
  • this mapping used to be done by a CodeBuild project and a script in sdlf-datalakeLibrary. They are no longer needed and have been removed.
  • it is now defined through the pPipelineDetails parameter when defining a dataset in sdlf-dataset. This parameter goes even further and can be used to store more information that stages can use. These details are stored in the Datasets DynamoDB table (as was already the case in SDLF 1.x).
• Stages in a pipeline are now driven by EventBridge rules exclusively.
  • the rule can be an event pattern or a schedule (cron expression).
  • stageA is no longer sending messages to a queue for stageB to process. StageB is configured with an event pattern to listen for stageA runs (pEventPattern in the example), and then process these events on a schedule (pSchedule)
  • it is easier now to have pipelines with a single stage, pipelines with dependent stages and overall more complex pipelines than in SDLF 1.x, as long as there is an event pattern to listen for.
• VPC support is available as an optional feature.
• sdlf-pipLibrary is now part of an optional feature called Lambda Layer Deployer. Files related to this feature are part of a team main repository (sdlf-main-{domain}-{team}) under the layers folder.
• The Glue Job Deployer is now an optional feature built in SDLF rather than an add-on from sdlf-utils. Files related to this feature are part of a team main repository (sdlf-main-{domain}-{team}) under the transforms folder.
• New optional component: sdlf-monitoring, with CloudTrail, ELK forwarding and SNS.
  • in SDLF 1.x Cloudtrail is optional but enabled by default. Here it is optional and not enabled as long as sdlf-monitoring is not deployed.
• Deequ support has been removed entirely. While it wasn’t enabled by default in SDLF 1.x, dedicated infrastructure was still created while deploying sdlf-foundations. This is no longer the case.
  • sdlf-stage-dataquality will soon be available as an example on how to add a third stage making use of Glue Data Quality.
• Outside the initial deploy.sh, there is no more shell scripts.

Full Changelog: 1.5.2...2.0.0

New Contributors

• @tomaszwrzonski made their first contribution in CloudFormationManagedUploadInfrastructure now requires s3:PutBucketPu… #143
• @Mreddy19 made their first contribution in Python 3.11 as default for CodeBuild runtimes. Also update Codebuild image to latest AmazonLinux2 #192
• @gwenika made their first contribution in create an athena workgroup per team #208

Thanks

We thank all the contributors/users for their work on this release!

---

This discussion was created from the release Serverless Data Lake Framework 2.0.0.

[matth-imrsnt]

Struggling to get this working with Github as opposed to CodeCommit since we can no longer create new repos