v0.0.10

What's Changed

Enhancements

• Added apply_changes_from_snapshot support in silver layer PR
• Added UI using databricks lakehouse app for onboarding/deploy commands PR
• Added support for non-Delta as sinks(delta, kafka) PR
• Added quarantine support in silver layer for data quality rules PR
• Added support for table comments, column comments, and cluster_by PR
• Added catalog support for sourceDetails and targetDetails PR
• Added DBDemos for dlt-meta PR
• Added YAML support for onboarding PR

Bug fixes

• Fixed issue cluster by not working with bronze append only table PR
• Fixed issue view name containing period when using DPM PR
• Fixed issue CLI onboarding overwrite option always set to True PR
• Fixed issue Silver DLT not creating based on passed database PR
• Fixed issue PyPI download stats display PR
• Fixed issue Silver Data Quality not working PR
• Fixed issue Removed DPM flag check inside dataflowpipeline PR
• Fixed issue Updated dlt-meta demos into Delta Live Tables Notebook github PR
• Fixed issue Adding multiple col support for auto_cdc api PR
• Fixed issue Added support for custom transformations for Kafka/Delta PR

New Contributors

• @JunHongPP made their first contribution in #162
• @junhonglau made their first contribution in #174
• @gchandra10 made their first contribution in #203

Full Changelog: v0.0.9...v0.0.10

v0.0.9

What's Changed

Enhancements

• Added apply_changes_from_snapshot api support in bronze layer: PR
• Added dlt append_flow api support for silver layer: PR
• Added support for file metadata columns for autoloader: PR
• Added support for Bring your own custom transformation: Issue
• Added support to Unify PyPI releases with GitHub OIDC: PR
• Added demo for append_flow and file_metadata options: PR
• Added Demo for silver fanout architecture: PR
• Added hugo-theme-relearn themee: PR
• Added unit tests to showcase silver layer fanout examples: PR
• Added liquid cluster support: PR

Bug fixes

• Fixed issue for No such file or directory: '/demo' :PR
• Fixed issue DLT-META CLI onboard command issue for Azure: databricks.sdk.errors.platform.ResourceAlreadyExists :PR
• Fixed issue Changed dbfs.create to mkdirs for CLI: PR
• Fixed issue DLT-META CLI should use pypi lib instead of whl : PR
• Fixed issue Onboarding with multiple partition columns errors out: PR

New Contributors

• @mithilmehta121287 made their first contribution in #89
• @dvanderwood made their first contribution in #101
• @sravanivadali made their first contribution in #116
• @harsha-pasala made their first contribution in #129
• @dattawalake made their first contribution in #127

Full Changelog: v0.0.8...v0.0.9

v0.0.8

Enhancements

• Added dlt append_flow api support: #58
• Added dlt append_flow api support for silver layer: #63
• Added support for file metadata columns for autoloader: #56
• Added support for Bring your own custom transformation: #68
• Added support to Unify PyPI releases with GitHub OIDC: #62
• Added demo for append_flow and file_metadata options: #74
• Added Demo for silver fanout architecture: #83
• Added documentation in docs site for new features: #64
• Added unit tests to showcase silver layer fanout examples: #67

Bug fixes

• Fixed issue for No such file or directory: '/demo' :#59
• Fixed issue DLT-META CLI onboard command issue for Azure: databricks.sdk.errors.platform.ResourceAlreadyExists :#51
• Fixed issue Changed dbfs.create to mkdirs for CLI: #53
• Fixed issue DLT-META CLI should use pypi lib instead of whl : #79

Full Changelog: v0.0.7...v0.0.8

v0.0.7

Enhancement

1. Mismatched Keys: Update read_dlt_delta() with key "source_database" instead of "database" #33

2. Create dlt-meta cli documentation #45

• Readme and docs to include above features

v0.0.6

Enhancement

1. Migrate to create_streaming_table api from create_streaming_live_table #37

Updates

• Readme and docs to include above features
• Added Data Quality support for silver layer
• Updated existing demos to incorporate unity catalog and integration test framework
• integration tests framework which can be used to launch demos

Unity Catalog and Databricks labs CLI support

New Features

1. Unity Catalog Support (#28)

2. Databricks Labs CLI Support (#28)

• Added two commands for DLT-META
• onboard: Captures all onboarding details from command line and launch onboarding job to your databricks workspace
• deploy: Captures all DLT pipeline details from command line and launch DLT pipeline to your databricks workspace

Updates

• Readme and docs to include above features
• Updated existing demos to incorporate unity catalog and integration test framework
• integration tests framework which can be used to launch demos

Bug Fixes

Bug Fixes

• Introduced new source detail option for eventhub: eventhub.accessKeySecretName

v0.0.3

Bug Fixes

• Infer datatypes from sequence_by to __START_AT, __END_AT for apply changes API

• Removed Git release tag from github actions

v0.0.2

v0.0.2

New Features

• Table properties support for bronze, quarantine and silver tables using create_streaming_live_table api call
• Support for track history column using apply_changes api
• Support for delta as source
• Validation for bronze/silver onboarding

Bug Fixes

• Input schema parsing issue in onboarding

Updates

• Readme and docs to include above features

Release v.0.0.1

v0.0.1

Removed pypi test from readme