add delta lake to iceberg blueprint and IT by derrickaw · Pull Request #4103 · GoogleCloudPlatform/DataflowTemplates

derrickaw · 2026-08-03T21:06:20Z

No description provided.

codecov · 2026-08-03T21:10:49Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 38.01%. Comparing base (3515682) to head (1b084e9).
⚠️ Report is 7 commits behind head on main.

❗ There is a different number of reports uploaded between BASE (3515682) and HEAD (1b084e9). Click for more details.

HEAD has 30 uploads less than BASE

Flag BASE (3515682) HEAD (1b084e9)

31 1

Additional details and impacted files

@@              Coverage Diff              @@
##               main    #4103       +/-   ##
=============================================
- Coverage     55.94%   38.01%   -17.94%     
+ Complexity     7485      511     -6974     
=============================================
  Files          1134      195      -939     
  Lines         70175    11761    -58414     
  Branches       8023     1181     -6842     
=============================================
- Hits          39261     4471    -34790     
+ Misses        28367     6924    -21443     
+ Partials       2547      366     -2181

Components	Coverage Δ
spanner-templates	`∅ <ø> (∅)`
spanner-import-export	`∅ <ø> (∅)`
spanner-live-forward-migration	`∅ <ø> (∅)`
spanner-live-reverse-replication	`∅ <ø> (∅)`
spanner-bulk-migration	`∅ <ø> (∅)`
gcs-spanner-dv	`∅ <ø> (∅)`
see 964 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

gemini-code-assist · 2026-08-18T17:48:00Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request adds a new Dataflow template that enables data migration from Delta Lake tables to Apache Iceberg tables. It includes the necessary YAML definitions, Java interface files, and updates to the template generation tooling to support this new pipeline. Additionally, an integration test has been included to verify the pipeline's correctness.

Highlights

New Dataflow Template: Added a new Delta Lake to Iceberg batch pipeline template to the repository.
Template Configuration: Introduced new YAML configuration files and a Java interface for the Delta Lake to Iceberg template.
Build Tooling Updates: Updated the Python generation script to dynamically handle imports and default parameters for Java templates.
Integration Testing: Added a comprehensive integration test for the new template to ensure end-to-end functionality.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces the Delta Lake to Iceberg batch pipeline template, including its YAML definition, Java metadata interface, documentation, and integration tests. It also updates the Python template generator to conditionally import the Beam SDK Default options. The feedback suggests improving the YAML syntax for requirements to use standard list notation and dynamically calculating the parquet file size in the integration test to avoid hardcoding.

chamikaramj

Thanks!

…ke2LHIT

derrickaw · 2026-09-02T17:35:24Z

Hi @chamikaramj, friendly ping. Thanks

chamikaramj

Thanks.

* add delta lake to iceberg IT * fix spotless and rerun auto-generation * switch to interrupt call * change to a dynamic import to fix error * add forgotten yaml definition file * add generated README for DeltaLakeToIceberg template * switch back to stop method for direct runner * address gemini comment * spotless * initial comments fixes * change iceberg to lakehouse * change to a wrapped writetoiceberg via writetolakehouse * update yaml file with wrapper transform * remove old writetolakehouse transform idea * spotless * change to renamed transform * address Delta client comment * spotless * fix generated file * create write to lakehouse * change package * oops - used wrong assumed date * fix uri catalog version * update jar to to be the same * change to managed and mimic deltalake * update one more version jar * add manifest resource transformer * add managed and iceberg * change pattern * force to schemaAwareExternalTransform * another try * add transformer * update per Cham's comments * fix formatting

* yml file * yml file * changes * deps: update Python dependencies (#4194) Co-authored-by: liferoad <7833268+liferoad@users.noreply.github.com> * [SpannerToSourceDb] Reverse replication Integration test with SSL based connection (#4192) * [SpannerToSourceDb] Reverse replication Integration test with SSL based connection * Moving process creation code into a method * Addressing comments * add auto-close dependabot step for requirements*.txt type files (#4220) * add auto-close dependabot step * add ignore comment * add delta lake to iceberg blueprint and IT (#4103) * add delta lake to iceberg IT * fix spotless and rerun auto-generation * switch to interrupt call * change to a dynamic import to fix error * add forgotten yaml definition file * add generated README for DeltaLakeToIceberg template * switch back to stop method for direct runner * address gemini comment * spotless * initial comments fixes * change iceberg to lakehouse * change to a wrapped writetoiceberg via writetolakehouse * update yaml file with wrapper transform * remove old writetolakehouse transform idea * spotless * change to renamed transform * address Delta client comment * spotless * fix generated file * create write to lakehouse * change package * oops - used wrong assumed date * fix uri catalog version * update jar to to be the same * change to managed and mimic deltalake * update one more version jar * add manifest resource transformer * add managed and iceberg * change pattern * force to schemaAwareExternalTransform * another try * add transformer * update per Cham's comments * fix formatting * fix naming determination (#4210) * fix naming * fix default imports issue * rerun some checks * clean up old named files * change pubsubtoiceberg yaml name * update readme with blueprint naming conventions * Refactor pipeline parameter options to seperate Options.java file (#4209) * yml file * yml file * changes * Refactor DataStreamToSpannerOptions to separate file * Delete .github/workflows/codelab-watchdog.yml * Refactor SpannerToSourceDbOptions to separate file * mistakes * Refactor GCSSpannerDVOptions to a separate file * spotless * Fix checkstyle formatting and missing package-info * year * year * fix tests * fix tests * testing * spotless * Fix test compilation failures in SpannerToSourceDbTest * spotless * change+test * revert test changes --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: liferoad <7833268+liferoad@users.noreply.github.com> Co-authored-by: darshan-sj <darshan-sj@users.noreply.github.com> Co-authored-by: Derrick Williams <derrickaw@google.com>

* add delta lake to iceberg IT * fix spotless and rerun auto-generation * switch to interrupt call * change to a dynamic import to fix error * add forgotten yaml definition file * add generated README for DeltaLakeToIceberg template * switch back to stop method for direct runner * address gemini comment * spotless * initial comments fixes * change iceberg to lakehouse * change to a wrapped writetoiceberg via writetolakehouse * update yaml file with wrapper transform * remove old writetolakehouse transform idea * spotless * change to renamed transform * address Delta client comment * spotless * fix generated file * create write to lakehouse * change package * oops - used wrong assumed date * fix uri catalog version * update jar to to be the same * change to managed and mimic deltalake * update one more version jar * add manifest resource transformer * add managed and iceberg * change pattern * force to schemaAwareExternalTransform * another try * add transformer * update per Cham's comments * fix formatting

…#4206) * yml file * yml file * changes * deps: update Python dependencies (GoogleCloudPlatform#4194) Co-authored-by: liferoad <7833268+liferoad@users.noreply.github.com> * [SpannerToSourceDb] Reverse replication Integration test with SSL based connection (GoogleCloudPlatform#4192) * [SpannerToSourceDb] Reverse replication Integration test with SSL based connection * Moving process creation code into a method * Addressing comments * add auto-close dependabot step for requirements*.txt type files (GoogleCloudPlatform#4220) * add auto-close dependabot step * add ignore comment * add delta lake to iceberg blueprint and IT (GoogleCloudPlatform#4103) * add delta lake to iceberg IT * fix spotless and rerun auto-generation * switch to interrupt call * change to a dynamic import to fix error * add forgotten yaml definition file * add generated README for DeltaLakeToIceberg template * switch back to stop method for direct runner * address gemini comment * spotless * initial comments fixes * change iceberg to lakehouse * change to a wrapped writetoiceberg via writetolakehouse * update yaml file with wrapper transform * remove old writetolakehouse transform idea * spotless * change to renamed transform * address Delta client comment * spotless * fix generated file * create write to lakehouse * change package * oops - used wrong assumed date * fix uri catalog version * update jar to to be the same * change to managed and mimic deltalake * update one more version jar * add manifest resource transformer * add managed and iceberg * change pattern * force to schemaAwareExternalTransform * another try * add transformer * update per Cham's comments * fix formatting * fix naming determination (GoogleCloudPlatform#4210) * fix naming * fix default imports issue * rerun some checks * clean up old named files * change pubsubtoiceberg yaml name * update readme with blueprint naming conventions * Refactor pipeline parameter options to seperate Options.java file (GoogleCloudPlatform#4209) * yml file * yml file * changes * Refactor DataStreamToSpannerOptions to separate file * Delete .github/workflows/codelab-watchdog.yml * Refactor SpannerToSourceDbOptions to separate file * mistakes * Refactor GCSSpannerDVOptions to a separate file * spotless * Fix checkstyle formatting and missing package-info * year * year * fix tests * fix tests * testing * spotless * Fix test compilation failures in SpannerToSourceDbTest * spotless * change+test * revert test changes --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: liferoad <7833268+liferoad@users.noreply.github.com> Co-authored-by: darshan-sj <darshan-sj@users.noreply.github.com> Co-authored-by: Derrick Williams <derrickaw@google.com>

pull-request-size Bot added the size/L label Aug 3, 2026

derrickaw changed the title ~~add delta lake to iceberg IT~~ add delta lake to iceberg blueprint and IT Aug 3, 2026

derrickaw added the addition New feature or request label Aug 3, 2026

derrickaw force-pushed the 20260803_addDeltaLake2LHIT branch from f0d7c7e to d642c4b Compare August 8, 2026 01:59

pull-request-size Bot added size/XL and removed size/L labels Aug 17, 2026

derrickaw added 7 commits August 18, 2026 16:37

add delta lake to iceberg IT

66fc208

fix spotless and rerun auto-generation

92aadfa

switch to interrupt call

9caddc2

change to a dynamic import to fix error

ece2e53

add forgotten yaml definition file

58cc7cc

add generated README for DeltaLakeToIceberg template

6f0eb7d

switch back to stop method for direct runner

353123c

derrickaw force-pushed the 20260803_addDeltaLake2LHIT branch from f6ff2b9 to 353123c Compare August 18, 2026 16:37

derrickaw marked this pull request as ready for review August 18, 2026 17:47

gemini-code-assist Bot reviewed Aug 18, 2026

View reviewed changes

Comment thread yaml/src/main/yaml/DeltaLakeToIceberg.yaml Outdated

Comment thread yaml/src/test/java/com/google/cloud/teleport/templates/yaml/DeltaLakeToLakehouseYamlIT.java Outdated

derrickaw added 2 commits August 18, 2026 18:20

address gemini comment

0b6fa4a

spotless

a9b0bcf

derrickaw force-pushed the 20260803_addDeltaLake2LHIT branch from e6ed587 to a9b0bcf Compare August 19, 2026 16:00

derrickaw requested a review from chamikaramj August 21, 2026 17:08

chamikaramj requested changes Aug 21, 2026

View reviewed changes

derrickaw added 7 commits August 25, 2026 14:25

initial comments fixes

9fdced5

change iceberg to lakehouse

ee83e6e

change to a wrapped writetoiceberg via writetolakehouse

99d9dd5

update yaml file with wrapper transform

0dc18f3

remove old writetolakehouse transform idea

e35c1d2

spotless

a0cf3b4

change to renamed transform

a323e94

derrickaw added 3 commits August 28, 2026 20:12

Merge remote-tracking branch 'upstream/main' into 20260803_addDeltaLa…

498d496

…ke2LHIT

fix generated file

3cfd16c

create write to lakehouse

12f799e

pull-request-size Bot added size/XXL and removed size/XL labels Aug 28, 2026

derrickaw added 12 commits August 29, 2026 01:55

change package

aa73a25

oops - used wrong assumed date

a6e7bad

fix uri catalog version

f0e93b3

update jar to to be the same

1300f82

change to managed and mimic deltalake

6333399

update one more version jar

ef2df8e

add manifest resource transformer

5dd1cc4

add managed and iceberg

86f5da5

change pattern

90c0458

force to schemaAwareExternalTransform

b6f16a0

another try

8973be4

add transformer

655f3a8

derrickaw requested a review from chamikaramj August 31, 2026 12:55

chamikaramj previously approved these changes Sep 5, 2026

View reviewed changes

Comment thread python/src/main/python/job-builder-util-transforms/write_to_lakehouse.py

Comment thread yaml/src/main/python/options/deltalake_options.yaml

Comment thread yaml/src/main/yaml/DeltaLakeToLakehouse.yaml

update per Cham's comments

9e2ba6a

derrickaw dismissed chamikaramj’s stale review via 9e2ba6a September 5, 2026 18:33

fix formatting

1b084e9

derrickaw merged commit 0c67c55 into GoogleCloudPlatform:main Sep 5, 2026
20 checks passed

derrickaw deleted the 20260803_addDeltaLake2LHIT branch September 5, 2026 20:13

Navigation Menu

Uh oh!

add delta lake to iceberg blueprint and IT - #4103

add delta lake to iceberg blueprint and IT#4103
derrickaw merged 35 commits into
GoogleCloudPlatform:mainfrom
derrickaw:20260803_addDeltaLake2LHIT

derrickaw commented Aug 3, 2026

Uh oh!

codecov Bot commented Aug 3, 2026 •

edited

Loading

Uh oh!

gemini-code-assist Bot commented Aug 18, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

Uh oh!

chamikaramj left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

derrickaw commented Sep 2, 2026

Uh oh!

chamikaramj left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

derrickaw commented Aug 3, 2026

Uh oh!

codecov Bot commented Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

gemini-code-assist Bot commented Aug 18, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

chamikaramj left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

derrickaw commented Sep 2, 2026

Uh oh!

chamikaramj left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov Bot commented Aug 3, 2026 •

edited

Loading