Skip to content

Update for nested fields#98

Merged
evie-autotrader merged 63 commits into
mainfrom
update-for-nested-fields
May 12, 2026
Merged

Update for nested fields#98
evie-autotrader merged 63 commits into
mainfrom
update-for-nested-fields

Conversation

@shugewang

@shugewang shugewang commented Apr 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Description

Since dbt-bigquery 1.11 you can now add struct fields to an incremental when on_schema_change is set to append_new_columns or sync_new_columns

This PR updates the schema change handlers to support struct schema changes

Fixes #96

Checklist:

  • I have run make verify and fixed any linting or test errors
  • I have added appropriate unit tests or if applicable an integration test
  • OPTIONAL: I have run make integration against a Big Query instance

cat-wates and others added 20 commits April 21, 2026 09:14
@cat-wates
Add test for append handler
d6a8eb0
@shugewang
Add utils for finding new nested fields
def1403
@shugewang
Add to dos
3e6331e
@cat-wates
Append missing fields to target table fields
a9ac57a
@cat-wates
[WIP] reconstruction logic
a1d8a83
@shugewang
Refactor utils to use FieldLineage class, change append_new_fields to…
be542cc 
… find_missing_fields
@shugewang
WIP - add failing test for build_predicted_table() and minor refactoring
0e1bfd8
@shugewang
WIP - add missing fields and build predicted table
89a489e
@shugewang
Add unit tests for add_missing_fields
58c9c24
@cat-wates
Delete sample test
1dd4875
@cat-wates
Change append_new_columns_handler to use util functions
e4b36f8
@cat-wates
Add test for preserving nest column order
7c3c1ae
@shugewang
Rename
169b9bb
@shugewang
build_predicted_table -> build_predicted_fields
6175011
@shugewang
Refactor build_predicted_fields to work for sync_all_columns handler
19d289f
@shugewang
Add more tests for schema change handlers
1f78320
@shugewang
Formating
5a03b09
@shugewang
Limit number of nested record types to 15 to match BQ limit
5380476
@shugewang
Set max level of recursion for collect_field_lineages
debc9db
@shugewang
Test repeated structs (arrays) are supported
0a41cb7
@shugewang shugewang marked this pull request as ready for review April 27, 2026 13:38
ccharlesgb
ccharlesgb reviewed Apr 28, 2026
View reviewed changes
Comment thread dbt_dry_run/models/table.py Outdated
ccharlesgb
ccharlesgb reviewed Apr 28, 2026
View reviewed changes
Comment thread dbt_dry_run/utils.py Outdated
@ccharlesgb

ccharlesgb commented Apr 28, 2026

Copy link
Copy Markdown
Collaborator

We should probably have at least one integration test. You can use this one as a reference

You will need to create an incremental in projects/test_incremental. The with compiled_project.create_state(manifest_node, columns): will set the initial table schema up before running the integration test so you could create a model that has a struct with one field. Then the model adds another field and the predicted schema has both fields.

evie-autotrader and others added 19 commits May 6, 2026 14:40
@evie-autotrader
add is_top_level property to table FieldPath
faee492
@evie-autotrader
make references to missing fields more specific
89cf194
@evie-autotrader
rename collect field paths method to add context
cdbe714
@evie-autotrader
refactor to add context around missing fields and make error messages…
ec4c817 
… less BQ specific
@shugewang
Refactor sync_all_columns_handler logic for removing top level fields
86037eb
@evie-autotrader
partial refactor for abstraction
8305489 
# Conflicts:
#	dbt_dry_run/nested_schema_change.py
#	dbt_dry_run/schema_change_handlers.py
#	dbt_dry_run/test/test_nested_schema_change.py
@shugewang
New test, linting
b080919
@evie-autotrader
rename nested_schema_change to remove unneeded qualifier
d086a0d
@shugewang
Fix imports
3333368
@evie-autotrader
refactor - make nested field removal check more abstract
247c35c
@evie-autotrader
refactored to clarify responsibilities and privacy of schema_change m…
da5987c 
…ethods and clarify test cases
@shugewang
Rename functions
e7b0b3b
@shugewang
WIP - renaming in merge_table_fields
bb90992
@evie-autotrader
test cases for merge_table_fields
118aa5a
@evie-autotrader
schema change handlers raise error when nested field removed
b4a15a4
@shugewang
Make methods in schema_manipulation role-agnostic
23a7015
@shugewang
WIP - add fail handler integration tests (nested failing)
25728e3
@shugewang
Update fail_handler to detect nested changes
3acc558
@shugewang
Refactor assert_no_nested_fields_removed to use is_top_level property
0d47ad5
@shugewang shugewang marked this pull request as draft May 11, 2026 08:33
shugewang and others added 2 commits May 11, 2026 10:30
@shugewang
Add path to TableField, removing the need for FieldPath. Move _collec…
23c1bc0 
…t_flattened_field_paths to Table class
@evie-autotrader
Move assert to schema_change_handlers.py
40c50bf 
Remove references to new and existing tables
Rename _get_new_fields
Move MAX_SUPPORTED_NESTED_FIELD_DEPTH to Table
@shugewang shugewang marked this pull request as ready for review May 11, 2026 10:46
ccharlesgb
ccharlesgb reviewed May 11, 2026
View reviewed changes

@ccharlesgb ccharlesgb left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Almost there. Just one question on the classmethod and a bit of confusion about the path attribute

Comment thread dbt_dry_run/models/table.py Outdated
Comment thread dbt_dry_run/models/table.py Outdated
@evie-autotrader evie-autotrader merged commit 1637382 into main May 12, 2026
4 checks passed
@evie-autotrader evie-autotrader deleted the update-for-nested-fields branch May 12, 2026 12:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ccharlesgb ccharlesgb ccharlesgb left review comments

@evie-autotrader evie-autotrader evie-autotrader left review comments

@stevenjwoods stevenjwoods stevenjwoods approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Incremental models now support struct schema changes

5 participants

@shugewang @ccharlesgb @stevenjwoods @evie-autotrader @cat-wates