fix(bigquery): qualify column refs in test_unique / test_not_null to avoid row-struct resolution - #1898
Conversation
|
Thanks for your pull request, and welcome to our community! We require contributors to sign our Contributor License Agreement and we don't seem to have your signature on file. Check out this article for more information on why we have a CLA. In order for us to review and merge your code, please submit the Individual Contributor License Agreement form attached above above. If you have questions about the CLA, or if you believe you've received this message in error, please reach out through a comment on this PR. CLA has not been signed by users: @1fanwang |
There was a problem hiding this comment.
Pull request overview
Fixes BigQuery generic unique/not_null tests when a column name collides with its model name by qualifying column references against an explicit source alias.
Changes:
- Update
bigquery__test_uniqueto alias the source relation and qualifycolumn_namereferences. - Add a BigQuery-specific
bigquery__test_not_nulloverride with the same qualification approach. - Add a functional regression test covering the column/table-name collision scenario and a changelog entry.
Reviewed changes
Copilot reviewed 3 out of 4 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| dbt-bigquery/src/dbt/include/bigquery/macros/adapters.sql | Qualifies column references in BigQuery unique and adds a BigQuery override for not_null. |
| dbt-bigquery/tests/functional/adapter/generic_tests/test_column_table_name_collision.py | New regression test ensuring both unique and not_null fail correctly for the collision case. |
| dbt-bigquery/.changes/unreleased/Fixes-20260427-083810.yaml | Adds a Fixes changelog entry describing the behavior change. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
3cc712e to
38f84fb
Compare
|
Thanks for your pull request, and welcome to our community! We require contributors to sign our Contributor License Agreement and we don't seem to have your signature on file. Check out this article for more information on why we have a CLA. In order for us to review and merge your code, please submit the Individual Contributor License Agreement form attached above above. If you have questions about the CLA, or if you believe you've received this message in error, please reach out through a comment on this PR. CLA has not been signed by users: @1fanwang |
|
Pushed 92a39c5 — wraps |
a92096a to
8180fb9
Compare
8180fb9 to
60d5aa1
Compare
…oid row-struct resolution When a column shares its name with the model, BigQuery's identifier resolution returns the row STRUCT rather than the column value for unqualified references. The default `bigquery__test_unique` and the inherited `default__test_not_null` therefore evaluated WHERE/GROUP BY against the row struct, masking duplicates and nulls. Alias the source relation as `dbt_test__source` and qualify the column references in `bigquery__test_unique`, and add a `bigquery__test_not_null` override that does the same. Adds a regression test that creates a single-column model whose column name matches the model name and asserts both generic tests detect failures. Resolves dbt-core#11067.
…not_null
`get_where_subquery` renders `model` as `(select * from <rel> where ...)
dbt_subquery` for `where:`-filtered generic tests. The previous form
`from {{ model }} dbt_test__source` produced an invalid double alias
`(...) dbt_subquery dbt_test__source` in that case. Wrap `model` in
`(select * from {{ model }})` before applying the `dbt_test__source`
alias so the macros work whether `model` is a bare relation or an
already-aliased subquery.
Add a regression test that exercises `unique` and `not_null` with the
column/table name collision under a `where:` config, and switch the
existing test's result lookup to prefix matching so it surfaces a clear
error if generic-test node naming changes.
60d5aa1 to
aac2f2f
Compare
resolves dbt-labs/dbt-core#11067
docs N/A
Problem
When a column shares its name with the model, BigQuery's
uniqueandnot_nullgeneric tests can incorrectly pass even with duplicate or null column values. Reproducer per dbt-core#11067:Root cause: BigQuery's identifier-resolution rule. When a column's name matches its containing relation's name and the relation is referenced without an alias, an unqualified reference like
ordersresolves to the row STRUCT containing all of the row's columns, not to the column value itself.The current
bigquery__test_uniqueand the inheriteddefault__test_not_nullemit unqualified column references:{# bigquery__test_unique today #} with dbt_test__target as ( select {{ column_name }} as unique_field from {{ model }} where {{ column_name }} is not null ) ...so the WHERE / GROUP BY apply against the row struct rather than the column, masking the violations the tests are meant to catch.
Solution
Alias the source relation as
dbt_test__sourceand qualify column references inbigquery__test_unique. Add abigquery__test_not_nulloverride that does the same.Same shape suggested in the review of the original stale attempt at dbt-core#2061 / PR dbt-core#2075. Limited the change to BigQuery because Postgres / Snowflake / Redshift / Spark resolve unqualified column references column-first, so the bug does not manifest there and changing the default macro would have a wider blast radius.
Tradeoffs considered:
default__test_unique/default__test_not_nullinstead. Cleaner long-term, but riskier for adapters that override those defaults or user macros that shim them. BigQuery-only is safer; the default macros can still be changed if reviewers want the broader fix.dbt_test__sourcemirrors the existingdbt_test__targetCTE. A column literally nameddbt_test__sourcewould still resolve correctly becausedbt_test__source.{{ column_name }}is alias-dot-column syntax. This can be renamed if reviewers prefer something less collision-prone.Tests
Added
dbt-bigquery/tests/functional/adapter/generic_tests/test_column_table_name_collision.py. It builds the reproducer model from the issue, runsdbt test, and asserts that both theuniqueandnot_nulltests fail with at least one failure each.I have not been able to run the BigQuery integration tests locally because this environment has no project credentials. CI will need to validate the fixture shape.
Checklist