Skip to content

fix(bigquery): qualify column refs in test_unique / test_not_null to avoid row-struct resolution - #1898

Open
1fanwang wants to merge 2 commits into
dbt-labs:mainfrom
1fanwang:fix/bigquery-test-unique-not-null-table-column-collision
Open

fix(bigquery): qualify column refs in test_unique / test_not_null to avoid row-struct resolution#1898
1fanwang wants to merge 2 commits into
dbt-labs:mainfrom
1fanwang:fix/bigquery-test-unique-not-null-table-column-collision

Conversation

@1fanwang

@1fanwang 1fanwang commented Apr 27, 2026

Copy link
Copy Markdown

resolves dbt-labs/dbt-core#11067
docs N/A

Problem

When a column shares its name with the model, BigQuery's unique and not_null generic tests can incorrectly pass even with duplicate or null column values. Reproducer per dbt-core#11067:

# models/orders.sql
select 1 as orders union all
select 1 as orders union all
select cast(null as int64) as orders

# models/_models.yml
models:
  - name: orders
    columns:
      - name: orders
        data_tests:
          - unique
          - not_null

Root cause: BigQuery's identifier-resolution rule. When a column's name matches its containing relation's name and the relation is referenced without an alias, an unqualified reference like orders resolves to the row STRUCT containing all of the row's columns, not to the column value itself.

The current bigquery__test_unique and the inherited default__test_not_null emit unqualified column references:

{# bigquery__test_unique today #}
with dbt_test__target as (
  select {{ column_name }} as unique_field
  from {{ model }}
  where {{ column_name }} is not null
)
...

so the WHERE / GROUP BY apply against the row struct rather than the column, masking the violations the tests are meant to catch.

Solution

Alias the source relation as dbt_test__source and qualify column references in bigquery__test_unique. Add a bigquery__test_not_null override that does the same.

{% macro bigquery__test_unique(model, column_name) %}
with dbt_test__target as (
  select dbt_test__source.{{ column_name }} as unique_field
  from {{ model }} dbt_test__source
  where dbt_test__source.{{ column_name }} is not null
)
...
{% endmacro %}

{% macro bigquery__test_not_null(model, column_name) %}
{% set column_list = '*' if should_store_failures() else 'dbt_test__source.' ~ column_name %}
select {{ column_list }}
from {{ model }} dbt_test__source
where dbt_test__source.{{ column_name }} is null
{% endmacro %}

Same shape suggested in the review of the original stale attempt at dbt-core#2061 / PR dbt-core#2075. Limited the change to BigQuery because Postgres / Snowflake / Redshift / Spark resolve unqualified column references column-first, so the bug does not manifest there and changing the default macro would have a wider blast radius.

Tradeoffs considered:

  • Updating default__test_unique / default__test_not_null instead. Cleaner long-term, but riskier for adapters that override those defaults or user macros that shim them. BigQuery-only is safer; the default macros can still be changed if reviewers want the broader fix.
  • Different alias name. dbt_test__source mirrors the existing dbt_test__target CTE. A column literally named dbt_test__source would still resolve correctly because dbt_test__source.{{ column_name }} is alias-dot-column syntax. This can be renamed if reviewers prefer something less collision-prone.

Tests

Added dbt-bigquery/tests/functional/adapter/generic_tests/test_column_table_name_collision.py. It builds the reproducer model from the issue, runs dbt test, and asserts that both the unique and not_null tests fail with at least one failure each.

I have not been able to run the BigQuery integration tests locally because this environment has no project credentials. CI will need to validate the fixture shape.

Checklist

  • I have read the contributing guide and understand what's expected of me
  • I have run this code in development and it appears to resolve the stated issue (BQ integration tests not run locally; relying on CI)
  • This PR includes tests, or tests are not required/relevant for this PR
  • This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX

Copilot AI review requested due to automatic review settings April 27, 2026 08:42
@1fanwang
1fanwang requested a review from a team as a code owner April 27, 2026 08:42
@cla-bot

cla-bot Bot commented Apr 27, 2026

Copy link
Copy Markdown

Thanks for your pull request, and welcome to our community! We require contributors to sign our Contributor License Agreement and we don't seem to have your signature on file. Check out this article for more information on why we have a CLA.

In order for us to review and merge your code, please submit the Individual Contributor License Agreement form attached above above. If you have questions about the CLA, or if you believe you've received this message in error, please reach out through a comment on this PR.

CLA has not been signed by users: @1fanwang

@github-actions github-actions Bot added the community A PR, or an issue with a PR, from a community member label Apr 27, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes BigQuery generic unique/not_null tests when a column name collides with its model name by qualifying column references against an explicit source alias.

Changes:

  • Update bigquery__test_unique to alias the source relation and qualify column_name references.
  • Add a BigQuery-specific bigquery__test_not_null override with the same qualification approach.
  • Add a functional regression test covering the column/table-name collision scenario and a changelog entry.

Reviewed changes

Copilot reviewed 3 out of 4 changed files in this pull request and generated 4 comments.

File Description
dbt-bigquery/src/dbt/include/bigquery/macros/adapters.sql Qualifies column references in BigQuery unique and adds a BigQuery override for not_null.
dbt-bigquery/tests/functional/adapter/generic_tests/test_column_table_name_collision.py New regression test ensuring both unique and not_null fail correctly for the collision case.
dbt-bigquery/.changes/unreleased/Fixes-20260427-083810.yaml Adds a Fixes changelog entry describing the behavior change.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread dbt-bigquery/src/dbt/include/bigquery/macros/adapters.sql
Comment thread dbt-bigquery/src/dbt/include/bigquery/macros/adapters.sql Outdated
Comment thread dbt-bigquery/src/dbt/include/bigquery/macros/adapters.sql
@1fanwang
1fanwang force-pushed the fix/bigquery-test-unique-not-null-table-column-collision branch from 3cc712e to 38f84fb Compare April 27, 2026 09:01
@cla-bot

cla-bot Bot commented Apr 27, 2026

Copy link
Copy Markdown

Thanks for your pull request, and welcome to our community! We require contributors to sign our Contributor License Agreement and we don't seem to have your signature on file. Check out this article for more information on why we have a CLA.

In order for us to review and merge your code, please submit the Individual Contributor License Agreement form attached above above. If you have questions about the CLA, or if you believe you've received this message in error, please reach out through a comment on this PR.

CLA has not been signed by users: @1fanwang

@cla-bot cla-bot Bot added the cla:yes The PR author has signed the CLA label Apr 27, 2026
@1fanwang

Copy link
Copy Markdown
Author

Pushed 92a39c5 — wraps model in (select * from {{ model }}) before the dbt_test__source alias in both macros so where:-filtered tests don't end up double-aliased, switched the result lookup to prefix matching, and added a where-filtered regression case. Updated the changelog accordingly.

@1fanwang
1fanwang force-pushed the fix/bigquery-test-unique-not-null-table-column-collision branch from a92096a to 8180fb9 Compare May 7, 2026 01:59
@1fanwang
1fanwang force-pushed the fix/bigquery-test-unique-not-null-table-column-collision branch from 8180fb9 to 60d5aa1 Compare May 12, 2026 06:49
1fanwang added 2 commits May 12, 2026 09:13
…oid row-struct resolution

When a column shares its name with the model, BigQuery's identifier
resolution returns the row STRUCT rather than the column value for
unqualified references. The default `bigquery__test_unique` and the
inherited `default__test_not_null` therefore evaluated WHERE/GROUP BY
against the row struct, masking duplicates and nulls.

Alias the source relation as `dbt_test__source` and qualify the column
references in `bigquery__test_unique`, and add a `bigquery__test_not_null`
override that does the same. Adds a regression test that creates a
single-column model whose column name matches the model name and
asserts both generic tests detect failures.

Resolves dbt-core#11067.
…not_null

`get_where_subquery` renders `model` as `(select * from <rel> where ...)
dbt_subquery` for `where:`-filtered generic tests. The previous form
`from {{ model }} dbt_test__source` produced an invalid double alias
`(...) dbt_subquery dbt_test__source` in that case. Wrap `model` in
`(select * from {{ model }})` before applying the `dbt_test__source`
alias so the macros work whether `model` is a bare relation or an
already-aliased subquery.

Add a regression test that exercises `unique` and `not_null` with the
column/table name collision under a `where:` config, and switch the
existing test's result lookup to prefix matching so it surfaces a clear
error if generic-test node naming changes.
@1fanwang
1fanwang force-pushed the fix/bigquery-test-unique-not-null-table-column-collision branch from 60d5aa1 to aac2f2f Compare May 12, 2026 16:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla:yes The PR author has signed the CLA community A PR, or an issue with a PR, from a community member

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Test for uniqueness passes when duplicates are present in BigQuery column

3 participants