Skip to content

[Bug] Buggy enable_truthy_nulls_equals_macro behaviour with BigQuery incremental models #1902

Description

@sriramr98

Is this a new bug?

  • I believe this is a new bug
  • I have searched the existing issues, and I could not find an existing issue for this bug

Which packages are affected?

  • dbt-adapters
  • dbt-tests-adapter
  • dbt-athena
  • dbt-athena-community
  • dbt-bigquery
  • dbt-postgres
  • dbt-redshift
  • dbt-snowflake
  • dbt-spark

Current Behavior

When enable_truthy_nulls_equals_macro is enabled (which becomes the default in 1.12 via #1877), bigquery__equals emits IS NOT DISTINCT FROM. This clause is reused inside the MERGE ON predicate of the default get_merge_sql macro whenever unique_key is a string.

For an incremental model that has partition_by set together with require_partition_filter=True, BigQuery's partition-pruning analyzer no longer recognizes the auxiliary (<partition_field> is null or <partition_field> is not null) predicate (added by predicate_for_avoid_require_partition_filter) as a valid partition filter once the unique-key match uses IS NOT DISTINCT FROM. The MERGE then fails at runtime.

Expected Behavior

Incremental models that use merge strategy with a string unique_key, partition_by, and require_partition_filter=True should run successfully when enable_truthy_nulls_equals_macro is enabled, just as they do when the flag is disabled.

Steps To Reproduce

  1. Configure a BigQuery target.

  2. Enable the behavior flag in dbt_project.yml:

    flags:
      enable_truthy_nulls_equals_macro: true
  3. Define an incremental model with a string unique_key, partition_by, and require_partition_filter=True. Example (models/incremental_merge_time_with_require_partition.sql):

    {{
      config(
        materialized="incremental",
        unique_key="id",
        cluster_by="id",
        require_partition_filter=true,
        partition_by={
          "field": "date_time",
          "data_type": "dateTime"
        }
      )
    }}
    
    with data as (
      {% if not is_incremental() %}
        select 1 as id, cast('2020-01-01' as datetime) as date_time union all
        select 2 as id, cast('2020-01-01' as datetime) as date_time
      {% else %}
        select 1 as id, cast('2020-01-02' as datetime) as date_time union all
        select 3 as id, cast('2020-01-02' as datetime) as date_time
      {% endif %}
    )
    select * from data
  4. Run dbt run once to materialize the model (full refresh).

  5. Run dbt run again to trigger the incremental MERGE.

The second run fails.

Relevant log output

Database Error in model incremental_merge_time_with_require_partition (models/incremental_merge_time_with_require_partition.sql)
  Cannot query over table 'dbt-test-env.<schema>.incremental_merge_time_with_require_partition' without a filter over column(s) 'date_time' that can be used for partition elimination
  compiled code at target/run/test/models/incremental_merge_time_with_require_partition.sql


Generated MERGE ON clause (problematic):


merge into `<target>` as DBT_INTERNAL_DEST
    using (...) as DBT_INTERNAL_SOURCE
    on (DBT_INTERNAL_SOURCE.id IS NOT DISTINCT FROM DBT_INTERNAL_DEST.id)
       and (`DBT_INTERNAL_DEST`.`date_time` is null
            or `DBT_INTERNAL_DEST`.`date_time` is not null)


CI run that surfaced this: https://github.com/dbt-labs/dbt-adapters/actions/runs/24562327293/job/72021560451 (PR #1877, which flips the flag default to `true` for 1.12).

Environment

- OS: macOS 26.2 (also reproduces on Ubuntu 22.04 in CI)
- Python: 3.12.4 (CI uses 3.10)
- dbt-adapters: 1.22.10
- dbt-bigquery: 1.11.0rc3

Additional Context

The other adapters that emit IS NOT DISTINCT FROM from their equals macro (postgres, snowflake, athena) are unaffected — only BigQuery's MERGE ON / partition-pruning interaction breaks.

Metadata

Metadata

Assignees

Labels

pkg:dbt-bigqueryIssue affects dbt-bigquerytriage:productIn Product's queuetype:bugSomething isn't working as documented

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions