feat(snowflake): add transient tmp relation type for incremental models

[b-per]

Summary

• Adds tmp_relation_type: transient config option for Snowflake incremental models. Unlike session-scoped temporary tables, transient tables persist in the catalog, making them visible to Snowflake's native lineage tracking, while avoiding the 7-day fail-safe storage costs of permanent tables.
• Adds snowflake__resolve_incremental_tmp_relation dispatch macro, allowing users to override the schema/database of the tmp relation — useful for avoiding name collisions when concurrent runs share the same target schema.
• Allows transient alongside table for delete+insert and microbatch strategies; both are stable across multi-statement executions.

Note: Iceberg (catalog-linked database) incremental models continue to use a permanent table for the tmp relation — CLD schemas only support Iceberg tables, so neither temporary nor transient tables can be used there.

Closes #1893

Usage

-- Enable lineage tracking via transient tmp table
{{ config(
    materialized='incremental',
    unique_key='id',
    tmp_relation_type='transient',
) }}

-- Redirect tmp relation to a scratch schema to avoid collisions
{% macro snowflake__resolve_incremental_tmp_relation(tmp_relation) %}
  {{ return(tmp_relation.incorporate(schema='scratch')) }}
{% endmacro %}

Test plan

• Existing unit tests pass (hatch run unit-tests)
• New functional tests in test_incremental_transient.py pass against a real Snowflake connection (verifies transient DDL is emitted and row counts are correct)

[Copilot]

Pull request overview

This PR extends the Snowflake incremental materialization to support a new tmp_relation_type: transient option for staging relations, aiming to enable Snowflake lineage visibility while avoiding permanent-table fail-safe storage costs, and to improve behavior for Iceberg (catalog-linked) incremental models.

Changes:

• Add a snowflake__create_table_transient_sql macro to create transient CTAS tables for incremental staging.
• Update the incremental materialization to (a) select transient tmp relations for catalog-linked databases and (b) allow transient for delete+insert and microbatch strategies.
• Introduce snowflake__resolve_incremental_tmp_relation to allow overriding where the incremental tmp relation is created, plus new functional coverage and a changelog entry.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.

File	Description
dbt-snowflake/tests/functional/adapter/incremental/test_incremental_transient.py	Adds functional tests covering tmp_relation_type='transient' for default and delete+insert incremental strategies.
dbt-snowflake/src/dbt/include/snowflake/macros/relations/table/create.sql	Adds a Snowflake macro to create transient tables via CTAS for incremental tmp relations.
dbt-snowflake/src/dbt/include/snowflake/macros/materializations/incremental.sql	Updates tmp-relation type selection/creation to support transient tmp relations, uses transient for Iceberg, and adds a tmp-relation resolution hook.
dbt-snowflake/.changes/unreleased/Features-20260424-transient-tmp-relation.yaml	Adds a changelog entry describing the new transient tmp relation support and Iceberg behavior change.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Comments suppressed due to low confidence (1)

dbt-snowflake/src/dbt/include/snowflake/macros/materializations/incremental.sql:186

• The tmp-relation creation path no longer treats catalog-linked (Iceberg/CLD) models specially. For CLD, tmp_relation_type resolves to table, which will hit this create_table_as(True, ...) branch and emit CREATE OR REPLACE TEMPORARY TABLE ... via snowflake__create_table_as, but Iceberg/CLD explicitly does not support temporary tables. Reintroduce an is_catalog_linked_db branch here (as previously) to create a non-temporary staging relation for CLD (or otherwise ensure the DDL is not temporary).

    {% else %}
        {%- call statement('create_tmp_relation', language=language) -%}
          {{ create_table_as(True, tmp_relation, compiled_code, language) }}
        {%- endcall -%}

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

dbt-snowflake/src/dbt/include/snowflake/macros/materializations/incremental.sql:186

• For catalog-linked database (Iceberg/CLD) incremental models, tmp_relation_type is forced to table, which means this branch will call create_table_as(True, ...) and attempt CREATE OR REPLACE TEMPORARY TABLE. Temporary tables are not supported for Iceberg/CLD (and the comment above says CLD should use a permanent table), so this will break CLD incremental runs. Consider restoring a CLD-specific branch here (or basing the decision on whether tmp_relation is catalog-linked after resolve_incremental_tmp_relation) to create a non-temporary table for CLD tmp relations.

    {% else %}
        {%- call statement('create_tmp_relation', language=language) -%}
          {{ create_table_as(True, tmp_relation, compiled_code, language) }}
        {%- endcall -%}

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

dbt-snowflake/src/dbt/include/snowflake/macros/materializations/incremental.sql:186

• The tmp-relation creation logic no longer special-cases catalog-linked databases (Iceberg/CLD). When is_catalog_linked_db is true, tmp_relation_type is forced to table, so this branch calls create_table_as(True, ...) which emits CREATE OR REPLACE TEMPORARY TABLE (via snowflake__create_table_temporary_sql). CLD schemas do not support temporary relations, so incremental runs against Iceberg tables will fail. Reintroduce the CLD branch here (create the tmp relation as a non-temporary table/Iceberg table) before checking tmp_relation_type for view/transient.

    {% else %}
        {%- call statement('create_tmp_relation', language=language) -%}
          {{ create_table_as(True, tmp_relation, compiled_code, language) }}
        {%- endcall -%}

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[b-per]

CI failure analysis

The integration test suite shows 1 failed, 400 passed — our new tests both passed:

• TestIncrementalTransientTmpRelation::test_incremental_transient ✅
• TestIncrementalTransientTmpRelationDeleteInsert::test_incremental_transient_delete_insert_runs ✅

The sole failure is TestSnowflakeIcebergRestCatalogIntegration::test_basic_iceberg_rest_catalog_integration.

After further investigation, this failure was caused by our PR: we accidentally removed the is_catalog_linked_db branch from the tmp relation creation section, causing CLD incremental models to use create_table_as(True, ...) (temporary table) instead of create_table_as(False, ...) (non-temporary Iceberg-compatible table). This broke the second run_dbt(["run"]) call (the incremental run) with SQL Compilation Error: This operation is not supported in a catalog-linked database.

The fix has been pushed in commit 569e000, which restores the original CLD creation path.

[b-per]

I am creating a draft doc PR for it.

Is anyone working on porting it to Fusion as well?