feat(snowflake): add interactive table materialization support - #2042
Conversation
Add support for Snowflake Interactive Tables as a new materialization type (`materialized='interactive_table'`), addressing the GA of Snowflake's interactive analytics feature. Interactive tables are optimized for low-latency, high-concurrency queries when used with interactive warehouses. They support both static (one-time populate) and dynamic (auto-refresh via TARGET_LAG) variants. Key changes: - New `InteractiveTable` relation type and config dataclass - CREATE/REPLACE/DROP SQL macros for interactive table DDL - Materialization macro with create/replace/no-op routing - Detection via `is_interactive` column in SHOW OBJECTS/TABLES - Catalog integration for `dbt docs` - 20 unit tests for config parsing and changeset detection Since Snowflake has no ALTER INTERACTIVE TABLE, all config changes require CREATE OR REPLACE (full refresh). Refs: #1780 Made-with: Cursor
The is_dynamic check in _parse_list_relations_result only matched 'Y', while the adjacent is_interactive and is_iceberg checks accepted both 'Y' and 'YES'. This inconsistency could cause dynamic tables to be misidentified as regular tables if Snowflake returns 'YES'. Made-with: Cursor
The get_create_interactive_table_as_sql macro accepted sql as a parameter but ignored it, using the compiled_code global instead. This is fragile and inconsistent with the replace macro. Now passes the sql parameter through to the inner macro. Made-with: Cursor
1. describe_interactive_table: filter SHOW TABLES results to exact name match, since LIKE uses pattern matching and can return multiple rows. 2. target_lag removal detection: remove the `is not None` guard so that switching from dynamic to static (removing target_lag) is properly detected as a config change requiring CREATE OR REPLACE. 3. catalog.sql: remove is_interactive reference from information_schema query. Per Snowflake docs, is_interactive only exists in SHOW TABLES output (BCR-2165), not in information_schema.tables. The column reference would cause the catalog query to fail. 4. Remove unused Union import from interactive_table.py. 5. Always include 'text' in describe_interactive_table base_columns since parse_relation_results depends on it for the query field. 6. Add unit test for target_lag removal change detection. Made-with: Cursor
In _parse_list_relations_result, the interactive table and dynamic table type assignments used self.Relation.InteractiveTable/DynamicTable which are classproperties returning plain str values. This created a type inconsistency with the SnowflakeRelationType enum returned by get_relation_type on line 345. Now uses SnowflakeRelationType enum members directly, matching the Optional[SnowflakeRelationType] type annotation on the relation's type field. Made-with: Cursor
Made-with: Cursor
Made-with: Cursor
Refactor interactive flag detection to handle both `is_interactive` and `is_adaptive` column names from Snowflake metadata. Fix mypy union-attr error on nullable identifier by guarding with `or ""`. Add quoting-aware identifier matching in describe_interactive_table. Add unit tests for the new flag detection helpers. Made-with: Cursor
…ined error Annotate the relation_configs class variable with ClassVar[Dict[str, Type[SnowflakeRelationConfigBase]]] so mypy recognizes that dict values have from_relation_config(). Made-with: Cursor
… table indicator Simplify interactive table detection to only check the is_interactive column. Remove _INTERACTIVE_TABLE_COLUMN_NAMES, _TRUTHY_FLAG_VALUES, _is_interactive_flag(), _get_interactive_flag() helpers and their tests. Made-with: Cursor
…n_results Interactive tables always require cluster_by. If Snowflake metadata returns an empty or sentinel value, raise a CompilationError rather than silently setting cluster_by to None, which would violate the str type annotation on the dataclass field. Made-with: Cursor
…detection Use SHOW INTERACTIVE TABLES (returns target_lag, refresh_warehouse, schema/db/text), read refresh_warehouse column, and strip parens from cluster_by to avoid spurious CREATE OR REPLACE. Adds functional tests and changie entry. Co-authored-by: jimmyxie-figma <jimmyxie-figma@users.noreply.github.com> Co-authored-by: Colin Rogers <colin.rogers@dbtlabs.com>
…active-table # Conflicts: # dbt-snowflake/src/dbt/adapters/snowflake/impl.py # dbt-snowflake/src/dbt/include/snowflake/macros/adapters.sql
New snowflake_interactive_warehouses config attaches the table to each listed warehouse after every run, like grants.
|
Took another pass after the rework — the static/dynamic split looks great. Static interactive tables now Blocking: Warehouse names are interpolated raw in the attach DDL. alter warehouse {{ warehouse }} add tables ({{ relation }})
One thing to confirm when you fix it: quoting changes Snowflake's case semantics (an unquoted Non-blocking follow-ups: 1. 2. |
|
@colin-rogers-dbt Thanks for the reviewing pass. Addressed all three: Blocking — raw warehouse interpolation. Good catch. I went with a verbatim identifier contract rather than adapter.quote(), to stay consistent with snowflake_warehouse (which is interpolated verbatim in the create/replace DDL two lines away) and to preserve Snowflake's uppercase folding — quoting analytics_wh → "analytics_wh" would make it case-sensitive and break the common bare-name case. The macro now documents the contract explicitly, and the PR body / config docs spell it out: write the name as Snowflake expects, and wrap case-sensitive or special-character names in double quotes in the config string (e.g. '"my-wh"'). (cbb2114) 1 — format/case-sensitivity of change detection. Left the behavior as-is since it matches dynamic_table (same bare-paren strip, raw warehouse equality, no LINEAR(...)/case normalization), and added a comment at parse_relation_results recording exactly what you noted: the dynamic no-op / multicol functional tests are the guard against a live-account format surprise; the unit tests mock SHOW output so they won't catch one. (7a4e59b) 2 — dead requires_full_refresh. Dropped the changeset-level property (nothing reads it — no ALTER INTERACTIVE TABLE, so the materialization only branches on configuration_changes is none). Kept the three ...ConfigChange properties since they satisfy the abstract RelationConfigChange contract, with a one-line comment saying so. (ce80bbc) |
xuliangs
left a comment
There was a problem hiding this comment.
My comments can be addressed in a separate PR. Thank you.
|
|
||
|
|
||
| # There is no ALTER INTERACTIVE TABLE, so every change replaces and the materialization | ||
| # only branches on `configuration_changes is none`. requires_full_refresh is never read; it |
There was a problem hiding this comment.
Did we pressure test the idempotent nature of dbt with how we are bringing in snowflake interactive tables? What is the intended --full-refresh behavior?
There was a problem hiding this comment.
Static interactive tables (cluster_by only):
- Always rebuilds via
CREATE OR REPLACEevery run. This is unavoidable because Snowflake returnsNULLfor query text inSHOW INTERACTIVE TABLES, so there's no way to diff and check if it changed.. - Fully idempotent — running twice produces identical results.
- SQL changes are reflected on the next run because we rebuild every time anyway.
- Warehouse attachment: CREATE OR REPLACE automatically drops the old table and detaches it from all warehouses. Then dbt re-attaches to every listed warehouse via ALTER WAREHOUSE ... ADD TABLES(), making the config authoritative each run.
Dynamic interactive tables (target_lag + warehouse):
- Config-change detection on
cluster_by,target_lag,snowflake_warehouse. - No rebuild if config unchanged → skip
CREATE OR REPLACE, preserve auto-refresh window. - Idempotent when config unchanged; safe when config changes (all changes trigger rebuild).
- Warehouse attachment runs similarly to static.
Test coverage: All testing performed against live Snowflake accounts (both unit/functional tests in the PR and separate live project validation).
--full-refresh Behavior
For dynamic tables: --full-refresh forces a rebuild regardless of config changes (bypasses the change-detection no-op). Intended behavior to keep it aligned with "normal" dynamic_table.
For static tables: --full-refresh has no additional effect since we are anyways rebuilding on every run.
There was a problem hiding this comment.
I don't think this addresses what I am asking about because this response (some of it likely AI) is referring to dynamic tables. Dynamic tables and Interactive tables are fundamentally different -- dynamic is for real time data coming in while interactive is for low query latency.
There was a problem hiding this comment.
It is partially generated by AI. But interactive tables have 2 types: static and dynamic. Dynamic interactive table accepts target lag to keep the data up to date.
Problem
Snowflake's interactive tables (now GA) — optimized for low-latency, high-concurrency queries — had no dbt materialization.
Solution
This carries forward the great community contribution from @jimmyxie-figma in #1798, which added the original implementation. That PR stalled while awaiting review feedback, so we've picked it up and applied the requested fixes so it can land against
main. Many thanks to @jimmyxie-figma for the original work! 🙏Adds a
materialized='interactive_table'materialization supporting static and dynamic (target_lag) variants. Since Snowflake has noALTER INTERACTIVE TABLE, all rebuilds go throughCREATE OR REPLACE:cluster_byonly): doesn't auto-refresh, andSHOW INTERACTIVE TABLESreturns no query to diff — so it rebuilds viaCREATE OR REPLACEon every run, like a normaltable. A SQL change is reflected on the next run.target_lag+snowflake_warehouse): uses config-change detection oncluster_by/target_lag/snowflake_warehouse(matchingdynamic_table). A SQL-only change is picked up with--full-refresh.Interactive warehouse attachment
Adds a
snowflake_interactive_warehousesconfig (a warehouse name or a list). After each run the table is attached to every listed interactive warehouse viaALTER WAREHOUSE <wh> ADD TABLES (<this>)— modeled ongrants, so it runs every run and the config stays authoritative.{{ config( materialized='interactive_table', cluster_by='id', snowflake_interactive_warehouses=['my_iw'], ) }}Behavior & limitations
CREATE OR REPLACEdrops all attachments; a dynamic table that no-ops will not detach a de-listed warehouse. Full reconcile (detaching de-listed warehouses) is intentionally out of scope.MODIFYrequirement; an over-cap or unauthorized attach surfaces the Snowflake error directly rather than a dbt-side pre-check.MODIFYon each.Review fixes applied on top of the original branch:
SHOW INTERACTIVE TABLES(using therefresh_warehouseandtarget_lagcolumns).cluster_byvalues to avoid spurious rebuilds on unchanged tables.queryfield is now optional —SHOW INTERACTIVE TABLESreturns a NULL query for static tables, which previously crashed the second run. Verified end-to-end against a live Snowflake account, with unit + functional regression tests added.Resolves #1780.