|
| 1 | +# SnowflakeDbtProjectComponent |
| 2 | + |
| 3 | +Run a dbt project **natively on Snowflake** (the |
| 4 | +[dbt Projects on Snowflake](https://docs.snowflake.com/en/user-guide/data-engineering/dbt-projects-on-snowflake) |
| 5 | +feature) and expose it to Dagster as a set of assets. |
| 6 | + |
| 7 | +Unlike `dagster_dbt.DbtProjectComponent` (which runs `dbt` locally) this component never executes |
| 8 | +dbt on the host. It is modeled on `dagster_dbt.DbtCloudComponent`: there is **no local copy of the |
| 9 | +dbt project**, and execution happens remotely — here, inside Snowflake's managed runtime via |
| 10 | +`EXECUTE DBT PROJECT`. |
| 11 | + |
| 12 | +> **Preview.** This component is in preview and may change in patch releases. |
| 13 | +
|
| 14 | +## Prerequisites |
| 15 | + |
| 16 | +1. The dbt project is already deployed to Snowflake as a `DBT PROJECT` object |
| 17 | + (`CREATE DBT PROJECT …` / `snow dbt deploy`). |
| 18 | +2. Install the optional dbt extra: |
| 19 | + |
| 20 | + ```bash |
| 21 | + pip install 'dagster-snowflake[dbt]' |
| 22 | + ``` |
| 23 | + |
| 24 | +3. A **non-interactive** Snowflake credential. Key-pair auth is strongly recommended (it is exempt |
| 25 | + from MFA and works headless). Do **not** use `authenticator: externalbrowser` (SSO) — it can't |
| 26 | + complete from a Dagster run and fails with a SAML error. |
| 27 | + |
| 28 | +## Quickstart |
| 29 | + |
| 30 | +```yaml |
| 31 | +# defs.yaml |
| 32 | +type: dagster_snowflake.SnowflakeDbtProjectComponent |
| 33 | +attributes: |
| 34 | + snowflake_dbt_project_name: "analytics.dbt.jaffle_shop" # database.schema.project |
| 35 | + snowflake: |
| 36 | + account: "{{ env.SNOWFLAKE_ACCOUNT }}" |
| 37 | + user: "{{ env.SNOWFLAKE_USER }}" |
| 38 | + private_key_path: "{{ env.SNOWFLAKE_PRIVATE_KEY_PATH }}" |
| 39 | + private_key_password: "{{ env.SNOWFLAKE_PRIVATE_KEY_PASSPHRASE }}" # omit if key is unencrypted |
| 40 | + role: TRANSFORMER |
| 41 | + warehouse: TRANSFORMING |
| 42 | + database: ANALYTICS |
| 43 | + schema: DBT |
| 44 | + cli_args: [build] |
| 45 | + include_metadata: [row_count, column_metadata] # optional, see below |
| 46 | +``` |
| 47 | +
|
| 48 | +## How it works |
| 49 | +
|
| 50 | +- **Asset graph (state refresh).** The component fetches the dbt manifest _from Snowflake_: it runs |
| 51 | + `EXECUTE DBT PROJECT … ARGS='parse'`, locates the artifacts with |
| 52 | + `SYSTEM$LOCATE_DBT_ARTIFACTS(<query_id>)`, downloads `manifest.json`, and caches it as the |
| 53 | + component's defs-state. No dbt artifacts are committed to your repo. (Use `manifest_args: [compile]` |
| 54 | + if your project needs compilation to parse.) |
| 55 | +- **Execution.** `EXECUTE DBT PROJECT … ARGS='build …'` runs over the `SnowflakeResource` |
| 56 | + connection. This is **synchronous** — the statement blocks until the dbt run finishes (there is no |
| 57 | + intermediate streaming). The dbt log is then fetched via `SYSTEM$GET_DBT_LOG` and surfaced in the |
| 58 | + Dagster logs. |
| 59 | +- **Materializations & metadata.** After the run, `run_results.json` is downloaded and translated |
| 60 | + into `MaterializeResult`s / `AssetCheckResult`s with metadata at parity with the dbt core and dbt |
| 61 | + Cloud components (`unique_id`, `invocation_id`, `execution_duration`, completed-at timestamp, test |
| 62 | + status), plus the static dbt metadata carried on each `AssetSpec`. |
| 63 | + |
| 64 | +## Subsetting |
| 65 | + |
| 66 | +`can_subset=True`. When Dagster launches a subset, the component builds a `--select`ed |
| 67 | +`EXECUTE DBT PROJECT` statement covering exactly the selected models/checks, using the same |
| 68 | +selection logic (`get_subset_selection_for_context`) as the dbt core and dbt Cloud components. |
| 69 | + |
| 70 | +## Optional metadata (`include_metadata`) |
| 71 | + |
| 72 | +Off by default (matching dbt core). Each addon runs extra warehouse queries per model after the run: |
| 73 | + |
| 74 | +- `row_count` → `SELECT count(*)` per materialized model (views skipped) → `dagster/row_count`. |
| 75 | +- `column_metadata` → column schema via `DESCRIBE TABLE` plus column-level lineage (derived from the |
| 76 | + run's compiled SQL, extracted from `dbt_artifacts.zip`) → `dagster/column_schema`, |
| 77 | + `dagster/column_lineage`. |
| 78 | + |
| 79 | +## Observation mode (`create_sensor`) |
| 80 | + |
| 81 | +Set `create_sensor: true` to add a polling sensor that reports dbt runs executed in Snowflake |
| 82 | +**outside** Dagster (Snowflake Tasks, Snowsight, `snow dbt execute`). It polls |
| 83 | +`DBT_PROJECT_EXECUTION_HISTORY`, downloads each new run's `run_results.json`, and emits |
| 84 | +`AssetMaterialization`s — analogous to the dbt Cloud component's external-run sensor. |
| 85 | + |
| 86 | +**De-duplication.** Like dbt Cloud (which filters its dedicated adhoc-job runs), Dagster-triggered |
| 87 | +runs are skipped: while the sensor is enabled, `execute()` tags its invocations with a sentinel dbt |
| 88 | +var that appears in the history's `ARGS` column, and the sensor filters those out. So |
| 89 | +Dagster-materialized and externally-triggered runs can coexist without double-reporting. |
| 90 | + |
| 91 | +## Customization |
| 92 | + |
| 93 | +Subclass and override `get_asset_spec` (asset keys/metadata/kinds), `execute` (run behavior), or the |
| 94 | +translation function in YAML — the same extension points as the other dbt components. |
| 95 | + |
| 96 | +## Caveats |
| 97 | + |
| 98 | +- Preview feature; assumes the `DBT PROJECT` object is already deployed (the component does not |
| 99 | + `CREATE OR REPLACE`/upload it). |
| 100 | +- Per-model metadata fetching is sequential (dbt core parallelizes it); fine for typical projects. |
| 101 | +- Observation-mode de-dup relies on the `ARGS` column reflecting the injected marker; if it doesn't, |
| 102 | + the worst case is double-reporting (never a crash). Marker injection only happens when |
| 103 | + `create_sensor` is enabled. |
0 commit comments