Skip to content

Commit 6a02f16

Browse files
maximearmstrongDagster Devtools
authored andcommitted
[connections] document configurable connection freshness (#24474)
## Summary & Motivation As title Internal-RevId: be78fb47b0999cc0382f3ba9b730a2c7b637d931
1 parent d09dc3c commit 6a02f16

4 files changed

Lines changed: 28 additions & 0 deletions

File tree

docs/docs/guides/labs/connections/bigquery.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -161,6 +161,8 @@ Use filtering to control which projects, datasets, tables, and views are synced.
161161

162162
## Connection freshness
163163

164+
import ConnectionFreshnessCadence from '@site/docs/partials/\_ConnectionFreshnessCadence.md';
165+
164166
Once a BigQuery Connection is created, Dagster automatically tracks changes to your BigQuery tables and emits a materialization on the corresponding Connection asset each time the underlying table is updated in BigQuery. As a result, your Connection assets reflect the most recent state of the BigQuery tables without needing to define a sensor or schedule.
165167

166168
### Prerequisites
@@ -174,6 +176,10 @@ The roles granted in [Grant required permissions](#step-12-grant-required-permis
174176

175177
Tables must live in a region included in [`region_qualifiers`](#optional-configure-region-qualifiers) (defaults to `region-us` and `region-eu`); changes in regions not listed there will not be detected.
176178

179+
### Configuring the cadence
180+
181+
<ConnectionFreshnessCadence />
182+
177183
### Triggering downstream assets
178184

179185
Each detected change produces a standard Dagster materialization event on the Connection asset, so code-defined assets can depend on a Connection asset like any other upstream. Combine that with an [Automation Condition](/guides/automate/declarative-automation) to materialize a downstream asset whenever its BigQuery parent changes:

docs/docs/guides/labs/connections/databricks.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -145,6 +145,8 @@ Use filtering to control which catalogs, schemas, tables, views, and notebooks a
145145

146146
## Connection freshness
147147

148+
import ConnectionFreshnessCadence from '@site/docs/partials/\_ConnectionFreshnessCadence.md';
149+
148150
When the Databricks Connection has a SQL warehouse configured, Dagster automatically tracks changes to your Unity Catalog tables and emits a materialization on the corresponding Connection asset each time the underlying table is updated in Databricks. As a result, your Connection assets reflect the most recent state of the Databricks tables without needing to define a sensor or schedule.
149151

150152
### Prerequisites
@@ -156,6 +158,10 @@ Connection freshness for Databricks shares the same configuration as lineage tra
156158

157159
If no warehouse is configured, freshness signals are not emitted.
158160

161+
### Configuring the cadence
162+
163+
<ConnectionFreshnessCadence />
164+
159165
### Triggering downstream assets
160166

161167
Each detected change produces a standard Dagster materialization event on the Connection asset, so code-defined assets can depend on a Connection asset like any other upstream. Combine that with an [Automation Condition](/guides/automate/declarative-automation) to materialize a downstream asset whenever its Databricks parent changes:

docs/docs/guides/labs/connections/snowflake.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -128,6 +128,8 @@ Use filtering to control which databases, schemas, and tables are synced. Patter
128128

129129
## Connection freshness
130130

131+
import ConnectionFreshnessCadence from '@site/docs/partials/\_ConnectionFreshnessCadence.md';
132+
131133
Once a Snowflake Connection is created, Dagster automatically tracks changes to your Snowflake tables and emits a materialization on the corresponding Connection asset each time the underlying table is updated in Snowflake. As a result, your Connection assets reflect the most recent state of the Snowflake tables without needing to define a sensor or schedule.
132134

133135
### Prerequisites
@@ -136,6 +138,10 @@ Connection freshness for Snowflake reuses the warehouse, role, and credentials a
136138

137139
The privileges granted in [Step 1.1: Create role and user to use with Dagster Connections](#step-11-create-role-and-user-to-use-with-dagster-connections) (`USAGE` on databases and schemas, `REFERENCES` on tables and views) are sufficient for the role to see rows in `INFORMATION_SCHEMA.TABLES` for the objects it has access to.
138140

141+
### Configuring the cadence
142+
143+
<ConnectionFreshnessCadence />
144+
139145
### Triggering downstream assets
140146

141147
Each detected change produces a standard Dagster materialization event on the Connection asset, so code-defined assets can depend on a Connection asset like any other upstream. Combine that with an [Automation Condition](/guides/automate/declarative-automation) to materialize a downstream asset whenever its Snowflake parent changes:
Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
By default, Dagster polls the source for changes every minute. You can change the cadence in the **Freshness** tab of the connection editor.
2+
3+
Available presets:
4+
5+
- **Every minute** (default)
6+
- **Every 5 minutes**
7+
- **Every 15 minutes**
8+
- **Custom cron** — any standard 5-field cron string (e.g. `0 */2 * * *`)
9+
10+
A timezone selector accompanies the cron, which is useful when you pick a less frequent schedule (for example, "every day at 2 AM in America/New_York"). Slower cadences reduce queries against your source at the cost of longer worst-case detection lag.

0 commit comments

Comments
 (0)