Office hours live stream [October 10, 2025]

[C00ldudeNoonan]

Upcoming office hours

Hey everyone!

We are going to be hosting an event next Friday, October 10th at 2 PM EST / 11 AM PT, to give you all the opportunity to ask questions that we will answer live. Need some help troubleshooting some code? Want to ask about an upcoming feature? Or do you just want to talk about the data engineering landscape? We'll be happy to chat about all of that.

https://dagster.io/events/october-community-office-hours

You can use this thread to provide questions before hand, otherwise you are also welcome to ask them at the time of the event.

The event will be recorded and uploaded to our YouTube channel if you're unable to make it.

[OrionYeung]

@dehume would you share the gist link to sample usage of resources for cache?

[dehume]

https://gist.github.com/dehume/6ff8e04317b70c9b8966be79001453b3

[jimmyzzxhlh]

Hi! I'd like to cover a couple of questions during the office hours mentioned in the following threads:

• How to configure Python structlog to work with Dagster? #32284
• UI events incorrectly propagated to backdrop when configuring a Dagster asset in Launchpad #31104

[kinghuang]

How do I figure out what template variables are available in a component, and what their properties are? For example, the Sling component has stream_definition available as a template variable. But, other than by word of mouth, how do I figure that out? And, what's available on the stream_definition object other than name?

I'm most interested in knowing this for the dbt and Sling components.

[dehume]

Short version: today there’s no “official” CLI/docs page that lists a component’s template variables. The reliable way is to look at the component’s translation code (where integration-specific context is injected into the template). For Sling, stream_definition is injected there and is effectively a dict coming from Sling’s own spec so its available fields are whatever Sling provides (not just name). A proper, discoverable listing is planned but not yet wired up.

What’s available on stream_definition?

Your Sling replication config and Sling’s model produce for that stream. In your example, you’ll at least see the stream’s name (gml_data) and anything inherited/overridden from defaults like mode: incremental and update_key: row_id, plus any stream-specific fields you add (e.g., explicit source table, schema, filters, etc.). Since it’s not a Dagster-typed object, don’t expect IDE type hints—inspect at runtime or read the Sling component translation code.

[kinghuang]

I'm poring through the code already. But, as noted on the stream, it's a chore!

How do I inspect this at runtime?

[kinghuang]

I have assets created via a Sling component. The Sling streams are configured for incremental replication. Here's an example replication config.

source: MY_SNOWFLAKE
target: MY_DUCKLAKE

defaults:
  mode: incremental
  update_key: row_id

streams:
  gml_data:

How should I implement checks for the resulting Dagster assets so that they check the value of row_id, and trigger materializations if there are new rows to replicate? Ideally, this would be driven by the Sling replication config.

[dehume]

You’ve got two solid patterns—pick based on how “smart” you want the trigger to be.

1. Run Sling on a schedule. Since Sling manages incremental state (update_key: row_id), the easiest setup is to run the Sling component on a cadence (automation condition or schedule). It will only move rows with row_id > last_synced_row_id.
   Pros: minimal code; uses Sling’s own state.
   Cons: you’ll do a no-op run when there’s nothing new.

2. Create a sensor that reads the config and checks MAX(row_id). This avoids no-op runs by checking whether new rows exist before triggering Sling. Parse your Sling replication config (YAML) to discover which streams exist, the update_key (fallback to defaults.update_key), the source table/view each stream reads (derive from your naming or keep it explicit in the YAML).

   Query the source system (Snowflake or the target, if that’s more convenient) for each stream’s MAX(update_key). Compare with a persisted cursor (stored in the sensor cursor or a small key–value table). If increased, trigger a materialization for that Sling stream’s Dagster asset(s) (and advance the cursor). If unchanged, do nothing.

[kinghuang]

Thanks for going over this on Friday! My question is related to #14758 and observable source assets. As it stands, the Sling component is declaring stub dependencies for every one of my Sling streams. I'm wondering if there's a good way to leverage that by recording the row_id on the stub/external asset, observing changes, and setting automation conditions on the stream derived assets to materialize when there's changes.

With respect to your second suggestion, Dagster already has the concept of a DataVersion. Can I somehow integrate/show Sling's incremental state tracking with that?

I could be completely overcomplicating this, too. 😅

[mathieu-coast]

2 questions:

1. Would love to hear an authoritative best practices about managing a library of reusable resources (and io managers) in a shared library

• how to test them (esp. best practices for fixtures)
• how to populate the resources in a downstream pipeline and add those to definitions - dagster configuration vs. env vars or an in-house configuration solution
• how to test assets that leverage these resources in integration tests
• some more tricky contexts like an io manager which relies on another resource - like a redshift io manager needing a redshiftClient (passed in as a RedshiftClientResource)

2. How to better understand (and control!) when assets are materialized vs. skipped in our pipelines
   for example: I have an upstream asset A that reads a table from my DWH and then downstream assets B,C,D... if I run multiple runs like in a backfill, I know the data in A won't change, so there is no point to re-materialize it obsessively. What's the right technique to apply in such cases?

[dehume]

1. Where to put shared code. Use a shared module and import it from all code locations. The Dagster example shows three workable patterns:

   • Colocate the shared package in the monorepo and add it as an editable path dependency.
   • Git submodule the shared package into each repo.
   • Publish the shared package to a private registry (CodeArtifact / Artifact Registry / Azure Artifacts) and version-pin it per code location.

2. The Dagster University “Testing with Dagster” course walks through unit tests, mocks, and integration tests end-to-end. Use its structure as your backbone: unit tests for pure transforms, mocked resources/IO managers for boundary tests, and selective integration tests where you stand up the real client or a lightweight local alternative.

[miriamcastel]

How can I determine the impact of changing the behaviour of an automation condition widely used in a large dbt project, especially when the dependency graph is too complex for manual tracing?

[dehume]

Here’s a practical, safe way to assess the blast radius of an automation-condition change in a big dbt/Dagster graph:

1. First line of defense: standardize conditions. Before testing, shrink the problem. Replace bespoke per-asset conditions with a small set (e.g., hourly, 3_hourly, eager). Fewer distinct behaviors = less surface area to reason about and validate.

2. High-fidelity check: staging environment. Spin up a staging (or “shadow”) Dagster instance with a copy of your metadata DB and code. Deploy the condition change there and observe. This is the most accurate method because it exercises the real scheduler, automations, and run coordination.

3. Fast iteration: unit-test simulation with evaluate_automation_conditions. Use Dagster’s programmatic evaluation API to simulate decisions without launching runs.

[miriamcastel]

How can we prevent concurrent dbt snapshot runs, from both automated and manual runs, which can corrupt our snapshot tables? We are using declarative automation.

[dehume]

Use step-level concurrency limits with pools, and isolate your snapshots into their own dbt multi-asset(s).

What to do

1. Split snapshots from everything else. Create one @dbt_assets for only snapshots (via select or exclude). Put the rest of your dbt models in a separate @dbt_assets.

2. Put the snapshot step(s) in a pool of size 1. Assign op_tags={"dagster-pool": "dbt-snapshots"} (or executor config) to the snapshot multi-asset. Configure the "dbt-snapshots" pool with max concurrency = 1. All executions—manual materializations and automation-triggered runs—respect pools, so you won’t get two snapshot steps at once.

3. If you have many snapshots, create several snapshot multi-assets (e.g., 10 each by selector) and:

• Either keep them all in the same pool of size 1 (strictly serializing all snapshots), or
• Use multiple pools (e.g., size 1 each per group) if you want limited parallelism without cross-table contention.

[miriamcastel]

When multiple dbt models are materialised together in a single run, if one model fails, the entire run is marked as failed, which can incorrectly trigger the execution_failed() automation condition for all assets in that run, even those that succeeded. Is there a way to prevent this?

[dehume]

Yes, this was identified as a bug. A fix is landing in the next Dagster release, so upgrading will make execution_failed() evaluate correctly per-asset instead of treating the whole multi-model dbt run as failed.

[dehume]

Thanks for all the questions. Tried to reply and answer for every question based on our discussion during office hours. Here is the recording if you want to see your question discussed in more detail.

[jimmyzzxhlh]

Can I get my question addressed as it was not addressed during the office hours? I posted a video to show how the bug is reproduced.
#31104