Skip to content

Releases: dagster-io/dagster

1.11.14 (core) / 0.27.14 (libraries)

09 Oct 20:17
Compare
Choose a tag to compare

New

  • UnionTypes (e.g. Foo | Bar) are now supported in ConfigurableResources. (Thanks, @DominikHallab!)
  • Added an output_metadata parameter to build_output_context (Thanks, @zyd14!)
  • ResolvedAssetSpec and related resolvers now support setting the freshness_policy field.
  • dagster-dbt project prepare-and-package --components . will no longer attempt to load components outside of DbtProjectComponent, preventing errors when attempting to run this command in environments that do not have the necessary env vars set for other components.
  • click<8.2 upper bound has been removed
  • [dagster-airbyte][dagster-fivetran][dagster-powerbi][dagster-sling][dagster-dlt] The AirbyteWorkspaceComponent, FivetranAccountComponent, PowerBIWorkspaceComponent, SlingReplicationCollectionComponent, and DltLoadCollectionComponent components have been updated to include a get_asset_spec method that can be overridden by subclasses to modify translation behavior.
  • [dagster-airbyte][dagster-fivetran] The AirbyteWorkspaceComponent and FivetranAccountComponent have been updated to include an execute() method that can be overridden by subclasses to modify runtime execution behavior.
  • [dagster-airbyte] The AirbyteWorkspaceComponent no longer binds an "io_manager" or "airbyte" resource, meaning it can be used in tandem with other definitions that use those keys without causing conflicts.
  • [dagster-dbt] The DbtProjectComponent now supports overriding get_asset_spec and get_asset_check_spec methods when subclassing.
  • [dagster-fivetran] The FivetranAccountComponent no longer binds an "io_manager" or "fivetran" resource, meaning it can be used in tandem with other definitions that use those keys without causing conflicts.

Bugfixes

  • Fixed a bug that would cause errors when instantiating a TimeWindowPartitionsDefinition with a monthly schedule and the default day offset.
  • [ui] The Materialize button in the lineage view is now disabled while the view is updating to avoid inconsistencies when launching runs.
  • [ui] Fixed an issue where the "View error" link in the popup that displayed when a backfill failed to launch was very difficult to see.
  • [dagster-dbt] Fixed issue where the select and exclude parameters to @dbt_assets would be ignored when generating AssetCheckSpecs (Thanks, @nathanskone!)
  • [dagster-powerbi] Previously, assets generated for semantic models would have a kind tag with an invalid space character ("semantic model"). Now, they have the kind tag "semantic_model".
  • [dagster-sling] Resolved issue that caused the substring "INF" to be stripped from all logs.

1.11.13 (core) / 0.27.13 (libraries)

02 Oct 18:16
Compare
Choose a tag to compare

New

  • [dagster-deltalake,dagster-deltalake-polars] BREAKING CHANGE - we now support deltalake>=1.0.0 for dagster-deltalake and dagster-deltalake-polars and we will no longer support deltalake<1.0.0 moving forward. End user APIs remain the same for both libraries.
  • [dagster-databricks] Spark Python and Python Wheel tasks are now supported in PipesDatabricksServerlessClient.
  • [dagster-dbt] dagster-dbt project prepare-and-package --components . will no longer attempt to load components outside of DbtProjectComponent, preventing errors when attempting to run this command in environments that do not have the necessary env vars set for other components.
  • [dg] adds dg api secret list and dg api secret get

Bugfixes

  • Fixed a bug in the backfill daemon where an asset backfill with CANCELING​ or FAILING​ status could become permanently stuck in CANCELING​ or FAILING​ if the partitions definitions of the assets changed.
  • Fixed an issue introduced in the 1.11.12 release where auto-complete in the Launchpad for nested fields stopped working.
  • Fixed an issue where backfills would fail if a TimeWindowPartitionsDefinition's start date was changed in the middle of the backfill, even if it did not remove any of the targeted partitions.
  • [ui] Fixed the link to "View asset lineage" on runs that don't specify an asset selection.

1.11.12 (core) / 0.27.12 (libraries)

25 Sep 20:06
Compare
Choose a tag to compare

New

  • [ui] Allow searching across code locations with * wildcard in selection inputs for jobs and automations.
  • [ui] Added AutomationCondition.all_new_executed_with_tags, which allows automation conditions to be filtered to partitions that have been materialized since the last tick from runs with certain tags. This condition can be used to require or prevent certain run tags from triggering downstream declarative automation conditions.

Bugfixes

  • In dagster==1.11.1, partitioned_config was unintentionally removed from the public exports of the top-level dagster package. This has been fixed.
  • Avoid adding trailing whitespace in env vars that use dot notation in components. Thanks @edgarrmondragon!
  • [dagster-airbyte] Fix the pagination url issue for the Airbyte API. Thanks @stevenayers!
  • [dagster-dbt] Fixed an issue with the DbtCloudWorkspaceClient that would cause errors when calling trigger_job_run with no steps_override parameter.

Dagster Plus

  • [ui] Add Cost insights.
  • [ui] For users who have inherited org roles from a team, show those roles when editing the user.
  • [ui] Fix per-asset and per-job insights graphs.

1.11.11 (core) / 0.27.11 (libraries)

18 Sep 21:22
Compare
Choose a tag to compare

New

  • anthropic, mcp, and claude-code-sdk dependencies of dagster-dg-cli are now under a separate ai extra, allowing dagster-dg-cli to be installed without these dependencies.
  • Added AutomationCondition.all_new_updates_have_run_tags and AutomationCondition.any_new_update_has_run_tags, which allows automation conditions to be filtered to partitions that have been materialized since the last tick from runs with certain tags. This condition can be used to require or prevent certain run tags from triggering downstream declarative automation conditions. These conditions are similar to AutomationCondition.executed_with_tags, but look at all new runs since the most recent tick instead of just looking at the latest run.

Bugfixes

  • Fixed a bug which would cause steps downstream of an asset with skippable=True and a blocking asset check to execute as long as the asset check output was produced, even if the asset output was skipped.
  • When a backfill fails, it will now cancel all of its in-progress runs before terminating.
  • Fixed an issue that would cause trailing whitespace to be added to env vars using dot notation ({{ env.FOO }}) when listing the env vars used by a component. (Thanks, @edgarrmondragon!)
  • Fixed issue that would cause errors when using multi to single partition mappings with DbIOManagers.
  • [ui] Fixed issue with the "Report materialization" dialog for non-partitioned assets.
  • [ui] Typing large YAML documents in the launchpad when default config is present is now more performant.
  • [ui] Fixed an issue where setting a FloatMetadataValue to float('inf') or float('-inf') would cause an error when loading that metadata over graphql.
  • [ui] The "Clear" button in the dimension partition text input for multi-partitioned assets now clears invalid selections as expected.
  • [dagster-dbt] Fixed an issue with the DbtCloudWorkspaceClient that would cause errors when calling trigger_job_run with no steps_override parameter.

1.11.10 (core) / 0.27.10 (libraries)

11 Sep 17:21
Compare
Choose a tag to compare

New

  • Added inline-component command to the publicly available scaffold commands in the Dagster CLI.
  • Added a new require_upstream_step_success config param to all executors. If {"step_dependency_config": {"require_upstream_step_success": False}} is set, this will allow downstream steps to execute immediately after all required upstream outputs have finished, even if the upstream step has not completed in its entirety yet. This can be useful particularly in cases where there are large multi-assets with downstream assets that depend on only a subset of the assets in the upstream step.
  • The logsForRun​ resolvers and eventConnection​ resolvers in the Dagster GraphQL API will now apply a default limit of 1000 to the number of logs returned from a single graphql query. The cursor​ field in the response can be used to continue iterating through the logs for a given run.
  • [dagster-airbyte] @airbyte_assets and AirbyteWorkspaceComponent (previously AirbyteCloudWorkspaceComponent) now support Airbyte OSS and Enterprise.

Bugfixes

  • Fixed an issue where the dagster_dg_cli package failed to import when using Python 3.9.
  • Fixed an issue with AutomationCondition.eager() that could cause runs for materializable assets to be launched at the same time as an upstream observable source asset that had an automation condition, even if the upstream observation would not result in a new data version.
  • Fixed an issue which could, in some circumstances, cause errors during Declarative Automation evaluation after a dynamic partition was deleted.
  • Fixed an issue that could cause confusing errors when attempting to supply attributes configuration to Component subclasses that did not inherit from Resolvable.
  • Added a Matillion kind tag, thanks @RobBrownFreeAgent!
  • [ui] Fixed an issue where the "Report materialization events" dialog for partitioned assets only worked if the partition was failed or missing.
  • [ui] Fixed a browser crash which could occur in the global asset graph.
  • [ui] Fixed a bug with the sensor preview behavior that would cause run requests contianing run_keys that had already been submitted to show up in the preview result.
  • [dagster-dbt] Fixed an issue that would cause the DbtCloudWorkspace to error before yielding asset events if the associated dbt Cloud run failed. Now, it will raise the error after all relevant asset events have been produced.
  • [dagster-dbt] Added the dbt-core dependency back to dagster-dbt as it is still required for the dbt Cloud integration. If both dbt-core and dbt Fusion are installed, dagster-dbt will still prefer using dbt Fusion by default.

Documentation

  • Introduced a new "Post-processing components" guide.
  • Fixed incorrect YAML code snippets for alert policies docs page.
  • Fixed incorrect chart keys in Helm documentation. Thanks, @charlottevdscheun!
  • Fixed incorrect owner tags in Components docs. Thanks, @aaronprice00!

Dagster Plus

  • Improved the Dagster+ agent's retry behavior during when it experiences outbound connection timeouts while a code location is being deployed.

1.11.9 (core) / 0.27.9 (libraries)

04 Sep 22:59
Compare
Choose a tag to compare

New

  • Subclasses of Resolved now support fields of type dict[str, T].
  • [ui] Added a new 'arrow' icon to the set of supported kind tags (thanks @aleewen!)

Bugfixes

  • Launching a backfill of a non-subsettable multi-asset without including every asset will now raise a clear error at backfill submission time, instead of failing with a confusing error after the backfill has started.
  • Fixed an issue where passing in an empty list to the assetKeys argument of the assetsOrError field in the GraphQL API would return every asset instead of an empty list of assets.
  • [dagster-dbt] Fixed an issue that would cause the DbtCloudWorkspace to error before yielding asset events if the associated DBT Cloud run failed. Now, it will raise the error after all relevant asset events have been produced.

Dagster Plus

  • Serverless pex builds now support pyproject.toml-based packages.

1.11.8 (core) / 0.27.8 (libraries)

28 Aug 21:52
Compare
Choose a tag to compare

New

  • A param exclusions was added to time window partition definitions to support custom calendars.
  • The dagster library now supports protobuf==6.x
  • [dg] dg scaffold defs --help now shows descriptions for subcommands.
  • [dg] A new dg check toml command has been added to validate your TOML configuration files.
  • [dagster-databricks] The DatabricksAssetBundleComponent has been added in preview. Databricks tasks can now be represented as assets and submitted via Dagster.
  • [dagster-dbt] The DbtProjectComponent now takes an optional cli_args configuration to allow customizing the command that is run when your assets are executed.
  • [dagster-dbt] The polling interval and timeout used for runs triggered with the DbtCloudWorkspace resource can now be customized with the DAGSTER_DBT_CLOUD_POLL_INTERVAL and DAGSTER_DBT_CLOUD_POLL_TIMEOUT environment variables.
  • [ui] Added the ability to filter to failed/missing partitions in the asset report events dialog.
  • [ui] A tree view has been added in the Global Asset Lineage.
  • [telemetry] Telemetry disclaimer now prints to stderr.

Bugfixes

  • Fixed an issue that would require config provided to backfills to contain config for all assets in the code location rather than just the selected ones.

1.11.7 (core) / 0.27.7 (libraries)

22 Aug 18:25
Compare
Choose a tag to compare

New

  • dg will now report multiple detected errors in a configuration file instead of failing on the first detected error.
  • It is now possible to supply run config when launching an asset backfill.
  • Updated the root URL to display the Overview/Timeline view for locations with schedules/automations, but no jobs (thanks @dschafer!)
  • Added tzdata as a dependency to dagster, to ensure that declaring timezones like US/Central work in all environments.
  • [dagster-dg-cli] Updated scaffolded file names to handle consecutive upper case letters (ACMEDatabricksJobComponent → acme_databricks_job_component.py not a_c_m_e_databricks_job_component.py)
  • [dagster-dg-cli] Validating requirements.env is now opt-in for dg check yaml.
  • [dagster-dbt] DAGSTER_DBT_CLOUD_POLL_INTERVAL and DAGSTER_DBT_CLOUD_POLL_TIMEOUT environment variables can now be used to configure the polling interval and timeout for fetching data from dbt Cloud.

Deprecations

  • [components] Removed deprecated and non-functional asset_post_processors fields from SlingReplicationCollectionComponent and AirflowInstanceComponent.

1.11.6 (core) / 0.27.6 (libraries)

14 Aug 21:31
Compare
Choose a tag to compare

New

  • Allow explicit git platform selection in link_code_references_to_git, thanks @chazmo03!

Bugfixes

  • Fixed issue causing AutomationCondition.replace to not update built-in sub-conditions that did not have an explicit label, thanks @dschafer!
  • Fixed an issue where assets were considered stubs if they were a stub in any code location.
  • Projects using components no longer cause "job definitions changed while uploading" errors on older agent versions.
  • [dagster-dbt] Fixed a bug that could cause execution to fail if enable_code_references was set to True on the DagsterDbtTranslatorSettings.

Documentation

  • Updated documentation of dagster.yaml to include the nux option, thanks @dwisdom0!

Dagster Plus

  • Fix "Create a support ticket" dialog submissions.

1.11.5 (core) / 0.27.5 (libraries)

07 Aug 22:10
Compare
Choose a tag to compare

New

  • Static functions on classes decorated with @template_var can now optionally accept a ComponentLoadContext argument.
  • [dg] A MCP server is available to expose dg CLI capabilities to MCP clients. See the dg mcp CLI group for details.
  • [dagster-dbt] The dagster-dbt package no longer has a dependency on dbt-core.
  • [dagster-dbt][preview] Users of the dbt Fusion CLI can now use the dagster-dbt package to run dbt commands with no changes to their existing dagster code. This support is still in preview as the format of the log messages produced by the dbt Fusion CLI is still subject to change. Let us know if you notice any incompatibilities.
  • [dagster-databricks] Added a PipesDatabricksServerlessClient to support Databricks Serverless jobs with Dagster pipes.
  • [dagster-databricks] Added additional options for cluster configuration (thanks @jmccartin!)

Bugfixes

  • Various bugfixes for backfills that target assets which change their partitions definition mid-backfill.
  • [ui] Fixed issue that could cause errors related to the ObjectMetadataValue class.

Documentation

  • Added docs for using Spark Connect and Databricks Connect with Dagster.