Releases: dagster-io/dagster
Releases · dagster-io/dagster
1.11.14 (core) / 0.27.14 (libraries)
New
UnionTypes
(e.g.Foo | Bar
) are now supported inConfigurableResources
. (Thanks, @DominikHallab!)- Added an
output_metadata
parameter tobuild_output_context
(Thanks, @zyd14!) ResolvedAssetSpec
and related resolvers now support setting thefreshness_policy
field.dagster-dbt project prepare-and-package --components .
will no longer attempt to load components outside ofDbtProjectComponent
, preventing errors when attempting to run this command in environments that do not have the necessary env vars set for other components.click<8.2
upper bound has been removed- [dagster-airbyte][dagster-fivetran][dagster-powerbi][dagster-sling][dagster-dlt] The
AirbyteWorkspaceComponent
,FivetranAccountComponent
,PowerBIWorkspaceComponent
,SlingReplicationCollectionComponent
, andDltLoadCollectionComponent
components have been updated to include aget_asset_spec
method that can be overridden by subclasses to modify translation behavior. - [dagster-airbyte][dagster-fivetran] The
AirbyteWorkspaceComponent
andFivetranAccountComponent
have been updated to include anexecute()
method that can be overridden by subclasses to modify runtime execution behavior. - [dagster-airbyte] The
AirbyteWorkspaceComponent
no longer binds an"io_manager"
or"airbyte"
resource, meaning it can be used in tandem with other definitions that use those keys without causing conflicts. - [dagster-dbt] The
DbtProjectComponent
now supports overridingget_asset_spec
andget_asset_check_spec
methods when subclassing. - [dagster-fivetran] The
FivetranAccountComponent
no longer binds an"io_manager"
or"fivetran"
resource, meaning it can be used in tandem with other definitions that use those keys without causing conflicts.
Bugfixes
- Fixed a bug that would cause errors when instantiating a
TimeWindowPartitionsDefinition
with a monthly schedule and the default day offset. - [ui] The
Materialize
button in the lineage view is now disabled while the view is updating to avoid inconsistencies when launching runs. - [ui] Fixed an issue where the "View error" link in the popup that displayed when a backfill failed to launch was very difficult to see.
- [dagster-dbt] Fixed issue where the
select
andexclude
parameters to@dbt_assets
would be ignored when generatingAssetCheckSpecs
(Thanks, @nathanskone!) - [dagster-powerbi] Previously, assets generated for semantic models would have a kind tag with an invalid space character (
"semantic model"
). Now, they have the kind tag"semantic_model"
. - [dagster-sling] Resolved issue that caused the substring "INF" to be stripped from all logs.
1.11.13 (core) / 0.27.13 (libraries)
New
- [dagster-deltalake,dagster-deltalake-polars] BREAKING CHANGE - we now support
deltalake>=1.0.0
fordagster-deltalake
anddagster-deltalake-polars
and we will no longer supportdeltalake<1.0.0
moving forward. End user APIs remain the same for both libraries. - [dagster-databricks] Spark Python and Python Wheel tasks are now supported in
PipesDatabricksServerlessClient
. - [dagster-dbt]
dagster-dbt project prepare-and-package --components .
will no longer attempt to load components outside ofDbtProjectComponent
, preventing errors when attempting to run this command in environments that do not have the necessary env vars set for other components. - [dg] adds
dg api secret list
anddg api secret get
Bugfixes
- Fixed a bug in the backfill daemon where an asset backfill with CANCELING or FAILING status could become permanently stuck in CANCELING or FAILING if the partitions definitions of the assets changed.
- Fixed an issue introduced in the 1.11.12 release where auto-complete in the Launchpad for nested fields stopped working.
- Fixed an issue where backfills would fail if a TimeWindowPartitionsDefinition's start date was changed in the middle of the backfill, even if it did not remove any of the targeted partitions.
- [ui] Fixed the link to "View asset lineage" on runs that don't specify an asset selection.
1.11.12 (core) / 0.27.12 (libraries)
New
- [ui] Allow searching across code locations with
*
wildcard in selection inputs for jobs and automations. - [ui] Added
AutomationCondition.all_new_executed_with_tags
, which allows automation conditions to be filtered to partitions that have been materialized since the last tick from runs with certain tags. This condition can be used to require or prevent certain run tags from triggering downstream declarative automation conditions.
Bugfixes
- In
dagster==1.11.1
,partitioned_config
was unintentionally removed from the public exports of the top-leveldagster
package. This has been fixed. - Avoid adding trailing whitespace in env vars that use dot notation in components. Thanks @edgarrmondragon!
- [dagster-airbyte] Fix the pagination url issue for the Airbyte API. Thanks @stevenayers!
- [dagster-dbt] Fixed an issue with the DbtCloudWorkspaceClient that would cause errors when calling
trigger_job_run
with no steps_override parameter.
Dagster Plus
- [ui] Add Cost insights.
- [ui] For users who have inherited org roles from a team, show those roles when editing the user.
- [ui] Fix per-asset and per-job insights graphs.
1.11.11 (core) / 0.27.11 (libraries)
New
anthropic
,mcp
, andclaude-code-sdk
dependencies ofdagster-dg-cli
are now under a separateai
extra, allowingdagster-dg-cli
to be installed without these dependencies.- Added
AutomationCondition.all_new_updates_have_run_tags
andAutomationCondition.any_new_update_has_run_tags
, which allows automation conditions to be filtered to partitions that have been materialized since the last tick from runs with certain tags. This condition can be used to require or prevent certain run tags from triggering downstream declarative automation conditions. These conditions are similar toAutomationCondition.executed_with_tags
, but look at all new runs since the most recent tick instead of just looking at the latest run.
Bugfixes
- Fixed a bug which would cause steps downstream of an asset with
skippable=True
and a blocking asset check to execute as long as the asset check output was produced, even if the asset output was skipped. - When a backfill fails, it will now cancel all of its in-progress runs before terminating.
- Fixed an issue that would cause trailing whitespace to be added to env vars using dot notation (
{{ env.FOO }}
) when listing the env vars used by a component. (Thanks, @edgarrmondragon!) - Fixed issue that would cause errors when using multi to single partition mappings with
DbIOManager
s. - [ui] Fixed issue with the "Report materialization" dialog for non-partitioned assets.
- [ui] Typing large YAML documents in the launchpad when default config is present is now more performant.
- [ui] Fixed an issue where setting a FloatMetadataValue to float('inf') or float('-inf') would cause an error when loading that metadata over graphql.
- [ui] The "Clear" button in the dimension partition text input for multi-partitioned assets now clears invalid selections as expected.
- [dagster-dbt] Fixed an issue with the
DbtCloudWorkspaceClient
that would cause errors when callingtrigger_job_run
with nosteps_override
parameter.
1.11.10 (core) / 0.27.10 (libraries)
New
- Added
inline-component
command to the publicly available scaffold commands in the Dagster CLI. - Added a new
require_upstream_step_success
config param to all executors. If{"step_dependency_config": {"require_upstream_step_success": False}}
is set, this will allow downstream steps to execute immediately after all required upstream outputs have finished, even if the upstream step has not completed in its entirety yet. This can be useful particularly in cases where there are large multi-assets with downstream assets that depend on only a subset of the assets in the upstream step. - The
logsForRun
resolvers andeventConnection
resolvers in the Dagster GraphQL API will now apply a default limit of 1000 to the number of logs returned from a single graphql query. Thecursor
field in the response can be used to continue iterating through the logs for a given run. - [dagster-airbyte]
@airbyte_assets
andAirbyteWorkspaceComponent
(previouslyAirbyteCloudWorkspaceComponent
) now support Airbyte OSS and Enterprise.
Bugfixes
- Fixed an issue where the
dagster_dg_cli
package failed to import when using Python 3.9. - Fixed an issue with
AutomationCondition.eager()
that could cause runs for materializable assets to be launched at the same time as an upstream observable source asset that had an automation condition, even if the upstream observation would not result in a new data version. - Fixed an issue which could, in some circumstances, cause errors during Declarative Automation evaluation after a dynamic partition was deleted.
- Fixed an issue that could cause confusing errors when attempting to supply
attributes
configuration toComponent
subclasses that did not inherit fromResolvable
. - Added a Matillion kind tag, thanks @RobBrownFreeAgent!
- [ui] Fixed an issue where the "Report materialization events" dialog for partitioned assets only worked if the partition was failed or missing.
- [ui] Fixed a browser crash which could occur in the global asset graph.
- [ui] Fixed a bug with the sensor preview behavior that would cause run requests contianing
run_key
s that had already been submitted to show up in the preview result. - [dagster-dbt] Fixed an issue that would cause the DbtCloudWorkspace to error before yielding asset events if the associated dbt Cloud run failed. Now, it will raise the error after all relevant asset events have been produced.
- [dagster-dbt] Added the
dbt-core
dependency back todagster-dbt
as it is still required for the dbt Cloud integration. If bothdbt-core
anddbt Fusion
are installed,dagster-dbt
will still prefer usingdbt Fusion
by default.
Documentation
- Introduced a new "Post-processing components" guide.
- Fixed incorrect YAML code snippets for alert policies docs page.
- Fixed incorrect chart keys in Helm documentation. Thanks, @charlottevdscheun!
- Fixed incorrect owner tags in Components docs. Thanks, @aaronprice00!
Dagster Plus
- Improved the Dagster+ agent's retry behavior during when it experiences outbound connection timeouts while a code location is being deployed.
1.11.9 (core) / 0.27.9 (libraries)
New
- Subclasses of
Resolved
now support fields of typedict[str, T]
. - [ui] Added a new 'arrow' icon to the set of supported kind tags (thanks @aleewen!)
Bugfixes
- Launching a backfill of a non-subsettable multi-asset without including every asset will now raise a clear error at backfill submission time, instead of failing with a confusing error after the backfill has started.
- Fixed an issue where passing in an empty list to the
assetKeys
argument of theassetsOrError
field in the GraphQL API would return every asset instead of an empty list of assets. - [dagster-dbt] Fixed an issue that would cause the DbtCloudWorkspace to error before yielding asset events if the associated DBT Cloud run failed. Now, it will raise the error after all relevant asset events have been produced.
Dagster Plus
- Serverless pex builds now support pyproject.toml-based packages.
1.11.8 (core) / 0.27.8 (libraries)
New
- A param
exclusions
was added to time window partition definitions to support custom calendars. - The
dagster
library now supportsprotobuf==6.x
- [dg]
dg scaffold defs --help
now shows descriptions for subcommands. - [dg] A new
dg check toml
command has been added to validate your TOML configuration files. - [dagster-databricks] The
DatabricksAssetBundleComponent
has been added in preview. Databricks tasks can now be represented as assets and submitted via Dagster. - [dagster-dbt] The DbtProjectComponent now takes an optional
cli_args
configuration to allow customizing the command that is run when your assets are executed. - [dagster-dbt] The polling interval and timeout used for runs triggered with the
DbtCloudWorkspace
resource can now be customized with theDAGSTER_DBT_CLOUD_POLL_INTERVAL
andDAGSTER_DBT_CLOUD_POLL_TIMEOUT
environment variables. - [ui] Added the ability to filter to failed/missing partitions in the asset report events dialog.
- [ui] A tree view has been added in the Global Asset Lineage.
- [telemetry] Telemetry disclaimer now prints to stderr.
Bugfixes
- Fixed an issue that would require config provided to backfills to contain config for all assets in the code location rather than just the selected ones.
1.11.7 (core) / 0.27.7 (libraries)
New
dg
will now report multiple detected errors in a configuration file instead of failing on the first detected error.- It is now possible to supply run config when launching an asset backfill.
- Updated the root URL to display the Overview/Timeline view for locations with schedules/automations, but no jobs (thanks @dschafer!)
- Added
tzdata
as a dependency todagster
, to ensure that declaring timezones likeUS/Central
work in all environments. - [dagster-dg-cli] Updated scaffolded file names to handle consecutive upper case letters (ACMEDatabricksJobComponent → acme_databricks_job_component.py not a_c_m_e_databricks_job_component.py)
- [dagster-dg-cli] Validating
requirements.env
is now opt-in fordg check yaml
. - [dagster-dbt]
DAGSTER_DBT_CLOUD_POLL_INTERVAL
andDAGSTER_DBT_CLOUD_POLL_TIMEOUT
environment variables can now be used to configure the polling interval and timeout for fetching data from dbt Cloud.
Deprecations
- [components] Removed deprecated and non-functional
asset_post_processors
fields fromSlingReplicationCollectionComponent
andAirflowInstanceComponent
.
1.11.6 (core) / 0.27.6 (libraries)
New
- Allow explicit git
platform
selection inlink_code_references_to_git
, thanks @chazmo03!
Bugfixes
- Fixed issue causing
AutomationCondition.replace
to not update built-in sub-conditions that did not have an explicit label, thanks @dschafer! - Fixed an issue where assets were considered stubs if they were a stub in any code location.
- Projects using components no longer cause "job definitions changed while uploading" errors on older agent versions.
- [dagster-dbt] Fixed a bug that could cause execution to fail if
enable_code_references
was set toTrue
on theDagsterDbtTranslatorSettings
.
Documentation
- Updated documentation of
dagster.yaml
to include thenux
option, thanks @dwisdom0!
Dagster Plus
- Fix "Create a support ticket" dialog submissions.
1.11.5 (core) / 0.27.5 (libraries)
New
- Static functions on classes decorated with
@template_var
can now optionally accept aComponentLoadContext
argument. - [dg] A MCP server is available to expose
dg
CLI capabilities to MCP clients. See thedg mcp
CLI group for details. - [dagster-dbt] The
dagster-dbt
package no longer has a dependency ondbt-core
. - [dagster-dbt][preview] Users of the dbt Fusion CLI can now use the
dagster-dbt
package to run dbt commands with no changes to their existing dagster code. This support is still in preview as the format of the log messages produced by the dbt Fusion CLI is still subject to change. Let us know if you notice any incompatibilities. - [dagster-databricks] Added a
PipesDatabricksServerlessClient
to support Databricks Serverless jobs with Dagster pipes. - [dagster-databricks] Added additional options for cluster configuration (thanks @jmccartin!)
Bugfixes
- Various bugfixes for backfills that target assets which change their partitions definition mid-backfill.
- [ui] Fixed issue that could cause errors related to the
ObjectMetadataValue
class.
Documentation
- Added docs for using Spark Connect and Databricks Connect with Dagster.