You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: CHANGES.md
+66-17Lines changed: 66 additions & 17 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -5,11 +5,12 @@
5
5
**Breaking Changes**
6
6
7
7
There are a substantial number of breaking changes in the 0.7.0 release.
8
-
Please see 070_MIGRATION.md for instructions regarding migrating old code.
8
+
Please see `070_MIGRATION.md` for instructions regarding migrating old code.
9
9
10
10
**_Scheduler_**
11
11
12
-
- The scheduler configuration has been moved from the `@schedules` decorator to `DagsterInstance`. Existing schedules that have been running are no longer compatible with current storage. To migrate,
12
+
- The scheduler configuration has been moved from the `@schedules` decorator to `DagsterInstance`. Existing schedules
13
+
that have been running are no longer compatible with current storage. To migrate,
13
14
remove the `scheduler` argument on all `@schedules` decorators:
14
15
15
16
instead of:
@@ -38,13 +39,20 @@ Please see 070_MIGRATION.md for instructions regarding migrating old code.
38
39
39
40
Finally, if you had any existing schedules running, delete the existing `$DAGSTER_HOME/schedules` directory and run `dagster schedule wipe && dagster schedule up` to re-instatiate schedules in a valid state.
40
41
41
-
- The `should_execute` and `environment_dict_fn` argument to `ScheduleDefinition` now have a required first argument `context`, representing the `ScheduleExecutionContext`
42
+
- The `should_execute` and `environment_dict_fn` argument to `ScheduleDefinition` now have a required first argument
43
+
`context`, representing the `ScheduleExecutionContext`
42
44
43
45
**_Config System Changes_**
44
46
45
-
- In the config system, `Dict` has been renamed to `Shape`; `List` to `Array`; `Optional` to `Noneable`; and `PermissiveDict` to `Permissive`. The motivation here is to clearly delineate config use cases versus cases where you are using types as the inputs and outputs of solids as well as python typing types (for mypy and friends). We believe this will be clearer to users in addition to simplifying our own implementation and internal abstractions.
47
+
- In the config system, `Dict` has been renamed to `Shape`; `List` to `Array`; `Optional` to `Noneable`; and
48
+
`PermissiveDict` to `Permissive`. The motivation here is to clearly delineate config use cases versus cases where you
49
+
are using types as the inputs and outputs of solids as well as python typing types (for mypy and friends). We believe
50
+
this will be clearer to users in addition to simplifying our own implementation and internal abstractions.
46
51
47
-
Our recommended fix is _not_ to used Shape and Array, but instead to use our new condensed config specification API. This allow one to use bare dictionaries instead of `Shape`, lists with one member instead of `Array`, bare types instead of `Field` with a single argument, and python primitive types (`int`, `bool` etc) instead of the dagster equivalents. These result in dramatically less verbose config specs in most cases.
52
+
Our recommended fix is _not_ to used Shape and Array, but instead to use our new condensed config specification API.
53
+
This allow one to use bare dictionaries instead of `Shape`, lists with one member instead of `Array`, bare types
54
+
instead of `Field` with a single argument, and python primitive types (`int`, `bool` etc) instead of the dagster
55
+
equivalents. These result in dramatically less verbose config specs in most cases.
48
56
49
57
So instead of
50
58
@@ -65,8 +73,11 @@ Please see 070_MIGRATION.md for instructions regarding migrating old code.
65
73
66
74
No imports and much simpler, cleaner syntax.
67
75
68
-
-`config_field` is no longer a valid argument on `solid`, `SolidDefinition`, `ExecutorDefintion`, `executor`, `LoggerDefinition`, `logger`, `ResourceDefinition`, `resource`, `system_storage`, and `SystemStorageDefinition`. Use `config` instead.
69
-
- For composite solids, the `config_fn` no longer takes a `ConfigMappingContext`, and the context has been deleted. To upgrade, remove the first argument to `config_fn`.
76
+
-`config_field` is no longer a valid argument on `solid`, `SolidDefinition`, `ExecutorDefintion`, `executor`,
77
+
`LoggerDefinition`, `logger`, `ResourceDefinition`, `resource`, `system_storage`, and `SystemStorageDefinition`. Use
78
+
`config` instead.
79
+
- For composite solids, the `config_fn` no longer takes a `ConfigMappingContext`, and the context has been deleted. To
80
+
upgrade, remove the first argument to `config_fn`.
70
81
71
82
So instead of
72
83
@@ -97,34 +108,44 @@ Please see 070_MIGRATION.md for instructions regarding migrating old code.
97
108
**_Dagster Type System Changes_**
98
109
99
110
-`dagster.Set` and `dagster.Tuple` can no longer be used within the config system.
100
-
- Dagster types are now instances of `DagsterType`, rather than a class than inherits from `RuntimeType`. Instead of dynamically generating a class to create a custom runtime type, just create an instance of a `DagsterType`. The type checking function is now an argument to the `DagsterType`, rather than an abstract method that has to be implemented in subclass.
111
+
- Dagster types are now instances of `DagsterType`, rather than a class than inherits from `RuntimeType`. Instead of
112
+
dynamically generating a class to create a custom runtime type, just create an instance of a `DagsterType`. The type
113
+
checking function is now an argument to the `DagsterType`, rather than an abstract method that has to be implemented in
114
+
a subclass.
101
115
-`RuntimeType` has been renamed to `DagsterType` is now an encouraged API for type creation.
102
116
- Core type check function of DagsterType can now return a naked `bool` in addition
103
117
to a `TypeCheck` object.
104
-
-`type_check_fn` on `DagsterType` (formerly `type_check` and `RuntimeType`, respectively) now takes a first argument `context` of type `TypeCheckContext` in addition to the second argument of value.
118
+
-`type_check_fn` on `DagsterType` (formerly `type_check` and `RuntimeType`, respectively) now takes a first argument
119
+
`context` of type `TypeCheckContext` in addition to the second argument of `value`.
105
120
-`define_python_dagster_type` has been eliminated in favor of `PythonObjectDagsterType` .
106
121
-`dagster_type` has been renamed to `usable_as_dagster_type`.
107
122
-`as_dagster_type` has been removed and similar capabilities added as `make_python_type_usable_as_dagster_type`.
108
123
-`PythonObjectDagsterType` and `usable_as_dagster_type` no longer take a `type_check` argument. If
109
124
a custom type_check is needed, use `DagsterType`.
125
+
- As a consequence of these changes, if you were previously using `dagster_pyspark` or `dagster_pandas` and expecting
126
+
Pyspark or Pandas types to work as Dagster types, e.g., in type annotations to functions decorated with `@solid` to
127
+
indicate that they are input or output types for a solid, you will need to call
128
+
`make_python_type_usable_as_dagster_type` from your code in order to map the Python types to the Dagster types, or just
129
+
use the Dagster types (`dagster_pandas.DataFrame` instead of `pandas.DataFrame`) directly.
110
130
111
131
**_Other_**
112
132
113
133
- We no longer publish base Docker images. Please see the updated deployment docs for an example
114
134
Dockerfile off of which you can work.
115
-
116
135
-`step_metadata_fn` has been removed from `SolidDefinition` & `@solid`.
117
136
-`SolidDefinition` & `@solid` now takes `tags` and enforces that values are strings or
118
-
are safetly encoded as JSON. `metadata` is deprecated and will be removed in a future version.
137
+
are safely encoded as JSON. `metadata` is deprecated and will be removed in a future version.
119
138
-`resource_mapper_fn` has been removed from `SolidInvocation`.
120
139
121
140
**New**
122
141
142
+
- Dagit now includes a much richer execution view, with a Gantt-style visualization of step execution and a live timeline.
123
143
- Early support for Python 3.8 is now available, and Dagster/Dagit along with many of our libraries
124
144
are now tested against 3.8. Note that several of our upstream dependencies have yet to publish
125
145
wheels for 3.8 on all platforms, so running on Python 3.8 likely still involves building some
126
146
dependencies from source.
127
-
-`dagster/priority` tags can now be used to prioritize the order of execution for the built in in process and multiprocess engines.
147
+
-`dagster/priority` tags can now be used to prioritize the order of execution for the built-in in-process and
148
+
multiprocess engines.
128
149
-`dagster-postgres` storages can now be configured with separate arguments and environment variables, such as:
129
150
130
151
```
@@ -140,9 +161,11 @@ Please see 070_MIGRATION.md for instructions regarding migrating old code.
140
161
db_name: test
141
162
```
142
163
143
-
- Support for `RunLauncher`s on `DagsterInstance` allows for execution to be "launched" outside of the Dagit/Dagster process.
144
-
As one example, this is used by `dagster-k8s` to submit pipeline execution as a kubernetes batch job.
164
+
- Support for `RunLauncher`s on `DagsterInstance` allows for execution to be "launched" outside of the Dagit/Dagster
165
+
process. As one example, this is used by `dagster-k8s` to submit pipeline execution as a Kubernetes Job.
145
166
- Added support for adding tags to runs initiated from the `Playground` view in dagit.
167
+
- Added `@monthly_schedule` decorator.
168
+
- Added `Enum.from_python_enum` helper to wrap Python enums for config. (Thanks @kdungs!)
146
169
-**[dagster-bash]** The Dagster bash solid factory now passes along `kwargs` to the underlying
147
170
solid construction, and now has a single `Nothing` input by default to make it easier to create a
148
171
sequencing dependency. Also, logs are now buffered by default to make execution less noisy.
@@ -151,14 +174,40 @@ Please see 070_MIGRATION.md for instructions regarding migrating old code.
151
174
clusters, submitting jobs, and waiting for jobs/logs. We also now provide a
152
175
`emr_pyspark_resource`, which together with the new `@pyspark_solid` decorator makes moving
153
176
pyspark execution from your laptop to EMR as simple as changing modes.
154
-
-**[dagster-gcp]** GCS is now supported for system storage.
177
+
**[dagster-pandas]** Added `create_dagster_pandas_dataframe_type`, `PandasColumn`, and `Constraint` API's in order for
178
+
users to create custom types which perform column validation, dataframe validation, summary statistics emission, and
179
+
dataframe serialization/deserialization.
180
+
-**[dagster-gcp]** GCS is now supported for system storage, as well as being supported with the Dask executor. (Thanks
181
+
@habibutsu!) Bigquery solids have also been updated to support the new API.
155
182
156
183
**Bugfix**
157
184
158
185
- Ensured that all implementations of `RunStorage` clean up pipeline run tags when a run
159
186
is deleted. Requires a storage migration, using `dagster instance migrate`.
160
-
- The multiprocess engine now handles solid subsets correctly.
161
-
- The multiprocess engine will now correctly emit skip events for steps downstream of failures and other skips.
187
+
- The multiprocess and Celery engines now handle solid subsets correctly.
188
+
- The multiprocess and Celery engines will now correctly emit skip events for steps downstream of failures and other
189
+
skips.
190
+
- The `@solid` and `@lambda_solid` decorators now correctly wrap their decorated functions, in the sense of
191
+
`functools.wraps`.
192
+
- Performance improvements in Dagit when working with runs with large configurations.
193
+
- The Helm chart in `dagster_k8s` has been hardened against various failure modes and is now compatible with Helm 2.
194
+
- SQLite run and event log storages are more robust to concurrent use.
195
+
- Improvements to error messages and to handling of user code errors in input hydration and output materialization logic.
196
+
- Fixed an issue where the Airflow scheduler could hang when attempting to load dagster-airflow pipelines.
197
+
- We now handle our SQLAlchemy connections in a more canonical way (thanks @zzztimbo!).
198
+
- Fixed an issue using S3 system storage with certain custom serialization strategies.
199
+
- Fixed an issue leaking orphan processes from compute logging.
200
+
- Fixed an issue leaking semaphores from Dagit.
201
+
- Setting the `raise_error` flag in `execute_pipeline` now actually raises user exceptions instead of a wrapper type.
202
+
203
+
**Documentation**
204
+
205
+
- Our docs have been reorganized and expanded (thanks @habibutsu, @vatervonacht, @zzztimbo). We'd love feedback and
206
+
contributions!
207
+
208
+
**Thank you**
209
+
Thank you to all of the community contributors to this release!! In alphabetical order: @habibutsu, @kdungs,
0 commit comments