Skip to content

Commit 4dfccdb

Browse files
feat(cron_schedule): opt-in time-partitioned-job support (v0.8.2)
ScheduleDefinition errors on every tick against a partitioned asset job with "no partition key provided". Add an opt-in path that switches to build_schedule_from_partitioned_job when any time-based partition field is set, so each tick auto-targets the most-recent finished partition. - New fields (mirroring the synthetic_data_generator + sinks shape): partition_type, partition_start, partition_values, partition_dimensions, dynamic_partition_name. - cron_expression becomes Optional in the partitioned path — Dagster infers cadence from partitions_def. If supplied, validated against the partitions_def cadence (raises ValueError on mismatch instead of silently misfiring). - partition_type='static' / 'dynamic' raise ValueError in the partitioned path — build_schedule_from_partitioned_job is time-only. No change to the un-partitioned path: when neither partition_type nor partition_dimensions is set, behavior is byte-for-byte identical. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent a048624 commit 4dfccdb

3 files changed

Lines changed: 334 additions & 19 deletions

File tree

pyproject.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
44

55
[project]
66
name = "dagster-community-components"
7-
version = "0.8.1"
7+
version = "0.8.2"
88
description = "Community-maintained Dagster components — ingestion, transforms, IO managers, sensors, sinks, resources, and more."
99
readme = "README.md"
1010
requires-python = ">=3.10"

schedules/cron_schedule/README.md

Lines changed: 41 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,6 @@ Defines a `ScheduleDefinition` + an underlying asset job that materializes the c
1111
| Field | Type | Description |
1212
|---|---|---|
1313
| `schedule_name` | `str` | Unique schedule name shown in the UI. |
14-
| `cron_expression` | `str` | Cron expression (5 fields: m h dom mon dow). E.g. '0 9 * * *' = daily 09:00. |
1514
| `asset_keys` | `List[str]` | Slash-separated asset keys to materialize on each tick. |
1615

1716
### Catalog metadata
@@ -20,6 +19,15 @@ Defines a `ScheduleDefinition` + an underlying asset job that materializes the c
2019
|---|---|---|---|
2120
| `tags` | `Dict[str, str]` || Tags applied to runs created by this schedule. |
2221

22+
### Partitions
23+
24+
| Field | Type | Default | Description |
25+
|---|---|---|---|
26+
| `partition_type` | `str` || Partition type: 'daily', 'weekly', 'monthly', 'hourly', 'static', 'multi', 'dynamic', or None for unpartitioned. Note: only time-based types ('daily'/'weekly'/'monthly'/'hourly') work with this schedule's partitioned-job path — 'static' / 'dynamic' raise ValueError. |
27+
| `partition_start` | `str` || Partition start in ISO format (e.g. '2024-01-01' or '2024-01-01-00:00' for hourly). Required for time-based types. |
28+
| `partition_values` | `str` || Comma-separated values for static or multi partitioning. Only used if you also set partition_type='multi' (and even then the schedule rejects pure static — see partition_type). |
29+
| `partition_dimensions` | `List[Dict[str, Any]]` || Multi-axis partition spec: list of {name, type, start, values, dynamic_partition_name} dicts. Overrides flat fields when set. |
30+
2331
### Sensor configuration
2432

2533
| Field | Type | Default | Description |
@@ -31,7 +39,9 @@ Defines a `ScheduleDefinition` + an underlying asset job that materializes the c
3139

3240
| Field | Type | Default | Description |
3341
|---|---|---|---|
42+
| `cron_expression` | `str` || Cron expression (5 fields: m h dom mon dow). E.g. '0 9 * * *' = daily 09:00. REQUIRED when no partition fields are set; OPTIONAL in the partitioned-job path (cadence is inferred from the partitions_def). If supplied in the partitioned path, validated against the partitions_def cadence. |
3443
| `execution_timezone` | `str` | `"UTC"` | IANA timezone for cron evaluation, e.g. 'America/Los_Angeles'. |
44+
| `dynamic_partition_name` | `str` || Name for DynamicPartitionsDefinition (when partition_type='dynamic'). NOTE: the schedule rejects dynamic-only partitions. |
3545

3646
<!-- FIELDS:END -->
3747

@@ -47,3 +57,33 @@ attributes:
4757
tags:
4858
team: data-platform
4959
```
60+
61+
### Partitioned-job mode
62+
63+
When any time-based partition field (`partition_type`,
64+
`partition_dimensions`) is set, the component switches from
65+
`ScheduleDefinition` to `build_schedule_from_partitioned_job`. Each
66+
tick auto-targets the most-recent finished partition — without this
67+
switch, `ScheduleDefinition` against a partitioned asset job errors
68+
with "no partition key provided" on every tick.
69+
70+
`cron_expression` is OPTIONAL in this mode (Dagster infers cadence
71+
from the partitions_def). If you supply one, it's validated against
72+
the partitions_def cadence — a daily cron paired with hourly
73+
partitions raises `ValueError` at load time rather than failing
74+
silently.
75+
76+
```yaml
77+
type: dagster_component_templates.CronScheduleComponent
78+
attributes:
79+
schedule_name: hourly_orders_pipeline
80+
asset_keys: [python_hourly_orders_for_pipe, orders_to_s3]
81+
partition_type: hourly
82+
partition_start: "2026-05-26-00:00"
83+
default_status: RUNNING
84+
```
85+
86+
`partition_type: static` and `partition_type: dynamic` raise
87+
`ValueError` in this path — `build_schedule_from_partitioned_job`
88+
needs a time-based partitions_def. For non-time partitions, use the
89+
un-partitioned schedule and select keys downstream.

0 commit comments

Comments
 (0)