Skip to content

[cron] replace croniter with something that doesn't suck#33952

Open
sbquinlan wants to merge 3 commits into
dagster-io:masterfrom
sbquinlan:sbquinlan/cron
Open

[cron] replace croniter with something that doesn't suck#33952
sbquinlan wants to merge 3 commits into
dagster-io:masterfrom
sbquinlan:sbquinlan/cron

Conversation

@sbquinlan

Copy link
Copy Markdown
Contributor

Note: currently the PRs to add croniter compatibility to the cron crate are still landing. This pr adds a dependency to a specific branch cron = { git = "https://github.com/sbquinlan/cron.git", branch = "codex/ci-supply-chain-hygiene" }. I'm open to vendoring the entirety of the cron crate here or any other alternative. Unfortunately review is slow in the cron crate.

Summary & Motivation

Replace Dagster's vendored croniter usage and custom Python cron iteration fast paths with a small Rust-backed package, dagster-cron. dagster-cron offers 20x to 100x performance improvements over croniter's implementation.

Dagster's public scheduling API remains unchanged: callers still use is_valid_cron_string, cron_string_iterator, reverse_cron_string_iterator, schedule_execution_time_iterator, and the existing time partition APIs. Internally, those functions now delegate to dagster_cron as quickly as possible instead of routing through croniter or Dagster's custom Python _find_* iteration helpers.

Changes

  • Add python_modules/libraries/dagster-cron, a maturin/PyO3 package exposing a thin native surface: Schedule, ScheduleIterator, CronStringIterator, includes, forward/reverse iteration, and the small set of schedule configuration enums Dagster needs.
  • Replace dagster._utils.schedules cron iteration with calls into dagster_cron.
  • Remove vendored croniter from dagster._vendored.
  • Keep has_out_of_range_cron_interval as raw Python field validation because it does not need cron iteration.
  • Add a wheel-building workflow for dagster-cron across Linux, macOS, and Windows.
  • Add native-package smoke tests, Dagster compatibility tests, and copied croniter compatibility tests with a test-only shim.
  • Add dagster-cron to Dagster dependency/dev-install/lockfile wiring.

Compatibility

No breaking changes are intended for Dagster's public scheduling APIs or time partition APIs.

Supported through the Dagster runtime path:

  • Five-field cron expressions.
  • Aliases @hourly, @daily, @weekly, @monthly, @yearly.
  • Croniter-compatible aliases @annually and @midnight, normalized in Rust.
  • Sunday as either 0 or 7.
  • Named months and weekdays.
  • ? in day-of-month and day-of-week fields.
  • L, nearest weekday W, and nth weekday #.
  • Random R fields.
  • Wraparound ranges supported by the active cron crate branch.
  • Forward and reverse iteration in named timezones.
  • DST nonexistent-time handling using Dagster's existing next-existent behavior.
  • Hourly DST fold behavior and non-hourly ambiguous-time behavior covered by existing scheduler tests.
  • Time-window partition, partition mapping, partitioned schedule, and freshness-check paths continue through the public Dagster interfaces and now use the Rust-backed iterator.

Intentional or known unsupported croniter surfaces:

  • Hashed H expressions. (not relevant in dagster)
  • @reboot. (also not relevant)
  • expand_from_start_time. (flag not used by dagster)
  • implement_cron_bug. (flag not used by dagster)
  • croniter private expanded-state/helper APIs.
  • croniter's strict default 1970-2099 year-range reachability checks. (cron doesn't have this default, uses a 400 year max iteration with no matches)
  • croniter-specific behavior around some unions of special specifiers.

Validation notes:

  • Dagster still catches invalid native construction and maps it to the existing boolean validation path where appropriate.
  • Impossible day/month combinations such as February 30/31 are rejected by the Rust schedule construction path.
  • Some validation may be slightly more permissive than croniter where the expression is otherwise supported by the cron crate. The compatibility suite documents those differences explicitly.

Test Plan

  • cargo test --locked
  • cargo fmt --check
  • uv lock --check in python_modules/libraries/dagster-cron
  • uv lock --check in python_modules/dagster
  • uv run --group test --reinstall-package dagster-cron pytest -q
  • uv run --group test python tests/test_croniter_compatibility.py --skip-known-unsupported -q
  • croniter compatibility suite: 140 passed, 75 skipped

Changelog

Dagster cron schedule validation and iteration now use the Rust-backed dagster-cron package instead of vendored croniter and Dagster's previous custom Python cron iteration implementation. Public scheduling APIs are unchanged.

@greptile-apps

greptile-apps Bot commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR replaces Dagster's vendored croniter Python library and ~600 lines of custom cron-iteration fast-path code with dagster-cron, a new Rust/PyO3 package that wraps the cron crate and exposes a thin native surface to Python. Public scheduling APIs (is_valid_cron_string, cron_string_iterator, reverse_cron_string_iterator, schedule_execution_time_iterator) are unchanged.

  • New dagster-cron package (src/lib.rs): implements Schedule, ScheduleIterator, CronStringIterator, DST-aware forward/reverse iteration, alias normalisation (@annually, @midnight), and is_pre_transition_ambiguous skipping for non-hourly schedules.
  • schedules.py rewrite: all cron iteration now delegates to _native_cron_string_iterator; adds batched_cron_string_iterator and cron_string_iterator_batch for callers that benefit from bulk retrieval.
  • Call-site updates: auto_materialize_rule_impls.py adopts the batched API; time_window.py replaces next(cron_string_iterator(...)) == ts with the new cron_string_includes function.

Confidence Score: 5/5

Safe to merge; public scheduling APIs are unchanged and the PR ships a compatibility test suite alongside the new native implementation.

The change is well-scoped, replacing an internal implementation while keeping public APIs identical. The croniter compatibility suite, Dagster integration tests, and DST-handling logic in Rust all provide good coverage. The only findings are minor: a dead guard that can never fire and an unpinned CI dependency.

python_modules/libraries/dagster-cron/Cargo.toml — the git-branch dependency on a personal fork should be replaced with a versioned crates.io release once the upstream cron crate merges the required features.

Important Files Changed

Filename Overview
python_modules/libraries/dagster-cron/src/lib.rs New Rust/PyO3 core: exposes Schedule, ScheduleIterator, CronStringIterator, and top-level helper functions. Contains one dead guard in new_dagster_schedule (always-false condition) but the real validation occurs in normalize_dagster_expression, so runtime behaviour is unaffected.
python_modules/dagster/dagster/_utils/schedules.py Major simplification: removes ~600 lines of Python cron iteration logic and delegates to dagster_cron native functions. Adds batched_cron_string_iterator and cron_string_iterator_batch helpers, and keeps has_out_of_range_cron_interval as pure Python field validation.
python_modules/libraries/dagster-cron/Cargo.toml Declares the maturin/PyO3 package with a git-branch dependency on the author's personal fork of the cron crate; Cargo.lock pins a specific commit so --locked builds are reproducible, but the branch reference is a temporary supply-chain risk.
.github/workflows/build-dagster-cron-wheels.yml New wheel-build workflow for Linux, macOS-14 (arm64), and Windows. Unpinned maturin install could cause non-reproducible builds; all other toolchain steps use version-tagged actions.
python_modules/dagster/dagster/_core/definitions/auto_materialize_rule_impls.py Switches missed_cron_ticks from unbatched cron_string_iterator to batched_cron_string_iterator; semantics are identical (early-return on first tick past evaluation_time).
python_modules/dagster/dagster/_core/definitions/partitions/definition/time_window.py Replaces next(cron_string_iterator(...)) == window_start with cron_string_includes(window_start.timestamp(), ...) for exclusion checks — semantically equivalent and more efficient.
python_modules/libraries/dagster-cron/python/dagster_cron/init.pyi Comprehensive stub file covering all exported types and functions with correct signatures.

Reviews (2): Last reviewed commit: "Add native cron batch helpers" | Re-trigger Greptile

[dependencies]
chrono = { version = "~0.4", default-features = false, features = ["clock"] }
chrono-tz = "0.10"
cron = { git = "https://github.com/sbquinlan/cron.git", branch = "codex/ci-supply-chain-hygiene" }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Mutable git branch dependency on a personal fork

cron is pinned to a branch (codex/ci-supply-chain-hygiene) on the author's personal fork rather than a released crates.io version. The PR description acknowledges this is temporary while upstream PRs land. The Cargo.lock pins commit d74e27a5437baab1dee83d9a9238bcd4dcdd9472, so --locked builds are reproducible today, but cargo update would silently follow the branch tip. A force-push, branch deletion, or a supply-chain compromise of the fork would affect any rebuild that does not use the lock file. Once the upstream cron crate merges and releases the required features, this should be replaced with a versioned crates.io dependency.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Comment thread python_modules/libraries/dagster-cron/src/lib.rs
Comment thread python_modules/libraries/dagster-cron/src/lib.rs
@sbquinlan

Copy link
Copy Markdown
Contributor Author

@gibsondan can you unblock CI for me please?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant