Skip to content

Commit ae854dc

Browse files
OwenKephartDagster Devtools
authored andcommitted
Add event timestamps to automation conditions for precise trigger/reset resolution (#21494)
When both trigger and reset conditions fire on the same evaluation tick, timestamp comparison now determines which event happened more recently (per-partition). This eliminates the special-case hack for CronTickPassed + NewlyUpdated and enables correct behavior for all condition combinations. ## Key Changes - Add `TimingMetadata` class to map event timestamps to entity subsets, stored in-memory on `AutomationResult` - Create `TimedSubsetAutomationCondition` base class for conditions that support per-partition timestamps - Enrich `CronTickPassedCondition` and `NewlyUpdatedCondition` to return timing metadata (cron tick time and materialization timestamps respectively) - Add `compute_subsets_by_latest_materialization_timestamp()` to asset graph view for efficient per-partition timestamp lookup with configurable threshold fallback - Replace SinceCondition's hardcoded `CronTickPassed + NewlyUpdated` special case with generic timestamp-based resolution via `subset_with_later_timestamps_than()` <details> <summary>Files Changed</summary> ### Added (2 files) - `dagster/_core/asset_graph_view/timing_metadata.py` - TimingMetadata class for timestamp → EntitySubset mapping - `dagster_tests/declarative_automation_tests/automation_condition_tests/builtins/test_event_timestamps.py` - Comprehensive timestamp resolution tests ### Modified (5 files) - `dagster/_core/definitions/declarative_automation/automation_condition.py` - Add timing_metadata field to AutomationResult - `dagster/_core/definitions/declarative_automation/operands/subset_automation_condition.py` - New TimedSubsetAutomationCondition base class - `dagster/_core/definitions/declarative_automation/operands/operands.py` - CronTickPassed and NewlyUpdatedCondition now return TimingMetadata - `dagster/_core/asset_graph_view/asset_graph_view.py` - Add compute_subsets_by_latest_materialization_timestamp() with threshold-based fallback - `dagster_tests/declarative_automation_tests/automation_condition_tests/fundamentals/test_result_value_hash.py` - Adjust timing in tests </details> ## Critical Notes - Timestamps are in-memory only (not serialized); consumed same-tick by parent SinceCondition - Per-partition timestamp fetching respects configurable threshold (env var `DAGSTER_MAX_PARTITIONS_FOR_DA_TIMESTAMP_FETCH`, default 100); exceeding it falls back to tick timestamp - Fully backward compatible: conditions without timing metadata fall back to current behavior (reset wins when both fire simultaneously) <details> <summary>original-plan</summary> # Plan: Add Event Timestamps to Automation Condition Results ## Context The `SinceCondition` resolves trigger/reset at tick granularity (30s–few minutes). When both trigger and reset fire on the same tick, ordering is ambiguous — there's even a special-case hack (lines 136-170 in `since_operator.py`) for the `NewlyUpdated` + `CronTickPassed` combination. With real event timestamps, SinceCondition could make **per-partition, timestamp-precise** decisions about whether the trigger or reset happened more recently, eliminating the hack and enabling correct behavior for all condition combinations. ### Current problem illustrated 1. Cron tick fires at 12:00:00 2. Asset partition materializes at 12:00:05 3. DA tick runs at 12:00:30 — sees both CronTickPassed and NewlyUpdated as true 4. Without real timestamps: ambiguous ordering, resolved by a hardcoded special case 5. With real timestamps: CronTickPassed has event_timestamp=12:00:00, NewlyUpdated has event_timestamp=12:00:05 → trigger happened after reset → partition stays in true_subset ## Approach: `subsets_with_metadata` + `SubsetResult` return type Use the existing `subsets_with_metadata` field (already on `AutomationResult`, `AutomationConditionNodeCursor`, `AutomationConditionEvaluation`, and wired through GraphQL) to carry event timestamps. Conditions opt in by returning a `SubsetResult` from `compute_subset()`. Always-on for conditions that support it. Exact per-partition timestamps (not bucketed). --- ## Implementation Steps ### Step 1: Define `SubsetResult` in `SubsetAutomationCondition` **File**: `dagster-oss/python_modules/dagster/dagster/_core/definitions/declarative_automation/operands/subset_automation_condition.py` Add a result type that wraps a subset with optional metadata: ```python @record class SubsetResult: subset: EntitySubset subsets_with_metadata: Sequence[AssetSubsetWithMetadata] = () ``` Modify `SubsetAutomationCondition.evaluate()` to detect `SubsetResult` vs plain `EntitySubset` from `compute_subset()`, and pass `subsets_with_metadata` through to `AutomationResult`: ```python async def evaluate(self, context): if context.candidate_subset.is_empty: true_subset = context.get_empty_subset() subsets_with_metadata = [] elif inspect.iscoroutinefunction(self.compute_subset): result = await self.compute_subset(context) else: result = self.compute_subset(context) if isinstance(result, SubsetResult): true_subset = result.subset subsets_with_metadata = result.subsets_with_metadata else: true_subset = result subsets_with_metadata = [] return AutomationResult(context, true_subset, subsets_with_metadata=subsets_with_metadata) ``` Note: `SubsetAutomationCondition.requires_cursor` is currently `False`. Since `subsets_with_metadata` is stored via `node_cursor`, and `node_cursor` returns `None` when `requires_cursor` is `False`, we need to either: - Override `requires_cursor` to return `True` when metadata is present, or - Store the metadata on `AutomationConditionEvaluation` only (not on cursor) since it's for same-tick consumption by the parent SinceCondition **Decision needed**: Since the timestamps are consumed by the parent SinceCondition on the *same* tick (not across ticks), we may not need cursor persistence. The `subsets_with_metadata` on the child's `AutomationResult` is directly accessible to SinceCondition via `trigger_result._subsets_with_metadata`. This avoids needing to enable cursoring on leaf conditions. ### Step 2: Enrich `CronTickPassedCondition` **File**: `dagster-oss/python_modules/dagster/dagster/_core/definitions/declarative_automation/operands/operands.py` When the cron tick has passed, return a `SubsetResult` with the exact cron tick timestamp: ```python def compute_subset(self, context): previous_cron_tick = self._get_previous_cron_tick(context.evaluation_time) if context.previous_evaluation_time is None or previous_cron_tick < context.previous_evaluation_time: return context.get_empty_subset() else: return SubsetResult( subset=context.candidate_subset, subsets_with_metadata=[AssetSubsetWithMetadata( subset=context.candidate_subset.convert_to_serializable_subset(), metadata={"event_timestamp": FloatMetadataValue(previous_cron_tick.timestamp())}, )], ) ``` Single entry — the cron tick time is uniform across all partitions. ### Step 3: Enrich `NewlyUpdatedCondition` with per-partition timestamps This requires extending the DB query layer. **File**: `dagster-oss/python_modules/dagster/dagster/_utils/caching_instance_queryer.py` The existing `get_asset_partitions_updated_after_cursor()` calls `_get_latest_materialization_or_observation_storage_ids_by_asset_partition()` which returns `Mapping[AssetKeyPartitionKey, int]` (partition → storage_id). We need to also get timestamps. Option A: Add a companion method `get_asset_partition_timestamps_updated_after_cursor()` that returns `Mapping[AssetKeyPartitionKey, float]`. Option B: Modify the return type to include both storage_id and timestamp. The underlying `instance.get_latest_storage_id_by_partition()` would need to also return timestamps. **Recommended**: Option A — less invasive. After getting the updated partition keys (which we already do), make a follow-up query to get event timestamps for those specific storage IDs. Since the set of newly updated partitions per tick is small, this is cheap. **File**: `dagster-oss/python_modules/dagster/dagster/_core/asset_graph_view/asset_graph_view.py` Add a new method `compute_updated_since_temporal_context_with_timestamps()` that returns both the subset and a mapping of partition keys to timestamps. The `NewlyUpdatedCondition` calls this instead. **File**: `dagster-oss/python_modules/dagster/dagster/_core/definitions/declarative_automation/operands/operands.py` Modify `NewlyUpdatedCondition.compute_subset()`: ```python async def compute_subset(self, context): if context.previous_temporal_context is None: return context.get_empty_subset() subset, partition_timestamps = await context.asset_graph_view.compute_updated_since_temporal_context_with_timestamps( key=context.key, temporal_context=context.previous_temporal_context ) # Group partitions by timestamp subsets_with_metadata = _group_by_timestamp(subset, partition_timestamps) return SubsetResult(subset=subset, subsets_with_metadata=subsets_with_metadata) ``` Helper `_group_by_timestamp` groups partition keys sharing the same timestamp into compressed subsets with metadata. ### Step 4: Use timestamps in `SinceCondition` for precise resolution **File**: `dagster-oss/python_modules/dagster/dagster/_core/definitions/declarative_automation/operators/since_operator.py` This is the core change. Replace the special-case ordering hack with timestamp-based per-partition resolution: ```python async def evaluate(self, context): # ... evaluate trigger and reset as before ... true_subset = context.previous_true_subset or context.get_empty_subset() # Extract per-partition timestamps from children trigger_timestamps = _extract_partition_timestamps(trigger_result) reset_timestamps = _extract_partition_timestamps(reset_result) # For partitions in trigger but not reset (or vice versa), simple union/difference trigger_only = trigger_result.true_subset.compute_difference(reset_result.true_subset) reset_only = reset_result.true_subset.compute_difference(trigger_result.true_subset) both = trigger_result.true_subset.compute_intersection(reset_result.true_subset) true_subset = true_subset.compute_union(trigger_only) true_subset = true_subset.compute_difference(reset_only) # For partitions in BOTH trigger and reset on same tick: # use timestamps to determine which happened more recently if not both.is_empty: if trigger_timestamps and reset_timestamps: # Per-partition comparison: keep if trigger timestamp > reset timestamp trigger_wins = _partitions_where_trigger_after_reset( both, trigger_timestamps, reset_timestamps ) true_subset = true_subset.compute_union(trigger_wins) # (partitions where reset wins are already excluded by not being unioned) else: # Fallback: no timestamps available, use current ordering behavior true_subset = true_subset.compute_union(both) # or difference, depending on default # ... update SinceConditionData, return result ... ``` The `_extract_partition_timestamps` helper reads `result._subsets_with_metadata` to build a `Mapping[str, float]` from partition key to `event_timestamp`. **Key benefit**: The special-case `reset_newly_true` hack for `CronTickPassed + NewlyUpdated` can be **removed** — timestamp comparison handles this generically for all condition combinations. ### Step 5: Propagate timestamps through SinceCondition result SinceCondition should also emit `subsets_with_metadata` on its own result, carrying the trigger timestamps for partitions in its `true_subset`. This allows parent conditions or the UI to see when each partition was triggered. --- ## Critical Files | File | Change | |------|--------| | `.../operands/subset_automation_condition.py` | Add `SubsetResult`, modify `evaluate()` | | `.../operands/operands.py` | Modify `CronTickPassedCondition` and `NewlyUpdatedCondition` | | `.../operators/since_operator.py` | Timestamp-based resolution, remove ordering hack | | `.../caching_instance_queryer.py` | Add timestamp fetching for updated partitions | | `.../asset_graph_view/asset_graph_view.py` | Add `compute_updated_since_..._with_timestamps()` | | `.../serialized_objects.py` | No changes needed — `subsets_with_metadata` already exists | ## Backward Compatibility - `subsets_with_metadata` was previously empty for modern conditions — populating it is additive - SinceCondition falls back to current behavior when timestamps aren't available - No cursor schema changes needed (timestamps are consumed same-tick, not persisted across ticks for this use case) - Existing serialized cursors deserialize fine ## Verification 1. Existing tests pass: `test_since_condition.py`, `test_updated_since_cron_condition.py` 2. New tests: - CronTickPassed returns `SubsetResult` with correct cron tick timestamp - NewlyUpdated returns `SubsetResult` with per-partition materialization timestamps - SinceCondition with both trigger+reset on same tick resolves correctly using timestamps - SinceCondition without timestamps (legacy conditions) falls back to current behavior - The CronTickPassed+NewlyUpdated special case works correctly without the hardcoded hack 3. Integration: verify via the automation condition evaluation UI that timestamps appear in metadata </details> <!-- WARNING: Machine-generated. Manual edits may break erk tooling. --> <!-- erk:metadata-block:plan-header --> <details> <summary>plan-header</summary> ```yaml schema_version: '2' created_at: '2026-03-09T11:09:55.909154+00:00' created_by: OwenKephart plan_comment_id: null last_dispatched_run_id: null last_dispatched_node_id: null last_dispatched_at: null last_local_impl_at: '2026-03-09T18:35:34.455616+00:00' last_local_impl_event: ended last_local_impl_session: 58b2ad56-024a-4c2e-968c-96499cb08551 last_local_impl_user: owen last_remote_impl_at: null last_remote_impl_run_id: null last_remote_impl_session_id: null branch_name: plnd/add-event-timestamps-since-03-09-1109 created_from_session: 97ef5a88-1256-4b7b-bdf7-1499b38cd062 lifecycle_stage: impl last_session_branch: async-learn/21494 last_session_id: 58b2ad56-024a-4c2e-968c-96499cb08551 last_session_at: '2026-03-09T11:35:41.024671+00:00' last_session_source: local worktree_name: internal-d ``` </details> <!-- /erk:metadata-block:plan-header --> --- To replicate this PR locally, run: ``` erk pr teleport 21494 ``` Internal-RevId: 94ca2d5557265db32db21c23a4721acc3858f8d4
1 parent 05f192f commit ae854dc

8 files changed

Lines changed: 411 additions & 33 deletions

File tree

python_modules/dagster/dagster/_core/asset_graph_view/asset_graph_view.py

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -923,6 +923,56 @@ async def _compute_updated_since_cursor_subset(
923923
).value
924924
return EntitySubset(self, key=key, value=_ValidatedEntitySubsetValue(value))
925925

926+
def compute_subsets_by_latest_materialization_timestamp(
927+
self, subset: EntitySubset[AssetKey]
928+
) -> dict[float, EntitySubset[AssetKey]]:
929+
"""Returns a mapping from materialization event timestamps to entity subsets.
930+
931+
For unpartitioned assets, uses the cached asset record. For partitioned assets,
932+
fetches storage IDs per partition and groups by timestamp.
933+
"""
934+
from dagster._core.event_api import AssetRecordsFilter
935+
936+
key = check.inst(subset.key, AssetKey)
937+
938+
# Fast path for unpartitioned assets — use cached asset record
939+
if not subset.is_partitioned:
940+
asset_record = self._queryer.get_asset_record(key)
941+
record = asset_record.asset_entry.last_materialization_record if asset_record else None
942+
if record is not None:
943+
return {record.timestamp: subset}
944+
return {}
945+
946+
# Partitioned: use caching queryer to get storage IDs per partition
947+
asset_partitions = subset.expensively_compute_asset_partitions()
948+
valid_storage_ids = {}
949+
for ap in asset_partitions:
950+
sid = self._queryer.get_latest_materialization_or_observation_storage_id(ap)
951+
if sid is not None:
952+
valid_storage_ids[ap] = sid
953+
if not valid_storage_ids:
954+
return {}
955+
956+
result = self._queryer.instance.fetch_materializations(
957+
AssetRecordsFilter(
958+
asset_key=key,
959+
storage_ids=list(valid_storage_ids.values()),
960+
),
961+
limit=len(valid_storage_ids),
962+
)
963+
storage_id_to_ts = {r.storage_id: r.timestamp for r in result.records}
964+
965+
# Group partitions by timestamp
966+
by_timestamp: dict[float, set[AssetKeyPartitionKey]] = {}
967+
for ap, sid in valid_storage_ids.items():
968+
if sid in storage_id_to_ts:
969+
by_timestamp.setdefault(storage_id_to_ts[sid], set()).add(ap)
970+
971+
return {
972+
ts: self.get_asset_subset_from_asset_partitions(key, akpks)
973+
for ts, akpks in by_timestamp.items()
974+
}
975+
926976
async def _compute_updated_since_time_subset(
927977
self, key: AssetCheckKey, time: datetime
928978
) -> EntitySubset[AssetCheckKey]:
Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
from collections.abc import Mapping
2+
from typing import Generic
3+
4+
from dagster._core.asset_graph_view.entity_subset import EntitySubset
5+
from dagster._core.definitions.asset_key import T_EntityKey
6+
from dagster._record import record
7+
8+
9+
@record
10+
class TimingMetadata(Generic[T_EntityKey]):
11+
"""Encapsulates per-timestamp entity subsets from an AutomationResult.
12+
13+
Stores a mapping from event timestamp to the EntitySubset that occurred at that
14+
timestamp. Used by SinceCondition to make timestamp-precise decisions when both
15+
trigger and reset conditions fire on the same evaluation tick.
16+
"""
17+
18+
timestamps: Mapping[float, EntitySubset[T_EntityKey]]
19+
20+
def subset_with_later_timestamps_than(
21+
self,
22+
other: "TimingMetadata[T_EntityKey]",
23+
empty: EntitySubset[T_EntityKey],
24+
) -> EntitySubset[T_EntityKey]:
25+
"""Returns the subset where self has a later timestamp than other."""
26+
result = empty
27+
for self_ts, self_subset in self.timestamps.items():
28+
to_add = self_subset
29+
for other_ts, other_subset in other.timestamps.items():
30+
if other_ts >= self_ts:
31+
to_add = to_add.compute_difference(other_subset)
32+
result = result.compute_union(to_add)
33+
return result

python_modules/dagster/dagster/_core/definitions/declarative_automation/automation_condition.py

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@
1111
from dagster._annotations import beta, hidden_param, only_allow_hidden_params_in_kwargs, public
1212
from dagster._core.asset_graph_view.entity_subset import EntitySubset
1313
from dagster._core.asset_graph_view.serializable_entity_subset import SerializableEntitySubset
14+
from dagster._core.asset_graph_view.timing_metadata import TimingMetadata
1415
from dagster._core.definitions.asset_key import (
1516
AssetCheckKey,
1617
AssetKey,
@@ -884,6 +885,7 @@ def _get_stable_unique_id(self, target_key: EntityKey | None) -> str:
884885
@hidden_param(param="subsets_with_metadata", breaking_version="", emit_runtime_warning=False)
885886
@hidden_param(param="structured_cursor", breaking_version="", emit_runtime_warning=False)
886887
@hidden_param(param="metadata", breaking_version="", emit_runtime_warning=False)
888+
@hidden_param(param="timing_metadata", breaking_version="", emit_runtime_warning=False)
887889
class AutomationResult(Generic[T_EntityKey]):
888890
"""The result of evaluating an AutomationCondition."""
889891

@@ -916,6 +918,11 @@ def __init__(
916918
kwargs.get("metadata"), "metadata", key_type=str, value_type=object
917919
)
918920

921+
# in-memory-only timing metadata for timestamp-precise trigger/reset resolution
922+
self._timing_metadata: TimingMetadata | None = check.opt_inst_param(
923+
kwargs.get("timing_metadata"), "timing_metadata", TimingMetadata
924+
)
925+
919926
# hidden_param which should only be set by builtin conditions which require high performance
920927
# in their serdes layer
921928
structured_cursor = kwargs.get("structured_cursor")
@@ -939,6 +946,10 @@ def key(self) -> T_EntityKey:
939946
def true_subset(self) -> EntitySubset[T_EntityKey]:
940947
return self._true_subset
941948

949+
@property
950+
def timing_metadata(self) -> TimingMetadata | None:
951+
return self._timing_metadata
952+
942953
@property
943954
def start_timestamp(self) -> float:
944955
return self._start_timestamp

python_modules/dagster/dagster/_core/definitions/declarative_automation/operands/operands.py

Lines changed: 51 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,12 @@
11
import datetime
2+
import logging
3+
import os
24

35
from dagster_shared.serdes import whitelist_for_serdes
46
from dagster_shared.serdes.utils import SerializableTimeDelta
57

68
from dagster._core.asset_graph_view.entity_subset import EntitySubset
9+
from dagster._core.asset_graph_view.timing_metadata import TimingMetadata
710
from dagster._core.definitions.asset_key import AssetCheckKey, AssetKey
811
from dagster._core.definitions.declarative_automation.automation_condition import (
912
AutomationResult,
@@ -12,6 +15,7 @@
1215
from dagster._core.definitions.declarative_automation.automation_context import AutomationContext
1316
from dagster._core.definitions.declarative_automation.operands.subset_automation_condition import (
1417
SubsetAutomationCondition,
18+
TimedSubsetAutomationCondition,
1519
)
1620
from dagster._core.definitions.freshness import FreshnessState
1721
from dagster._core.definitions.partitions.snap.snap import PartitionsSnap
@@ -185,18 +189,53 @@ def compute_subset(self, context: AutomationContext) -> EntitySubset:
185189

186190
@whitelist_for_serdes
187191
@record
188-
class NewlyUpdatedCondition(SubsetAutomationCondition):
192+
class NewlyUpdatedCondition(TimedSubsetAutomationCondition):
189193
@property
190194
def name(self) -> str:
191195
return "newly_updated"
192196

193-
async def compute_subset(self, context: AutomationContext) -> EntitySubset: # pyright: ignore[reportIncompatibleMethodOverride]
197+
async def compute_subset_with_timing_metadata( # pyright: ignore[reportIncompatibleMethodOverride]
198+
self, context: AutomationContext
199+
) -> tuple[EntitySubset, TimingMetadata | None]:
194200
# if it's the first time evaluating, just return the empty subset
195201
if context.previous_temporal_context is None:
196-
return context.get_empty_subset()
197-
return await context.asset_graph_view.compute_updated_since_temporal_context_subset(
202+
return context.get_empty_subset(), None
203+
204+
if not isinstance(context.key, AssetKey):
205+
# For non-asset keys (e.g. checks), no timestamp enrichment yet
206+
subset = await context.asset_graph_view.compute_updated_since_temporal_context_subset(
207+
key=context.key, temporal_context=context.previous_temporal_context
208+
)
209+
return subset, None
210+
211+
subset = await context.asset_graph_view.compute_updated_since_temporal_context_subset(
198212
key=context.key, temporal_context=context.previous_temporal_context
199213
)
214+
if subset.is_empty:
215+
return subset, None
216+
217+
max_partitions = int(os.environ.get("DAGSTER_MAX_PARTITIONS_FOR_DA_TIMESTAMP_FETCH", "100"))
218+
if subset.size > max_partitions:
219+
# Too many partitions to fetch individual timestamps — use the current
220+
# tick timestamp for the entire subset so that downstream SinceCondition
221+
# logic still has timing metadata to work with.
222+
logging.getLogger("dagster").warning(
223+
"Asset %s has %d updated partitions, exceeding the maximum of %d for"
224+
" per-partition timestamp fetching. Using tick timestamp as fallback.",
225+
context.key.to_user_string(),
226+
subset.size,
227+
max_partitions,
228+
)
229+
return subset, TimingMetadata(
230+
timestamps={context.asset_graph_view.effective_dt.timestamp(): subset}
231+
)
232+
233+
timing_subsets = (
234+
context.asset_graph_view.compute_subsets_by_latest_materialization_timestamp(subset)
235+
)
236+
if not timing_subsets:
237+
return subset, None
238+
return subset, TimingMetadata(timestamps=timing_subsets)
200239

201240

202241
@whitelist_for_serdes
@@ -232,7 +271,7 @@ async def compute_subset(self, context: AutomationContext) -> EntitySubset: # p
232271

233272
@whitelist_for_serdes
234273
@record
235-
class CronTickPassedCondition(SubsetAutomationCondition):
274+
class CronTickPassedCondition(TimedSubsetAutomationCondition):
236275
cron_schedule: str
237276
cron_timezone: str
238277

@@ -248,17 +287,21 @@ def _get_previous_cron_tick(self, effective_dt: datetime.datetime) -> datetime.d
248287
)
249288
return next(previous_ticks)
250289

251-
def compute_subset(self, context: AutomationContext) -> EntitySubset:
290+
def compute_subset_with_timing_metadata(
291+
self, context: AutomationContext
292+
) -> tuple[EntitySubset, TimingMetadata | None]:
252293
previous_cron_tick = self._get_previous_cron_tick(context.evaluation_time)
253294
if (
254295
# no previous evaluation
255296
context.previous_evaluation_time is None
256297
# cron tick was not newly passed
257298
or previous_cron_tick < context.previous_evaluation_time
258299
):
259-
return context.get_empty_subset()
300+
return context.get_empty_subset(), None
260301
else:
261-
return context.candidate_subset
302+
candidate_subset = context.candidate_subset
303+
cron_tick_ts = previous_cron_tick.timestamp()
304+
return candidate_subset, TimingMetadata(timestamps={cron_tick_ts: candidate_subset})
262305

263306

264307
@whitelist_for_serdes

python_modules/dagster/dagster/_core/definitions/declarative_automation/operands/subset_automation_condition.py

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22
from abc import abstractmethod
33

44
from dagster._core.asset_graph_view.entity_subset import EntitySubset
5+
from dagster._core.asset_graph_view.timing_metadata import TimingMetadata
56
from dagster._core.definitions.asset_key import T_EntityKey
67
from dagster._core.definitions.declarative_automation.automation_condition import (
78
AutomationResult,
@@ -36,3 +37,41 @@ async def evaluate( # pyright: ignore[reportIncompatibleMethodOverride]
3637
true_subset = self.compute_subset(context)
3738

3839
return AutomationResult(context, true_subset)
40+
41+
42+
@record
43+
class TimedSubsetAutomationCondition(BuiltinAutomationCondition[T_EntityKey]):
44+
"""Base class for conditions that compute a subset along with timing metadata.
45+
46+
Subclasses implement compute_subset_with_timing_metadata() which returns both the subset
47+
and an optional TimingMetadata. The evaluate() method stores timing metadata directly
48+
on the AutomationResult for use by SinceCondition.
49+
"""
50+
51+
@property
52+
def requires_cursor(self) -> bool:
53+
return False
54+
55+
@abstractmethod
56+
def compute_subset_with_timing_metadata(
57+
self, context: AutomationContext[T_EntityKey]
58+
) -> tuple[EntitySubset[T_EntityKey], TimingMetadata[T_EntityKey] | None]: ...
59+
60+
async def evaluate( # pyright: ignore[reportIncompatibleMethodOverride]
61+
self, context: AutomationContext[T_EntityKey]
62+
) -> AutomationResult[T_EntityKey]:
63+
# don't compute anything if there are no candidates
64+
if context.candidate_subset.is_empty:
65+
true_subset = context.get_empty_subset()
66+
return AutomationResult(context, true_subset)
67+
68+
if inspect.iscoroutinefunction(self.compute_subset_with_timing_metadata):
69+
true_subset, timing_metadata = await self.compute_subset_with_timing_metadata(context)
70+
else:
71+
true_subset, timing_metadata = self.compute_subset_with_timing_metadata(context)
72+
73+
return AutomationResult(
74+
context,
75+
true_subset,
76+
timing_metadata=timing_metadata,
77+
)

python_modules/dagster/dagster/_core/definitions/declarative_automation/operators/since_operator.py

Lines changed: 16 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -111,19 +111,6 @@ async def evaluate( # pyright: ignore[reportIncompatibleMethodOverride]
111111
# must evaluate child condition over the entire subset to avoid missing state transitions
112112
child_candidate_subset = context.asset_graph_view.get_full_subset(key=context.key)
113113

114-
from dagster._core.definitions.declarative_automation.operands import (
115-
CronTickPassedCondition,
116-
NewlyUpdatedCondition,
117-
)
118-
119-
# Bit of a hack to ensure that all_deps_updated_since_cron doesn't drop new updates that
120-
# happen in the same tick as the evaluation where the cron tick passes.
121-
122-
reset_newly_true = not (
123-
isinstance(self.reset_condition, CronTickPassedCondition)
124-
and isinstance(self.trigger_condition, NewlyUpdatedCondition)
125-
)
126-
127114
# compute result for trigger and reset conditions
128115
trigger_result, reset_result = await asyncio.gather(
129116
*[
@@ -143,17 +130,22 @@ async def evaluate( # pyright: ignore[reportIncompatibleMethodOverride]
143130
# take the previous subset that this was true for
144131
true_subset = context.previous_true_subset or context.get_empty_subset()
145132

146-
if reset_newly_true:
147-
# add in any newly true trigger asset partitions
148-
true_subset = true_subset.compute_union(trigger_result.true_subset)
149-
# remove any newly true reset asset partitions
150-
true_subset = true_subset.compute_difference(reset_result.true_subset)
151-
else:
152-
# remove any newly true reset asset partitions
153-
true_subset = true_subset.compute_difference(reset_result.true_subset)
133+
trigger_timing = trigger_result.timing_metadata
134+
reset_timing = reset_result.timing_metadata
135+
136+
# Step 1: Add all newly-true trigger partitions
137+
true_subset = true_subset.compute_union(trigger_result.true_subset)
138+
139+
# Step 2: Remove all newly-true reset partitions
140+
true_subset = true_subset.compute_difference(reset_result.true_subset)
154141

155-
# add in any newly true trigger asset partitions
156-
true_subset = true_subset.compute_union(trigger_result.true_subset)
142+
# Step 3: Use TimingMetadata to re-add partitions where trigger fired after reset
143+
both = trigger_result.true_subset.compute_intersection(reset_result.true_subset)
144+
if not both.is_empty and trigger_timing and reset_timing:
145+
trigger_wins = trigger_timing.subset_with_later_timestamps_than(
146+
reset_timing, empty=context.get_empty_subset()
147+
).compute_intersection(both)
148+
true_subset = true_subset.compute_union(trigger_wins)
157149

158150
# if anything changed since the previous evaluation, update the metadata
159151
condition_data = SinceConditionData.from_metadata(context.previous_metadata).update(
@@ -167,6 +159,7 @@ async def evaluate( # pyright: ignore[reportIncompatibleMethodOverride]
167159
context=context,
168160
true_subset=true_subset,
169161
child_results=[trigger_result, reset_result],
162+
timing_metadata=trigger_timing,
170163
metadata=condition_data.to_metadata(),
171164
)
172165

0 commit comments

Comments
 (0)