What's the issue?
The "Commit tick result" button in the Web UI only commits a sensor's cursor, dynamic partition requests, and run requests. Asset events detected by the sensor are silently dropped.
What did you expect to happen?
"Commit tick result" should also apply asset_events (report each one via the same mechanism the sensor daemon uses), so a committed tick fully replicates what a live daemon tick would do.
How to reproduce?
from dagster import (
AssetKey,
AssetMaterialization,
Definitions,
DynamicPartitionsDefinition,
RunRequest,
SensorResult,
asset,
sensor,
)
partitions_def = DynamicPartitionsDefinition(name="demo_keys")
@asset(partitions_def=partitions_def)
def demo_asset():
return None
@sensor(asset_selection="demo_asset")
def demo_sensor(context):
key = "demo-partition-1"
return SensorResult(
cursor=key,
dynamic_partitions_requests=[partitions_def.build_add_request([key])],
asset_events=[AssetMaterialization(asset_key=AssetKey("demo_asset"), partition=key)],
)
defs = Definitions(assets=[demo_asset], sensors=[demo_sensor])
dg dev with the above
- Open the Automation page, select
demo_sensor, then click "Preview tick result"
- Evaluate the tick. The preview shows the new dynamic partition and the reported asset event.
- Click "Commit tick result"
- Open
demo_asset's Partitions tab: demo-partition-1 now exists as a registered partition, but shows as unmaterialized (0%) — no materialization event was recorded, despite the sensor returning one.
Dagster version
1.13.12
Deployment type
Local
Deployment details
Reproduced against dg dev in a local uv-managed project. Also occurs in a Hybrid deployment, using the Dagster+ UI.
Additional information
Discovered while debugging a sensor that registers S3-file-arrival dynamic partitions and reports the raw file as materialized in the same SensorResult.
The use case for this is monitoring arrival of a raw file, treating the raw file as an externally produced asset, and using eager automation to trigger downstream assets. This seems to be also the goal of ObservableAssets, although support for only partially observing an asset is severely lacking there, which makes that feature unusable for buckets containing a large number of objects.
Message from the maintainers
Impacted by this issue? Give it a 👍! We factor engagement into prioritization.
What's the issue?
The "Commit tick result" button in the Web UI only commits a sensor's cursor, dynamic partition requests, and run requests. Asset events detected by the sensor are silently dropped.
What did you expect to happen?
"Commit tick result" should also apply
asset_events(report each one via the same mechanism the sensor daemon uses), so a committed tick fully replicates what a live daemon tick would do.How to reproduce?
dg devwith the abovedemo_sensor, then click "Preview tick result"demo_asset's Partitions tab:demo-partition-1now exists as a registered partition, but shows as unmaterialized (0%) — no materialization event was recorded, despite the sensor returning one.Dagster version
1.13.12
Deployment type
Local
Deployment details
Reproduced against
dg devin a local uv-managed project. Also occurs in a Hybrid deployment, using the Dagster+ UI.Additional information
Discovered while debugging a sensor that registers S3-file-arrival dynamic partitions and reports the raw file as materialized in the same
SensorResult.The use case for this is monitoring arrival of a raw file, treating the raw file as an externally produced asset, and using eager automation to trigger downstream assets. This seems to be also the goal of ObservableAssets, although support for only partially observing an asset is severely lacking there, which makes that feature unusable for buckets containing a large number of objects.
Message from the maintainers
Impacted by this issue? Give it a 👍! We factor engagement into prioritization.