Asset sensor internal cursor updates

[sharrik111]

What's the issue or suggestion?

Dagster docs tell that users are allowed to use cursors inside sensors to retain state between sensors' evaluations. Examples for this are given for dg.sensor.

Intuitively, it's assumed that dg.asset_sensor offers similar functionality. But in fact, dg.AssetSensorDefinition uses the cursor itself to control asset materializations. Thus, cursor updates made inside a sensor function are being overriden.

I leave this issue in the area of docs since I'm not sure is it a bug or feature. For me, it's more of a bug. But if it's a feature, it should be propertly documented (e.g. here

Additional information

Here's the place in code where the cursor is being updated

dagster/python_modules/dagster/dagster/_core/definitions/asset_sensor_definition.py

Line 156 in a28a23f

context.update_cursor(str(event_record.storage_id))

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

[johnny603]

I am working on this

[michaelstockton]

@johnny603 Thank you for looking into this bug. I tried your documented example by calling context.update_cursor before returning a dg.RunRequest, but I don't see the user-defined cursor statefully persisting between runs. This seems consistent with @sharrik111's bug report and the code in asset_sensor_definition where the cursor is updated after the materialization function is evaluated.

Can you clarify whether @dg.asset_sensor supports user-defined cursors? If update_cursor is called within a sensor function, why does context.cursor not reflect the user-defined value between runs? If it's not possible to statefully track a user-defined cursor, what's the alternative?