Dagster-like cursor

[dinya]

Hello team,

I want to create a workflow that contains periodic checking of the data source for new data and subsequent processing if ones are presented.

One of the conceptual problems for me is saving the time cursor (when the last check was done). The cursor is used to query to the data source like "get all the data changed since the last request".

For example "cursor" entity is a feature of Dagster (https://docs.dagster.io/guides/automate/sensors#cursors-and-high-volume-events).

Is there any generally accepted similar approach in Prefect?

At a glance, it looks something like setting/getting Variable (Block) during each task run.

[cicdw]

@dinya you're correct that using a Variable would be the best way to achieve this. You can store any JSON-able object (e.g., an ISO-formatted timestamp) in a Variable. I don't recommend using a Block as blocks are much heavier server-side entities that were really built for managing credentials more so than frequently updated bits of data.