A new asset freshness API #28828
Merged
A new asset freshness API #28828
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
anuthebananu
changed the title
anuthebananu
force-pushed
the
anurag/oper-1768-freshness-sla-python-api
branch
3 times, most recently
from
c87a0f4 to
45ee60e
Compare
anuthebananu
force-pushed
the
anurag/oper-1768-freshness-sla-python-api
branch
3 times, most recently
from
2ed1a73 to
29d5a43
Compare
anuthebananu
changed the title
![graphite-app[bot]](https://avatars.githubusercontent.com/in/158384?s=60&v=4){kind=small}
python_modules/dagster/dagster/_core/definitions/decorators/asset_decorator.py
Outdated
Show resolved
Hide resolved
anuthebananu
force-pushed
the
anurag/oper-1768-freshness-sla-python-api
branch
from
29d5a43 to
c43e303
Compare
anuthebananu
force-pushed
the
anurag/oper-1768-freshness-sla-python-api
branch
2 times, most recently
from
d26e982 to
18c9cfe
Compare
OwenKephart
requested changes
Apr 8, 2025
There was a problem hiding this comment.
not all the comments here are blocking, but the main thing is that I really don't think we can add a new non-hidden argument to AssetSpec that we plan to remove in the near future, especially due to the number of APIs we end up needing to thread this ephemeral parameter through.
python_modules/dagster/dagster/_core/definitions/decorators/asset_decorator.py
Outdated
Show resolved
Hide resolved
python_modules/dagster/dagster/_core/definitions/new_freshness_policy.py
Outdated
Show resolved
Hide resolved
python_modules/dagster/dagster/_core/definitions/new_freshness_policy.py
Outdated
Show resolved
Hide resolved
python_modules/dagster/dagster/_core/definitions/new_freshness_policy.py
Outdated
Show resolved
Hide resolved
python_modules/dagster/dagster/_core/definitions/new_freshness_policy.py
Outdated
Show resolved
Hide resolved
dpeng817
reviewed
Apr 9, 2025
python_modules/dagster/dagster/_core/definitions/new_freshness_policy.py
Outdated
Show resolved
Hide resolved
python_modules/dagster/dagster/_core/definitions/new_freshness_policy.py
Outdated
Show resolved
Hide resolved
python_modules/dagster/dagster/_core/definitions/new_freshness_policy.py
Outdated
Show resolved
Hide resolved
python_modules/dagster/dagster/_core/definitions/remote_asset_graph.py
Outdated
Show resolved
Hide resolved
anuthebananu
force-pushed
the
anurag/oper-1768-freshness-sla-python-api
branch
from
c56de48 to
985651a
Compare
dpeng817
reviewed
Apr 9, 2025
anuthebananu
force-pushed
the
anurag/oper-1768-freshness-sla-python-api
branch
from
6bab510 to
cf7f5bb
Compare
anuthebananu
force-pushed
the
anurag/oper-1768-freshness-sla-python-api
branch
from
9b6754f to
066d781
Compare
OwenKephart
approved these changes
Apr 9, 2025
..._modules/dagster/dagster/_core/definitions/decorators/decorator_assets_definition_builder.py
Outdated
Show resolved
Hide resolved
|
knew i was forgetting something lol |
anuthebananu
force-pushed
the
anurag/oper-1768-freshness-sla-python-api
branch
2 times, most recently
from
3e9bd60 to
6d652e4
Compare
|
Deploy preview for dagit-core-storybook ready! ✅ Preview Built with commit 1c20726. |
anuthebananu
force-pushed
the
anurag/oper-1768-freshness-sla-python-api
branch
from
69eb992 to
d2b2743
Compare
|
I've ignored a couple of pyright errors, mainly stemming from the fact that we have a serdes class Pyright somehow cannot recognize that it is not possible to instantiate because To me these feel safe enough to ignore for now to get this merged, if desired we can come up with a better solution in the future. |
anuthebananu
force-pushed
the
anurag/oper-1768-freshness-sla-python-api
branch
from
ac60857 to
5d3a983
Compare
anuthebananu
force-pushed
the
anurag/oper-1768-freshness-sla-python-api
branch
from
5d3a983 to
1c20726
Compare
benpankow
pushed a commit
that referenced
this pull request
Apr 10, 2025
Introduces a new API for defining freshness as a top-level attribute on an asset. This API will replace freshness checks and freshness policies as the way to model asset freshness in Dagster. For more details on the motivation behind needing a new API, please refer to the [internal product review](https://www.notion.so/dagster/New-Asset-Freshness-System-Product-Review-1c318b92e46280fe859ce86f5d78934a?d=1c718b92e462807ca2e1001c0a331dc0#1c718b92e46280c8973ef9358ec38367). The implementation here is slightly different than the desired end state, which is to reclaim the deprecated `FreshnessPolicy` construct. Instead we're writing the new freshness policy to the asset spec metadata. This will unblock internal testing and give us time to implement a longer-term migration plan for `FreshnessPolicy`. - Relocate `SerializableTimeDelta` from `dagster._core.definitions.declarative_automation.util` to `dagster_shared.serdes.utils` - new + existing unit tests for backward compatibility > Insert changelog entry or delete this section.
benpankow
pushed a commit
that referenced
this pull request
Apr 10, 2025
## Summary & Motivation Fixes a Python 3.9 backcompat issue with unit tests introduced in #28828. ## How I Tested These Changes bk ## Changelog > Insert changelog entry or delete this section.
benpankow
pushed a commit
that referenced
this pull request
Apr 10, 2025
## Summary & Motivation Fixes a Python 3.9 backcompat issue with unit tests introduced in #28828. ## How I Tested These Changes bk ## Changelog > Insert changelog entry or delete this section.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary & Motivation
Introduces a new API for defining freshness as a top-level attribute on an asset. This API will replace freshness checks and freshness policies as the way to model asset freshness in Dagster.
For more details on the motivation behind needing a new API, please refer to the internal product review.
The implementation here is slightly different than the desired end state, which is to reclaim the deprecated
FreshnessPolicyconstruct. Instead we're writing the new freshness policy to the asset spec metadata. This will unblock internal testing and give us time to implement a longer-term migration plan forFreshnessPolicy.Auxiliary Changes
SerializableTimeDeltafromdagster._core.definitions.declarative_automation.utiltodagster_shared.serdes.utilsHow I Tested These Changes
Changelog