Skip to content

[asset health] include freshness in asset health #29268

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Apr 15, 2025

Conversation

jamiedemaria
Copy link
Contributor

@jamiedemaria jamiedemaria commented Apr 14, 2025

Summary & Motivation

Includes the freshness status of each asset in asset health. Follows these rules:

  • HEALTHY - the freshness policy is in a PASS-ing state
  • WARNING - the freshness policy is in a WARN-ing state
  • DEGRADED - the freshness policy is in a FAIL-ing state
  • UNKNOWN - the freshness policy has never been evaluated or is in an UNKNOWN state
  • NOT_APPLICABLE - the asset does not have a freshness policy defined.

I need to access the freshness getter function from OSS, so i added an instance method for it. This could also be solved by moving the storage methods to a shared storage class (Run or EventLog probably). I don't have a preference and picked adding the Instance method for no specific reason. Similarly, I needed to move FreshnessRecord and FreshnessRecordBody to OSS so that I could use them as type hints

How I Tested These Changes

tests in https://github.com/dagster-io/internal/pull/15022

Copy link
Contributor Author

This stack of pull requests is managed by Graphite. Learn more about stacking.

Copy link

github-actions bot commented Apr 14, 2025

Deploy preview for dagit-core-storybook ready!

✅ Preview
https://dagit-core-storybook-d3ay9hlgv-elementl.vercel.app
https://jamie-asset-health-freshness.core-storybook.dagster-docs.io

Built with commit c0342ff.
This pull request is being automatically deployed with vercel-action

@jamiedemaria jamiedemaria changed the title wip [asset health] include freshness in asset health Apr 14, 2025
@jamiedemaria jamiedemaria marked this pull request as ready for review April 14, 2025 21:50
@jamiedemaria jamiedemaria force-pushed the jamie/asset-health-freshness branch from 3d71f5b to 6f561f8 Compare April 14, 2025 22:02
Comment on lines 81 to 95
@record
class FreshnessStateRecord:
entity_key: AssetKey
freshness_state: FreshnessState
updated_at: datetime
record_body: FreshnessStateRecordBody

@staticmethod
def from_db_row(db_row: Row):
return FreshnessStateRecord(
entity_key=check.not_none(AssetKey.from_db_string(db_row[0])),
freshness_state=FreshnessState(db_row[3]),
record_body=deserialize_value(db_row[4], FreshnessStateRecordBody),
updated_at=db_row[5],
)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The FreshnessStateRecord class should have the @whitelist_for_serdes decorator applied to ensure proper serialization/deserialization through Dagster's serialization system. This is consistent with other similar record classes in the codebase (like FreshnessStateRecordBody). Without this decorator, there may be issues when storing and retrieving these records.

@whitelist_for_serdes  # Add this decorator
@record
class FreshnessStateRecord:
    # ...
Suggested change
@record
class FreshnessStateRecord:
entity_key: AssetKey
freshness_state: FreshnessState
updated_at: datetime
record_body: FreshnessStateRecordBody
@staticmethod
def from_db_row(db_row: Row):
return FreshnessStateRecord(
entity_key=check.not_none(AssetKey.from_db_string(db_row[0])),
freshness_state=FreshnessState(db_row[3]),
record_body=deserialize_value(db_row[4], FreshnessStateRecordBody),
updated_at=db_row[5],
)
@whitelist_for_serdes
@record
class FreshnessStateRecord:
entity_key: AssetKey
freshness_state: FreshnessState
updated_at: datetime
record_body: FreshnessStateRecordBody
@staticmethod
def from_db_row(db_row: Row):
return FreshnessStateRecord(
entity_key=check.not_none(AssetKey.from_db_string(db_row[0])),
freshness_state=FreshnessState(db_row[3]),
record_body=deserialize_value(db_row[4], FreshnessStateRecordBody),
updated_at=db_row[5],
)

Spotted by Diamond

Is this helpful? React 👍 or 👎 to let us know.

Copy link
Contributor

This is going to require some coordination with the freshness stack, since the daemon and graphql rely on types that are moved to OSS by this PR:

👀 #14963 [4/n] gql resolver to fetch freshness state for asset +183/-240
👀 #14786 [3/n] freshness daemon and evaluators +813/-1
🚀 #14763 [2/n] freshness storage classes and methods +213/-1
🚀 #14759 [1/n] [freshness] Persist freshness state +72/-0

@@ -3395,6 +3395,9 @@ def report_runless_asset_event(
),
)

def get_entity_freshness_state(self, entity_key: AssetKey) -> Optional[FreshnessStateRecord]:
return None
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this need any warning or error message to signify that it's not implemented in OSS / is cloud only?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since the method isn't marked public i think it's probably fine? in other similar cases we will usually return None, [], etc and not have a warning, but there's probably no harm in having a warning either

@jamiedemaria jamiedemaria force-pushed the jamie/asset-health-freshness branch from 6f561f8 to 80c3a78 Compare April 15, 2025 13:59
@anuthebananu anuthebananu self-requested a review April 15, 2025 14:35
Copy link
Contributor

@anuthebananu anuthebananu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

@jamiedemaria jamiedemaria merged commit 3642d11 into master Apr 15, 2025
7 checks passed
@jamiedemaria jamiedemaria deleted the jamie/asset-health-freshness branch April 15, 2025 15:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants