Skip to content

[pac] add AssetCheckState#33243

Merged
OwenKephart merged 2 commits into
masterfrom
01-09-_pac_add_assetcheckstatuscachevalue
Jan 26, 2026
Merged

[pac] add AssetCheckState#33243
OwenKephart merged 2 commits into
masterfrom
01-09-_pac_add_assetcheckstatuscachevalue

Conversation

@OwenKephart

@OwenKephart OwenKephart commented Jan 12, 2026

Copy link
Copy Markdown
Contributor

Summary & Motivation

This adds in a new caching object that incrementally calculates the state across all partitions of a check.

The state transition logic here is fairly complex because the underlying reality of asset check status is itself very complex! In particular, we need information about both the asset check execution itself, as well as any materializations that target the relevant asset.

We have a reasonably extensive suite of tests in a downstack PR that this is validated against.

Note that this PR does NOT actually improve performance because we do not incrementally cache the state anywhere at this point in time. This purely rebuilds from scratch every single time we fetch this. This PR is purely for demonstrating that the state transition logic is correct, and an upstack PR will add database methods for storing incremental versions of this state.

We do a bit of a funky thing here in the storage layer, where we have a get_stored_asset_check_state as well as a regular get_asset_check_state.

This setup allows us to:

  • have a dedicated storage method for getting the most-up-to-date possible information, which we use for the LoadableBy framework stuff (i.e. you always want AssetCheckState.gen() to give you the most up to date information)
  • have the ability to modulate between different storage implementations what process is responsible for keeping the incremental value up to date -- in an upstack PR, we'll eventually override get_asset_check_state on SqlEventLogStorage to write the updated values back in place, but in other implementations we have the flexibility to have a completely separate process handle keeping the incremental value up to date, meaning get_asset_check_state will not mutate the state of the instance

How I Tested These Changes

Changelog

NOCHANGELOG

@OwenKephart
OwenKephart force-pushed the 01-09-_pac_add_get_asset_check_partitions_storage_method branch from 323aaa3 to c752369 Compare January 14, 2026 00:29
@OwenKephart
OwenKephart force-pushed the 01-09-_pac_add_assetcheckstatuscachevalue branch from 7c3dabb to 04a0928 Compare January 14, 2026 00:29
This was referenced Jan 14, 2026
@OwenKephart
OwenKephart force-pushed the 01-09-_pac_add_assetcheckstatuscachevalue branch from 04a0928 to 2d22edf Compare January 14, 2026 19:44
@OwenKephart
OwenKephart force-pushed the 01-09-_pac_add_get_asset_check_partitions_storage_method branch 2 times, most recently from 35d9920 to fc68c31 Compare January 14, 2026 21:10
@OwenKephart
OwenKephart force-pushed the 01-09-_pac_add_assetcheckstatuscachevalue branch from 2d22edf to a46ebcd Compare January 14, 2026 21:10
@OwenKephart
OwenKephart force-pushed the 01-09-_pac_add_get_asset_check_partitions_storage_method branch from fc68c31 to e438462 Compare January 14, 2026 21:45
@OwenKephart
OwenKephart force-pushed the 01-09-_pac_add_assetcheckstatuscachevalue branch from a46ebcd to 41c2116 Compare January 14, 2026 21:45
@OwenKephart
OwenKephart force-pushed the 01-09-_pac_add_get_asset_check_partitions_storage_method branch from e438462 to fccb6fe Compare January 14, 2026 23:04
@OwenKephart
OwenKephart force-pushed the 01-09-_pac_add_assetcheckstatuscachevalue branch 2 times, most recently from 8924e96 to 31a51b3 Compare January 16, 2026 00:42
@OwenKephart
OwenKephart force-pushed the 01-09-_pac_add_get_asset_check_partitions_storage_method branch 2 times, most recently from 94fb89a to b01a20b Compare January 16, 2026 00:43
@OwenKephart
OwenKephart force-pushed the 01-09-_pac_add_assetcheckstatuscachevalue branch from 31a51b3 to 3fd86b4 Compare January 16, 2026 00:43
@OwenKephart
OwenKephart force-pushed the 01-09-_pac_add_get_asset_check_partitions_storage_method branch from b01a20b to 4a49aff Compare January 16, 2026 01:03
@OwenKephart
OwenKephart force-pushed the 01-09-_pac_add_assetcheckstatuscachevalue branch from 3fd86b4 to 18cc9f0 Compare January 16, 2026 01:03
@OwenKephart
OwenKephart force-pushed the 01-09-_pac_add_get_asset_check_partitions_storage_method branch from 4a49aff to 7577749 Compare January 16, 2026 01:15
@OwenKephart
OwenKephart force-pushed the 01-09-_pac_add_assetcheckstatuscachevalue branch from 18cc9f0 to 2a99818 Compare January 16, 2026 01:15
@OwenKephart
OwenKephart force-pushed the 01-09-_pac_add_assetcheckstatuscachevalue branch from 2a99818 to e9e1b75 Compare January 16, 2026 20:41
@OwenKephart
OwenKephart force-pushed the 01-09-_pac_add_get_asset_check_partitions_storage_method branch from 7577749 to 243e63c Compare January 16, 2026 20:41
@OwenKephart
OwenKephart force-pushed the 01-09-_pac_add_assetcheckstatuscachevalue branch from b2d651b to b6a9634 Compare January 20, 2026 23:44
@OwenKephart
OwenKephart force-pushed the 01-09-_pac_add_get_asset_check_partitions_storage_method branch from f645bf8 to 58cd914 Compare January 20, 2026 23:44
@OwenKephart
OwenKephart force-pushed the 01-09-_pac_add_assetcheckstatuscachevalue branch from b6a9634 to 3c493b4 Compare January 21, 2026 00:01
@OwenKephart
OwenKephart force-pushed the 01-09-_pac_add_get_asset_check_partitions_storage_method branch from 58cd914 to 137c25d Compare January 21, 2026 00:01
@OwenKephart
OwenKephart force-pushed the 01-09-_pac_add_assetcheckstatuscachevalue branch from 3c493b4 to d3eef70 Compare January 22, 2026 00:11
@OwenKephart
OwenKephart force-pushed the 01-09-_pac_add_get_asset_check_partitions_storage_method branch from 137c25d to 8a6b75f Compare January 22, 2026 00:11
@OwenKephart
OwenKephart force-pushed the 01-09-_pac_add_get_asset_check_partitions_storage_method branch from 8a6b75f to a9ee90b Compare January 22, 2026 03:20
@OwenKephart
OwenKephart force-pushed the 01-09-_pac_add_assetcheckstatuscachevalue branch 2 times, most recently from f2824da to ca729a6 Compare January 22, 2026 21:25
@OwenKephart
OwenKephart force-pushed the 01-09-_pac_add_get_asset_check_partitions_storage_method branch from a9ee90b to 707bd1b Compare January 22, 2026 21:25
@OwenKephart
OwenKephart force-pushed the 01-09-_pac_add_get_asset_check_partitions_storage_method branch from 707bd1b to 888c311 Compare January 22, 2026 23:16
@OwenKephart
OwenKephart force-pushed the 01-09-_pac_add_assetcheckstatuscachevalue branch 2 times, most recently from 4271885 to bdf1795 Compare January 23, 2026 00:46
@OwenKephart
OwenKephart force-pushed the 01-09-_pac_add_get_asset_check_partitions_storage_method branch from 888c311 to 1bd7f5d Compare January 23, 2026 00:46
@OwenKephart
OwenKephart requested a review from gibsondan January 26, 2026 17:11
@OwenKephart
OwenKephart force-pushed the 01-09-_pac_add_get_asset_check_partitions_storage_method branch from 1bd7f5d to e4f8832 Compare January 26, 2026 18:26
@OwenKephart
OwenKephart force-pushed the 01-09-_pac_add_assetcheckstatuscachevalue branch from bdf1795 to b541011 Compare January 26, 2026 18:26
Comment thread python_modules/dagster/dagster/_core/storage/asset_check_state.py
Comment thread python_modules/dagster/dagster/_core/storage/asset_check_state.py
Comment thread python_modules/dagster/dagster/_core/storage/asset_check_state.py
Comment thread python_modules/dagster/dagster/_core/storage/asset_check_state.py
Comment on lines +118 to +132
# do a bulk fetch for runs across all states
finished_runs = (
instance.get_runs(
filters=RunsFilter(run_ids=list(run_ids_to_fetch), statuses=FINISHED_STATUSES)
)
if len(run_ids_to_fetch) > 0
else []
)
finished_runs_status_by_id = {run.run_id: run.status for run in finished_runs}
return {
key: initial_states[key].with_updates(
key, partitions_defs_by_key[key], infos_by_key[key], finished_runs_status_by_id
)
for key in check_keys
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how about the dreaded run deletion mid-execution case?

Comment thread python_modules/dagster/dagster/_core/storage/event_log/base.py

OwenKephart commented Jan 26, 2026

Copy link
Copy Markdown
Contributor Author

Merge activity

  • Jan 26, 8:50 PM UTC: A user started a stack merge that includes this pull request via Graphite.
  • Jan 26, 9:21 PM UTC: Graphite rebased this pull request as part of a merge.
  • Jan 26, 9:21 PM UTC: @OwenKephart merged this pull request with Graphite.

@OwenKephart
OwenKephart changed the base branch from 01-09-_pac_add_get_asset_check_partitions_storage_method to graphite-base/33243 January 26, 2026 21:18
@OwenKephart
OwenKephart changed the base branch from graphite-base/33243 to master January 26, 2026 21:19
@OwenKephart
OwenKephart force-pushed the 01-09-_pac_add_assetcheckstatuscachevalue branch from b541011 to a5cd5b3 Compare January 26, 2026 21:20
@OwenKephart
OwenKephart merged commit 6e8a71d into master Jan 26, 2026
4 of 5 checks passed
@OwenKephart
OwenKephart deleted the 01-09-_pac_add_assetcheckstatuscachevalue branch January 26, 2026 21:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants