[dagster-airlift][jobs 4/n] Definitions.execute_job_in_process #29243
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Apr 12, 2025
Merged
[dagster-airlift][jobs 1/n] move assets def construction to common code path; other refactors
#29206
Merged
dpeng817
changed the title
This was referenced Apr 15, 2025
dpeng817
force-pushed
the
dpeng817/airflow_defs_data
branch
from
b38fa41 to
c581bfc
Compare
dpeng817
force-pushed
the
defs_execute_in_process
branch
from
a7caf6a to
05fb3ae
Compare
dpeng817
force-pushed
the
dpeng817/airflow_defs_data
branch
from
c581bfc to
c580897
Compare
dpeng817
force-pushed
the
defs_execute_in_process
branch
from
05fb3ae to
bbb6263
Compare
This was referenced Apr 15, 2025
Closed
Closed
dpeng817
force-pushed
the
defs_execute_in_process
branch
from
6595ca7 to
f695d66
Compare
![graphite-app[bot]](https://avatars.githubusercontent.com/in/158384?s=60&v=4){kind=small}
| run_config=run_config, | ||
| ), | ||
| run_id=run_id, | ||
| asset_selection=frozenset(asset_selection), |
There was a problem hiding this comment.
The line asset_selection=frozenset(asset_selection) will raise a TypeError when asset_selection is None, which is a valid input according to the function signature. Since both this method and core_execute_in_process accept asset_selection as an Optional parameter, this should be changed to:
asset_selection=frozenset(asset_selection) if asset_selection else NoneThis matches the pattern used in JobDefinition.get_subset on line 743 and will properly handle the None case.
Suggested change
| asset_selection=frozenset(asset_selection), | |
| asset_selection=frozenset(asset_selection) if asset_selection else None, |
Spotted by Diamond
Is this helpful? React 👍 or 👎 to let us know.
dpeng817
force-pushed
the
dpeng817/airflow_defs_data
branch
from
ae1523b to
86bdbc4
Compare
dpeng817
force-pushed
the
defs_execute_in_process
branch
from
f695d66 to
55dd13b
Compare
dpeng817
force-pushed
the
dpeng817/airflow_defs_data
branch
from
86bdbc4 to
94984b0
Compare
dpeng817
force-pushed
the
defs_execute_in_process
branch
from
55dd13b to
16f8742
Compare
dpeng817
force-pushed
the
dpeng817/airflow_defs_data
branch
from
94984b0 to
5d3a0e4
Compare
dpeng817
force-pushed
the
defs_execute_in_process
branch
from
16f8742 to
997cd2c
Compare
dpeng817
force-pushed
the
dpeng817/airflow_defs_data
branch
from
5d3a0e4 to
f1727c1
Compare
dpeng817
force-pushed
the
defs_execute_in_process
branch
2 times, most recently
from
9a12662 to
62cf80a
Compare
dpeng817
force-pushed
the
dpeng817/airflow_defs_data
branch
from
f1727c1 to
b08c414
Compare
dpeng817
force-pushed
the
defs_execute_in_process
branch
2 times, most recently
from
f3a8535 to
c7c9046
Compare
This was referenced Apr 25, 2025
Closed
dpeng817
force-pushed
the
dpeng817/airflow_defs_data
branch
from
b08c414 to
48f6987
Compare
dpeng817
force-pushed
the
defs_execute_in_process
branch
from
c7c9046 to
d6909d9
Compare
This was referenced Apr 30, 2025
Merged
Merged
dpeng817
force-pushed
the
dpeng817/airflow_defs_data
branch
from
48f6987 to
11bbd68
Compare
dpeng817
force-pushed
the
defs_execute_in_process
branch
from
d6909d9 to
b98c899
Compare
dpeng817
force-pushed
the
dpeng817/airflow_defs_data
branch
from
11bbd68 to
b2cd82e
Compare
dpeng817
force-pushed
the
defs_execute_in_process
branch
from
b98c899 to
167e407
Compare
## Summary & Motivation This PR introduces a new monitoring job for dagstser-airlift that tracks Airflow runs as Dagster runs. The monitoring job polls the Airflow instance for activity and inserts new events and runs into the db. ## How I Tested These Changes - Added comprehensive unit tests for the monitoring job functionality - Live-tested with kitchen sink to ensure that runs were properly picked up.
|
Deploy preview for dagit-core-storybook ready! ✅ Preview Built with commit ad2c505. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary & Motivation
Adds a new internal execute_in_process method to Definitions class which contains full repository context. This ends up being really useful for the "monitoring job", which requires full repo context to build its mapping of items.
How I Tested These Changes
Additional execute_in_process test.