[dagster-airlift][migration 1/n] explicit methods for external jobs, use them in airlift #29706
+108
−91
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Apr 30, 2025
Closed
This stack of pull requests is managed by Graphite. Learn more about stacking. |
This was referenced Apr 30, 2025
Merged
Merged
dpeng817
changed the title
dpeng817
force-pushed
the
dpeng817/apply_storage_changes_to_airlift
branch
from
0bd90f6 to
be28681
Compare
dpeng817
force-pushed
the
external_job_handling
branch
from
e8e47bf to
bca05aa
Compare
dpeng817
force-pushed
the
dpeng817/apply_storage_changes_to_airlift
branch
from
be28681 to
16630c1
Compare
dpeng817
force-pushed
the
external_job_handling
branch
from
bca05aa to
0e5354a
Compare
Base automatically changed from
dpeng817/apply_storage_changes_to_airlift
to
dpeng817/timestamp_event_reporting_override
May 1, 2025 00:09
Base automatically changed from
dpeng817/timestamp_event_reporting_override
to
dpeng817/runs_obey_event_timestamp
May 1, 2025 00:09
dpeng817
force-pushed
the
dpeng817/runs_obey_event_timestamp
branch
from
126d4fd to
cbc50ed
Compare
dpeng817
force-pushed
the
external_job_handling
branch
from
0e5354a to
388a0b1
Compare
Base automatically changed from
dpeng817/runs_obey_event_timestamp
to
dpeng817/add_history_import_run_method
May 1, 2025 00:24
Base automatically changed from
dpeng817/add_history_import_run_method
to
dpeng817/configurable_monitor_job
May 1, 2025 00:27
Base automatically changed from
dpeng817/configurable_monitor_job
to
dpeng817/airlift_instance_scaffolder
May 1, 2025 00:27
Base automatically changed from
dpeng817/airlift_instance_scaffolder
to
airlift_instance_component
May 1, 2025 00:27
Base automatically changed from
dpeng817/airlift_log_retrieval
to
dpeng817/airlift_external_metadata
May 1, 2025 00:28
Base automatically changed from
dpeng817/airlift_external_metadata
to
airlift_jobs_ui
May 1, 2025 00:28
Base automatically changed from
airlift_jobs_ui
to
dpeng817/dpeng817/monitoring_job_real
May 1, 2025 00:32
dpeng817
force-pushed
the
external_job_handling
branch
from
388a0b1 to
e5d08a4
Compare
This was referenced May 1, 2025
Merged
[dagster-airlift][jobs 1/n] move assets def construction to common code path; other refactors
#29206
Merged
Closed
Closed
dpeng817
force-pushed
the
external_job_handling
branch
from
e5d08a4 to
1c0932b
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR has been used to merge the large set of upstack changes which comprise a P0 implementation of jobs-based airlift.
Summary & Motivation
Adds explicit methods for constructing "external jobs". These jobs have an asset layer with only asset keys defined, but can still be properly linked back/forth from the asset by the code location.
The upshot is that we're able to create job definitions for assets which are materializable outside of a specific job - because there's no direct linkage between the asset and an op within the job. Previously we could only accept asset specs in the job creation code path.
How I Tested These Changes
Added a new test, tested live in dagster dev.