[dagster-airlift][print metadata 2/2] Ingest metadata from airflow logs #29313
+160
−8
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Apr 15, 2025
dpeng817
changed the title
This was referenced Apr 16, 2025
dpeng817
force-pushed
the
metadata_ingest
branch
from
23d4d84 to
11f48ec
Compare
This was referenced Apr 17, 2025
Closed
Closed
dpeng817
force-pushed
the
dpeng817/airlift_log_retrieval
branch
from
30757ef to
ca5040f
Compare
dpeng817
force-pushed
the
metadata_ingest
branch
from
11f48ec to
0074330
Compare
dpeng817
force-pushed
the
dpeng817/airlift_log_retrieval
branch
from
ca5040f to
43f3973
Compare
dpeng817
force-pushed
the
metadata_ingest
branch
from
0074330 to
bc3c990
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary & Motivation
Hooks the metadata APIs up from previous PRs into the monitoring job; so that we actually log the external metadata on the object.
The task logs API is per-task, and so we parallelize using asyncio to avoid creating a huge bottleneck on log retrieval. Remains to be seen whether that's enough, my hunch is that it won't be enough for the largest of instances (or tasks that have massive log streams, which we'll probably have to process in a stream instead of concat-ing to a big string)
How I Tested These Changes
New assertions to existing tests.