Move pipes external metadata processing to its own layer.

[dpeng817]

Summary & Motivation

Pipes and airlift have kinda the same mandate here to use a well-structured format for defining metadata in an external process, that may not have access to the dagster package. So we can just move the processing code to a shared place accessible to both, and make the language more generic to instead be "external" metadata.

How I Tested These Changes

Existing pipes metadata tests

[dpeng817]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• [historical-runs 1/n][dagster-airlift] make monitor job range configurable #29339 : 2 dependent PRs (#29371 , #29381 )
• [dagster-airlift][components 2/n] scaffolder #29333
• [dagster-airlift][components 1/n] instance component #29331
• [dagster-airlift][print metadata 2/2] Ingest metadata from airflow logs #29313
• [dagster-airlift][print metadata 1/2] log retrieval API #29312
• Move pipes external metadata processing to its own layer. #29311 👈 (View in Graphite)
• [dagster-airlift][jobs 6/n] jobs UI #29278
• [dagster-airlift][jobs 5/n][RFC] Airflow monitoring job #29277
• [dagster-airlift][jobs 4/n] Definitions.execute_job_in_process #29243 : 1 other dependent PR (#29209 )
• [dagster-airlift][jobs 3/n] make AirflowDefinitionsData aware of jobs #29208
• [dagster-airlift][jobs 2/n] Job builder methods #29207
• [dagster-airlift][jobs 1/n] move assets def construction to common code path; other refactors #29206
• [dagster-airlift][datasets 3/3] dataset filtering #29164
• [dagster-airlift][datasets 2/3] auto-construct assets from datasets #29162
• [dagster-airlift][datasets 1/3] datasets storage layer, AirflowInstance retrieval methods #29155
• [dagster-airlift] Move around classes #29154
• [dagster-airlift] fix prop on AirflowDefinitionsData #29161
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.