Job-to-job lineage #2484
Description
Description of the Use Case
Currently, Marquez displays the parent jobs and their child jobs independently with no lineage connection. For exemple, a Spark job initiated in an Airflow DAG is not visually linked in the UI to its parent, although the information is present in the ParentRunFacet.
The Issue
The LineageAPI doesn’t currently allow for job-to-job lineage queries. To implement that feature, a backend representation of the job-to-job link must be designed first.
Disclamer: this is the result of a Slack discussion with @wslulciuc