discovery: OpenLineage emission from the Go control plane → OpenMetadata integration

[neochaotic]

Context

There is no data-lineage / catalog integration today (no OpenMetadata, no OpenLineage emission) — verified: zero references in code, docs, or ADRs. The only OpenLineage mention in the repo is a transitive import inside Airflow provider hooks that the compat shim must stub (openlineage.sqlparser, see project/planning/airflow-connector-compatibility.md), not a functioning integration.

Crucially, this does not come for free from the Airflow SDK: Airflow's OpenMetadata lineage is scheduler-side — the [lineage] backend in airflow.cfg fired by Airflow's scheduler, or the OpenMetadata Airflow Managed APIs plugin. Leoflow runs a Go control plane (no Airflow scheduler/webserver/airflow.cfg) with pod-per-task SDK execution, so that hook never fires. Any lineage/catalog integration is net-new, deliberate work.

With the enterprise/governance direction, cataloging + lineage is a plausible headline capability worth scoping.

Proposal (to evaluate)

Emit OpenLineage events from the leoflow control plane rather than binding to one catalog. OpenLineage is vendor-neutral and ingested by OpenMetadata, Marquez, and DataHub alike, so "OpenMetadata integration" becomes "point leoflow's OpenLineage endpoint at OpenMetadata's OL ingestion."

Open questions for the discovery

1. Emit site. Where in the Go control plane do RUN/JOB lifecycle events originate — scheduler/dispatcher transitions (start/complete/fail per task instance)? What is the cleanest hook that already sees the full run + task graph?
2. Facets. Which OpenLineage facets to populate first — run, job, and (the hard/most valuable one) dataset input/output facets for real column/table lineage. What can we derive without user annotation?
3. dbt is the natural first win. Leoflow already renders a dbt project into node-level tasks (ADR 0042/0043); dbt's own manifest.json/run_results.json carry model-level lineage. Emitting OL for dbt nodes may be the highest-value, lowest-effort slice — scope it separately.
4. Transport + config. OpenLineage HTTP transport (URL + API key) as opt-in server config; default off. How does it interact with the existing audit/event-log surface (reuse vs separate)?
5. Operators/sensors. For non-dbt tasks, how much lineage can we extract (the provider hooks already pull openlineage.sqlparser transitively — is there a path to reuse it, or is that out of scope for v1)?
6. Positioning. Is a generic OpenLineage emitter enough, or is a first-class "OpenMetadata connector" (with OM-specific setup docs) the enterprise ask?

Deliverable

An ADR proposing (or declining) OpenLineage emission — emit site, facet scope for a v1, the dbt-first slice, config surface — plus a rough effort estimate. Not committing to build; scoping the decision.

Not in scope here

Re-implementing Airflow's [lineage] backend. Leoflow's answer should be OpenLineage-native from the Go side, not an airflow.cfg mechanism it doesn't run.