Commit e6a4362
feat(lineage): drop lineage_to_openlineage, add lineage_to_openmetadata (v0.10.20)
lineage_to_openlineage DELETED:
- Was emitting a single OpenLineage RunEvent with the entire asset graph
stuffed into inputs[] and an empty outputs[] — not proper OL lineage
shape (OL expects per-execution events with directed input→output
relationships). Receiving backends like Marquez couldn't reconstruct
lineage from the dump.
- Real proper per-materialization OL emission is what dagster-openlineage
(the new official Dagster library) does. Customers wanting OL should
use that.
- Our snapshot-shaped sink family covers governance catalog sync — a
different layer than OL run events.
lineage_to_openmetadata NEW:
- Mirrors the lineage_to_datahub / _alation / _collibra / _purview /
_data360 / _file / _webhook family shape.
- OpenMetadata's REST API hierarchy is service → database → schema →
table, so we map Dagster's asset graph onto it:
- DatabaseService = "dagster" (configurable)
- Database = "default" (configurable)
- DatabaseSchema = one per Dagster `group`
- Table = one per Dagster asset (columns from asset metadata; falls
back to one synthetic _value column when no metadata)
- Lineage edges = one PUT /api/v1/lineage per asset-graph edge
- All PUTs are upsert/idempotent; only_push_on_change short-circuits the
network call when payload_hash matches the last successful push.
- DatabaseService type is set to 'Mysql' because OM requires a concrete
service type from its enum — cosmetic for the asset-graph use case.
Manifest: 836 → 835 (drop OL) → 836 (add OM).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>1 parent bda675b commit e6a4362
13 files changed
Lines changed: 538 additions & 396 deletions
File tree
- assets/sinks
- lineage_to_openlineage
- lineage_to_openmetadata
This file was deleted.
This file was deleted.
This file was deleted.
This file was deleted.
This file was deleted.
This file was deleted.
0 commit comments