Skip to content

Commit e6a4362

Browse files
feat(lineage): drop lineage_to_openlineage, add lineage_to_openmetadata (v0.10.20)
lineage_to_openlineage DELETED: - Was emitting a single OpenLineage RunEvent with the entire asset graph stuffed into inputs[] and an empty outputs[] — not proper OL lineage shape (OL expects per-execution events with directed input→output relationships). Receiving backends like Marquez couldn't reconstruct lineage from the dump. - Real proper per-materialization OL emission is what dagster-openlineage (the new official Dagster library) does. Customers wanting OL should use that. - Our snapshot-shaped sink family covers governance catalog sync — a different layer than OL run events. lineage_to_openmetadata NEW: - Mirrors the lineage_to_datahub / _alation / _collibra / _purview / _data360 / _file / _webhook family shape. - OpenMetadata's REST API hierarchy is service → database → schema → table, so we map Dagster's asset graph onto it: - DatabaseService = "dagster" (configurable) - Database = "default" (configurable) - DatabaseSchema = one per Dagster `group` - Table = one per Dagster asset (columns from asset metadata; falls back to one synthetic _value column when no metadata) - Lineage edges = one PUT /api/v1/lineage per asset-graph edge - All PUTs are upsert/idempotent; only_push_on_change short-circuits the network call when payload_hash matches the last successful push. - DatabaseService type is set to 'Mysql' because OM requires a concrete service type from its enum — cosmetic for the asset-graph use case. Manifest: 836 → 835 (drop OL) → 836 (add OM). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent bda675b commit e6a4362

13 files changed

Lines changed: 538 additions & 396 deletions

File tree

assets/sinks/lineage_to_openlineage/README.md

Lines changed: 0 additions & 88 deletions
This file was deleted.

assets/sinks/lineage_to_openlineage/__init__.py

Lines changed: 0 additions & 3 deletions
This file was deleted.

assets/sinks/lineage_to_openlineage/component.py

Lines changed: 0 additions & 257 deletions
This file was deleted.

assets/sinks/lineage_to_openlineage/example.yaml

Lines changed: 0 additions & 7 deletions
This file was deleted.

assets/sinks/lineage_to_openlineage/requirements.txt

Lines changed: 0 additions & 2 deletions
This file was deleted.

assets/sinks/lineage_to_openlineage/schema.json

Lines changed: 0 additions & 22 deletions
This file was deleted.

0 commit comments

Comments
 (0)