Skip to content

Commit 30d5558

Browse files
ayush-shahayush-shah
committed
docs: clarify pipeline and lineage behavior
1 parent a1cce58 commit 30d5558

2 files changed

Lines changed: 108 additions & 3 deletions

File tree

‎analytics-assets/README.md‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,29 @@ OpenMetadata 2.0 RC1 does not expose public service facades for the ML model,
1313
pipeline, or search services. Create those services first and supply their
1414
fully qualified names. Supply an existing database service for the query.
1515

16+
## Pipeline Service Versus Pipeline Asset
17+
18+
A Custom Pipeline **service** is the parent connection; it is not itself a
19+
Pipeline asset. For example:
20+
21+
```text
22+
Service FQN: custom-pipeline-m1
23+
Pipeline FQN: custom-pipeline-m1.my_daily_etl
24+
```
25+
26+
Creating only `custom-pipeline-m1` does not add that service—or a Pipeline
27+
beneath it—to `Pipelines.list()` or `Pipelines.list_all()`. A connector or
28+
`Pipelines.create()` call must create `my_daily_etl` beneath that service. This
29+
scenario demonstrates that second step using the service named by
30+
`OPENMETADATA_PIPELINE_SERVICE_FQN`.
31+
32+
`Pipelines.list()` returns one page, whose assets are in `.entities`;
33+
`Pipelines.list_all()` follows pagination and returns every Pipeline asset.
34+
Neither method lists Pipeline services. See the pinned RC1
35+
[`Pipelines` facade](https://github.com/open-metadata/OpenMetadata/blob/8e199f486b63df9f44d4c827a9f43559d9e56e07/ingestion/src/metadata/sdk/entities/pipelines.py)
36+
and
37+
[`list()`/`list_all()` implementation](https://github.com/open-metadata/OpenMetadata/blob/8e199f486b63df9f44d4c827a9f43559d9e56e07/ingestion/src/metadata/sdk/entities/base.py#L275-L333).
38+
1639
## Run
1740

1841
```bash

‎api-lineage-cicd/README.md‎

Lines changed: 85 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -69,11 +69,92 @@ Expected output (column-mapping lines omitted):
6969
Lineage: existing-orders-service.orders.getOrder → existing-fulfillment-service.shipments.getShipment
7070
7171
4. Verifying lineage...
72-
Verified 2 lineage edges
72+
Verified 2 lineage edges and 5 column mappings
7373
7474
Done ✅
7575
```
7676

77+
## A Pipeline Asset Is Optional
78+
79+
OpenMetadata lineage connects data assets directly. A Pipeline asset is not
80+
required between an API endpoint and a table—or between the two API endpoints
81+
in this example. `LineageDetails.pipeline` is optional context for identifying
82+
the orchestrated job that performs the movement.
83+
84+
Inside an SDK session already initialized with `configure()`, this direct API
85+
Endpoint → Table request is valid without a Pipeline:
86+
87+
```python
88+
from metadata.generated.schema.api.lineage.addLineage import AddLineageRequest
89+
from metadata.generated.schema.type.basic import FullyQualifiedEntityName
90+
from metadata.generated.schema.type.entityLineage import (
91+
ColumnLineage,
92+
EntitiesEdge,
93+
LineageDetails,
94+
)
95+
from metadata.generated.schema.type.entityReference import EntityReference
96+
from metadata.sdk import APIEndpoints, Tables
97+
from metadata.sdk.api.lineage import Lineage
98+
99+
endpoint = APIEndpoints.retrieve_by_name("orders-api.orders.createOrder")
100+
table = Tables.retrieve_by_name("warehouse.sales.public.orders")
101+
102+
Lineage.add_lineage_request(
103+
AddLineageRequest(
104+
edge=EntitiesEdge(
105+
fromEntity=EntityReference(id=endpoint.id, type="apiEndpoint"),
106+
toEntity=EntityReference(id=table.id, type="table"),
107+
lineageDetails=LineageDetails(
108+
columnsLineage=[
109+
ColumnLineage(
110+
fromColumns=[
111+
FullyQualifiedEntityName(
112+
"orders-api.orders.createOrder.responseSchema.order_id"
113+
)
114+
],
115+
toColumn=FullyQualifiedEntityName("warehouse.sales.public.orders.order_id"),
116+
)
117+
]
118+
),
119+
)
120+
)
121+
)
122+
```
123+
124+
The pinned RC1 schema defines the
125+
[`pipeline` reference as optional](https://github.com/open-metadata/OpenMetadata/blob/8e199f486b63df9f44d4c827a9f43559d9e56e07/openmetadata-spec/src/main/resources/json/schema/type/entityLineage.json#L45-L64),
126+
and its server implementation explicitly
127+
[`validates API endpoint request and response fields`](https://github.com/open-metadata/OpenMetadata/blob/8e199f486b63df9f44d4c827a9f43559d9e56e07/openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/LineageRepository.java#L1207-L1229)
128+
for column lineage. The generic edge-schema description still says lineage
129+
details are for table-to-table edges; this example follows the executable RC1
130+
server behavior for API endpoints.
131+
132+
## Troubleshooting Empty `columnsLineage`
133+
134+
RC1 keeps a valid entity-level edge but filters field mappings whose FQNs do
135+
not belong to that edge. This can leave the lineage visible in the UI while the
136+
matching edge returns `columnsLineage: []`.
137+
138+
Use the exact `fullyQualifiedName` values returned by OpenMetadata:
139+
140+
```text
141+
API response field: service.collection.endpoint.responseSchema.field
142+
API request field: service.collection.endpoint.requestSchema.field
143+
Table column: service.database.schema.table.column
144+
```
145+
146+
Do not rebuild these names from display names. Retrieve the endpoint with
147+
`requestSchema,responseSchema`, retrieve the table with `columns`, and copy the
148+
field and column FQNs from those responses. Confirm that the source fields
149+
belong to the edge's upstream entity and the destination field belongs to its
150+
downstream entity.
151+
152+
The RC1 server
153+
[`filters invalid source and destination fields`](https://github.com/open-metadata/OpenMetadata/blob/8e199f486b63df9f44d4c827a9f43559d9e56e07/openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/LineageRepository.java#L718-L750)
154+
instead of rejecting the whole edge. This example therefore reads the edge
155+
back and compares all requested mappings. It fails with `Column lineage
156+
mismatch` rather than reporting a partial write as successful.
157+
77158
## Idempotency and Overwrite Behavior
78159

79160
The script uses public `APICollections.create()` and `APIEndpoints.create()`
@@ -109,8 +190,9 @@ Choose the approach that fits your workflow:
109190
| `update()` | PATCH | Partial diff — only changed fields are sent | You want to coexist with manual UI edits |
110191

111192
The script reads lineage back after writing it and fails if either requested
112-
edge is missing. It does not delete services, collections, endpoints, or lineage.
113-
Credentials are read only from the environment and are never printed.
193+
edge or any requested column mapping is missing. It does not delete services,
194+
collections, endpoints, or lineage. Credentials are read only from the
195+
environment and are never printed.
114196

115197
## CI/CD Integration
116198

0 commit comments

Comments
 (0)