LIN-160 - Summary Lineage #4641

akshaysw · 2025-04-30T05:46:15Z

Change description

Connection level lineage
notion doc - https://www.notion.so/atlanhq/Summary-Lineage-eef721c5acfb48f9b5d3b84aa3da0d43

I have added new file LineagePreProcessor - >

Files check bulk API payload for create process request
checks whether the input/output assets belong to different connections.
If they do, Atlas verifies whether a connection lineage already exists:

If a connection lineage exists, no further action is taken.
If no connection lineage exists, Atlas creates one.

similarly delete process call is handled,

Type of change

Bug fix (fixes an issue)
New feature (adds functionality)

Related issues

Fix #1

Helm Config Changes for Running Tests (Staging PR)

Does this PR require Helm config changes for testing?

Tests are NOT required for this commit. (You can proceed with the PR.) ✅
No, Helm config changes are not needed. (You can proceed with the PR.) ✅
Yes, I have already updated the config-values on enpla9up36. (You can proceed with the PR.) ✅
Yes, but I have NOT updated the config-values. (Please update them before proceeding; or, tests will run with default values.)⚠️

Checklists

Development

Lint rules pass locally
Application changes have been tested thoroughly
Automated tests covering modified code pass

Security

Security impact of change has been considered
Code follows company security practices and guidelines

Code review

Pull request has a descriptive title and context useful to a reviewer. Screenshots or screencasts are attached as necessary
"Ready for review" label attached and reviewers assigned
Changes have been reviewed by at least one other contributor
Pull request linked to task tracker where applicable

...ava/org/apache/atlas/repository/store/graph/v2/preprocessor/lineage/LineagePreProcessor.java

repository/src/main/java/org/apache/atlas/discovery/EntityLineageService.java

sriram-atlan · 2025-05-07T09:23:26Z

repository/src/main/java/org/apache/atlas/discovery/EntityLineageService.java

            AtlasVertex datasetVertex = AtlasGraphUtilsV2.findByGuid(this.graph, guid);
            if (direction == AtlasLineageOnDemandInfo.LineageDirection.INPUT || direction == AtlasLineageOnDemandInfo.LineageDirection.BOTH)
-                traverseEdgesOnDemand(datasetVertex, true, depth, level, new HashSet<>(), atlasLineageOnDemandContext, ret, guid, inputEntitiesTraversed, traversalOrder);
+                traverseEdgesOnDemand(datasetVertex, true, depth, level, new HashSet<>(), atlasLineageOnDemandContext, ret, guid, inputEntitiesTraversed, traversalOrder, entityValidationResult);


traverseEdgesOnDemand has 11 parameters now. This is not getting any easy to maintain.

Always make an attempt to make code better than current! cc: @sumandas0 @aarshi0301

...ava/org/apache/atlas/repository/store/graph/v2/preprocessor/lineage/LineagePreProcessor.java

intg/src/main/java/org/apache/atlas/model/lineage/LineageListRequest.java

nikhilbonte21 · 2025-05-27T14:44:34Z

intg/src/main/java/org/apache/atlas/model/lineage/LineageListRequest.java

@@ -35,6 +36,9 @@ public void setImmediateNeighbours(Boolean immediateNeighbours) {
        this.immediateNeighbours = immediateNeighbours;
    }

+
+    private String                         lineageType = LINEAGE_TYPE_DATASET_PROCESS_LINEAGE;


Would lineageType always be LINEAGE_TYPE_DATASET_PROCESS_LINEAGE, I do not see any code to conditionally set it to CONNECTION_PROCESS_LINEAGE.

If so we do not need this field at all

We pass lineageType from payload, if not present in the payload then we assume default value LINEAGE_TYPE_DATASET_PROCESS_LINEAGE

This field can be set from the request payload. If not provided, defaults to LINEAGE_TYPE_DATASET_PROCESS_LINEAGE Supported values: "DatasetProcessLineage" (default), "ConnectionProcessLineage"

nikhilbonte21 · 2025-05-27T14:45:04Z

intg/src/main/java/org/apache/atlas/model/lineage/LineageListRequest.java

@@ -90,6 +94,13 @@ public Integer getDepth() {
    public void setDepth(Integer depth) {
        this.depth = depth;
    }
+    public String getLineageType() {


This will alway return LINEAGE_TYPE_DATASET_PROCESS_LINEAGE, please review need of the getter & field

Updated logic to set based on paload.

nikhilbonte21 · 2025-05-27T14:47:31Z

intg/src/main/java/org/apache/atlas/model/lineage/LineageOnDemandRequest.java

@@ -22,6 +23,7 @@ public class LineageOnDemandRequest {
    private Set<String>                             attributes;
    private Set<String>                             relationAttributes;
    private LineageOnDemandBaseParams               defaultParams;
+    private String                                  lineageType = LINEAGE_TYPE_DATASET_PROCESS_LINEAGE;


Same concern for lineageType, it will be always LINEAGE_TYPE_DATASET_PROCESS_LINEAGE

done above^

nikhilbonte21 · 2025-05-27T14:48:33Z

repository/src/main/java/org/apache/atlas/discovery/AtlasLineageListContext.java

@@ -35,6 +38,7 @@ public AtlasLineageListContext(LineageListRequest lineageListRequest, AtlasTypeR
        this.vertexTraversalPredicate = constructInMemoryPredicate(typeRegistry, lineageListRequest.getEntityTraversalFilters());
        this.edgeTraversalPredicate = constructInMemoryPredicate(typeRegistry, lineageListRequest.getRelationshipTraversalFilters());
        this.attributes = lineageListRequest.getAttributes();
+        this.lineageType = lineageListRequest.getLineageType();


Always set value to LINEAGE_TYPE_DATASET_PROCESS_LINEAGE

updated setter to set different value based on payload.

nikhilbonte21 · 2025-05-27T18:09:08Z

...ava/org/apache/atlas/repository/store/graph/v2/preprocessor/lineage/LineagePreProcessor.java

+            }
+
+            for (String connectionProcessQn : connectionProcessQNs) {
+                if (!checkIfChildProcessExistForConnectionProcess(connectionProcessQn)) {


If checkIfChildProcessExistForConnectionProcess returns GUIDs, you can avoid further getEntityVertex by atlasObjectId which does not use vertex cache, where GUID lookup with cache

nikhilbonte21 · 2025-05-27T18:12:11Z

...ava/org/apache/atlas/repository/store/graph/v2/preprocessor/lineage/LineagePreProcessor.java

+            }
+        } catch (Exception e) {
+            try {
+                String value = vertex.getProperty(PARENT_CONNECTION_PROCESS_QUALIFIED_NAME, String.class);


Again, why are you expecting String PARENT_CONNECTION_PROCESS_QUALIFIED_NAME? I think this is not needed

...ava/org/apache/atlas/repository/store/graph/v2/preprocessor/lineage/LineagePreProcessor.java

server-api/src/main/java/org/apache/atlas/RequestContext.java

akshaysw added 8 commits March 5, 2025 17:49

add changes to support summary lineage

63cf815

handle null exception

5d8e22b

WIP - change 1

ca30400

fix attribute naming

8b113e4

Merge branch 'master' into lin-160-master

8cb9cc8

add try cache to handle process delete failures

09e9d2e

add feature flag isConnectionLineageEnabled

ae5349a

add feature flag modification

bc9afe9

akshaysw added the DO NOT MERGE label Apr 30, 2025

akshaysw added READY FOR REVIEW and removed DO NOT MERGE labels May 7, 2025

sriram-atlan reviewed May 7, 2025

View reviewed changes

...ava/org/apache/atlas/repository/store/graph/v2/preprocessor/lineage/LineagePreProcessor.java Outdated Show resolved Hide resolved

sriram-atlan reviewed May 7, 2025

View reviewed changes

repository/src/main/java/org/apache/atlas/discovery/EntityLineageService.java Outdated Show resolved Hide resolved

sriram-atlan reviewed May 7, 2025

View reviewed changes

repository/src/main/java/org/apache/atlas/discovery/EntityLineageService.java Outdated Show resolved Hide resolved

sriram-atlan reviewed May 7, 2025

View reviewed changes

repository/src/main/java/org/apache/atlas/discovery/EntityLineageService.java Show resolved Hide resolved

sriram-atlan reviewed May 7, 2025

View reviewed changes

...ava/org/apache/atlas/repository/store/graph/v2/preprocessor/lineage/LineagePreProcessor.java Outdated Show resolved Hide resolved

akshaysw added 3 commits May 14, 2025 15:08

address review comments

9f91f87

address review comments - fix bug

2993fff

address review comments - revert change

3323a69

akshaysw requested a review from sriram-atlan May 27, 2025 07:20

nikhilbonte21 requested changes May 27, 2025

View reviewed changes

akshaysw and others added 5 commits May 28, 2025 15:21

remove redundant constant DATASET_PROCESS_LINEAGE

b317c37

add static variable isConnectionLineageEnabled

63051fc

rename LineagePreProcessor to ProcessPreProcessor

220b108

change data type from arraylist to list

7940aaf

addressed PR comments

08915e8

rahul-madaan requested a review from nikhilbonte21 June 12, 2025 11:40

LIN-160 - Summary Lineage #4641

Are you sure you want to change the base?

LIN-160 - Summary Lineage #4641

Uh oh!

Conversation

akshaysw commented Apr 30, 2025

Change description

Type of change

Related issues

Helm Config Changes for Running Tests (Staging PR)

Does this PR require Helm config changes for testing?

Checklists

Development

Security

Code review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!