ATLAS-4922: Atlas Async Import using Kafka #307

DishaTalreja3 · 2025-03-12T18:40:30Z

What changes were proposed in this pull request?

The existing synchronous import API causes performance bottlenecks with large or multiple requests. This patch introduces asynchronous imports using Kafka to queue requests via dedicated topics, enabling users to submit multiple import requests concurrently.

New APIs added to:

Perform asynchronous imports
Check import status (all operations or a specific import ID)
Abort a queued import operation

How was this patch tested?

Manual Testing, Unit Tests

…otification changes

mneethiraj

@DishaTalreja3 - I haven't completed review yet; sending comments so far. Please review and update the patch.

atlas-examples/sample-app/src/main/java/org/apache/atlas/examples/sampleapp/SampleApp.java

client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java

webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java

mneethiraj · 2025-03-13T23:32:59Z

webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java

+                    new SynchronousQueue<>(), // Direct handoff queue
+                    new ThreadFactoryBuilder().setNameFormat(THREADNAME_PREFIX + " thread-%d").build());
+        }
+        executors = executorService;


Review all writes to executors i.e. following methods:

startInternal()

startConsumers()

stop()

Prior to this patch, these methods are to be called only once in an Atlas instance. However, now executors can be overwritten for every call to async import. Review the usage carefully and update to avoid inappropriate overwrites.

In this new approach the executors are overWritten in
startConsumers() - only when the passed in executorService is null (essentially at the start of the instance or start of the first async import request). Please correct me f I am wrong.

…e PR comments

intg/src/main/java/org/apache/atlas/AtlasErrorCode.java

intg/src/main/java/org/apache/atlas/model/impexp/AtlasAsyncImportRequest.java

mneethiraj · 2025-03-14T23:11:44Z

intg/src/main/java/org/apache/atlas/model/impexp/AtlasAsyncImportRequest.java

+@JsonIgnoreProperties(ignoreUnknown = true)
+public class AtlasAsyncImportRequest extends AtlasBaseModelObject implements Serializable {
+    private static final long serialVersionUID = 1L;
+    public static final String ASYNC_IMPORT_TYPE_NAME = "__AtlasAsyncImportRequest";


Constants defined in line 47-49 seem to be used/needed only in Atlas server side. If yes, please move these to server side (like repository module).

These constants are used in this model class as well. Moving them to the repository module might introduce unnecessary dependencies.

None of the classes in model package referenes ASYNC_IMPORT_TYPE_NAME. This is referenced only from AsyncImportService and AtlasAsyncImportRequestDTO.

intg/src/main/java/org/apache/atlas/model/impexp/AtlasAsyncImportRequest.java

intg/src/main/java/org/apache/atlas/model/notification/ImportNotification.java

mneethiraj · 2025-03-14T23:46:44Z

notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java

-        KafkaProducer producer = getOrCreateProducer(topic);
-
-        sendInternalToProducer(producer, topic, messages);
+        sendInternal(topic, messages, SORT_NOT_NEEDED);


ok. I get it now. SORT_NOT_NEEDED doesn't add much to readability, given the called method is right above. I suggest to simply send false here.

notification/src/main/java/org/apache/atlas/notification/rest/RestNotification.java

notification/src/main/java/org/apache/atlas/notification/spool/Spooler.java

notification/src/test/java/org/apache/atlas/notification/spool/AtlasFileSpoolTest.java

…e PR comments

…sting override

…mments and made improvements

intg/src/main/java/org/apache/atlas/model/impexp/AsyncImportStatus.java

mneethiraj · 2025-03-25T00:37:11Z

intg/src/main/java/org/apache/atlas/model/impexp/AtlasAsyncImportRequest.java

+@JsonIgnoreProperties(ignoreUnknown = true)
+public class AtlasAsyncImportRequest extends AtlasBaseModelObject implements Serializable {
+    private static final long serialVersionUID = 1L;
+    public static final String ASYNC_IMPORT_TYPE_NAME = "__AtlasAsyncImportRequest";


None of the classes in model package referenes ASYNC_IMPORT_TYPE_NAME. This is referenced only from AsyncImportService and AtlasAsyncImportRequestDTO.

intg/src/main/java/org/apache/atlas/model/impexp/AtlasAsyncImportRequest.java

intg/src/main/java/org/apache/atlas/model/notification/ImportNotification.java

intg/src/main/java/org/apache/atlas/utils/AtlasAsyncImportTestUtil.java

intg/src/test/java/org/apache/atlas/utils/TestAtlasAsyncImportTestUtil.java

repository/src/main/java/org/apache/atlas/repository/impexp/AsyncImportService.java

repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java

...ory/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/MigrationImport.java

webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java

… the same

…ortTaskListenerImplTest

…ification of consumers and return 404 in case of import id not found

webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java

addons/models/0000-Area0/0010-base_model.json

client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java

intg/src/main/java/org/apache/atlas/AtlasConfiguration.java

repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceWithBackingDirectory.java

repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java

webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java

webapp/src/main/java/org/apache/atlas/notification/ImportTaskListenerImpl.java

…onstants

…an up at the end

client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java

mneethiraj · 2025-05-19T20:48:39Z

@DishaTalreja3 - can you please resolve conversations that you addressed?

Co-authored-by: jackhalfalltrades <[email protected]> (cherry picked from commit f3d302a)

DishaTalreja3 and others added 3 commits February 26, 2025 16:27

ATLAS-4922: Atlas Async Import using Kafka [1] - POJO & Data Access

2e9fba9

ATLAS-4922: Atlas Async Import using Kafka [2] - Client-side changes

3584a20

ATLAS-4922: Atlas Async Import using Kafka [3] - Implementation and N…

16b5449

…otification changes

DishaTalreja3 marked this pull request as ready for review March 12, 2025 18:41

rkundam requested review from rkundam, mneethiraj and pinal-shah March 12, 2025 20:33

ATLAS-4922: Atlas Async Import using Kafka [4] - Unit Tests

d678555

DishaTalreja3 force-pushed the ASYNC_IMPORT branch from a7dabe8 to d678555 Compare March 12, 2025 23:25

mneethiraj reviewed Mar 13, 2025

View reviewed changes

jackhalfalltrades force-pushed the ASYNC_IMPORT branch from 8c56a08 to 5a3a995 Compare March 14, 2025 19:08

ATLAS-4922: Atlas Async Import using Kafka [5] - Addressing few of th…

69ecd1b

…e PR comments

jackhalfalltrades force-pushed the ASYNC_IMPORT branch from 5a3a995 to 69ecd1b Compare March 14, 2025 19:24

mneethiraj reviewed Mar 14, 2025

View reviewed changes

Chandra Kanth Peravelli and others added 3 commits March 17, 2025 14:27

ATLAS-4922: Atlas Async Import using Kafka [6] - Addressing few of th…

6ae366a

…e PR comments

ATLAS-4922: Atlas Async Import using Kafka [7] - Adding support to te…

88ada02

…sting override

ATLAS-4922: Atlas Async Import using Kafka [8] - Addressed some PR co…

0f215e7

…mments and made improvements

DishaTalreja3 force-pushed the ASYNC_IMPORT branch from 7b2eb97 to 0f215e7 Compare March 25, 2025 03:31

addressed review comments

8fe6be6

mneethiraj reviewed Mar 25, 2025

View reviewed changes

addressed review comments

a2c731f

mneethiraj reviewed Mar 25, 2025

View reviewed changes

repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java Show resolved Hide resolved

repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java Show resolved Hide resolved

mneethiraj reviewed Mar 25, 2025

View reviewed changes

...ory/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/MigrationImport.java Outdated Show resolved Hide resolved

webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java Outdated Show resolved Hide resolved

mneethiraj and others added 5 commits March 25, 2025 01:05

addressed review comments

94ff706

addressed review comments

bfc3c15

fixed unit tests

06507ee

code readability improvements

3d66b3f

fixed graph query

ff7c0ec

jackhalfalltrades force-pushed the ASYNC_IMPORT branch from af12b9f to 099b166 Compare March 27, 2025 14:21

ATLAS-4922: Ensuring executors are not overwritten and tests to prove…

83ed723

… the same

jackhalfalltrades force-pushed the ASYNC_IMPORT branch from 099b166 to 83ed723 Compare March 28, 2025 00:57

Chandra Kanth Peravelli and others added 8 commits March 28, 2025 01:56

ATLAS-4922: Fix failing tests

128bf1b

Fix getTopicName()

3a4c62a

ATLAS-4922: Fix flaky tests

6ccc2ae

ATLAS-4922: Fix failing tests

580a0c2

ATLAS-4922: Fix stuck in processing state in case of no topic found

484f27d

ATLAS-4922: Fix enunciate build errors

21d62ab

ATLAS-4922: Fix test failures in NotificationHookConsumerTest and Imp…

a65a5f8

…ortTaskListenerImplTest

ATLAS-4922: Add new urls to notFilterd list for HA fix concurrent mod…

594765e

…ification of consumers and return 404 in case of import id not found

jackhalfalltrades force-pushed the ASYNC_IMPORT branch from cf85053 to 594765e Compare April 23, 2025 19:23

ATLAS-4922: Rename deleteImport to abortImport

666e7f2

mneethiraj reviewed Apr 29, 2025

View reviewed changes

jackhalfalltrades and others added 4 commits May 4, 2025 23:26

Merge branch 'apache:master' into ASYNC_IMPORT

a43629b

ATLAS-4922: PR Comments

854ce59

ATLAS-4922: PR Comments updating the request fields

3e84ea7

ATLAS-4922: Addressed PR comments for Client-side Changes & Renamed C…

73d1ac1

…onstants

DishaTalreja3 force-pushed the ASYNC_IMPORT branch from 822868d to 73d1ac1 Compare May 8, 2025 17:49

ATLAS-4922: Revert to manual commit and handle consumer and topic cle…

431bc90

…an up at the end

mneethiraj approved these changes May 15, 2025

View reviewed changes

client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java Outdated Show resolved Hide resolved

client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java Outdated Show resolved Hide resolved

DishaTalreja3 added 2 commits May 16, 2025 10:00

Renamed ASYNC_IMPORT_STATUS_ALL to ASYNC_IMPORT_STATUS

b141395

ATLAS-4922: Remove getAsyncImportStatus() overload

12fced1

ATLAS-4922: Fix failing tests in ImportServiceTest

2bc64d4

mneethiraj merged commit f3d302a into apache:master May 21, 2025
1 check passed

mneethiraj pushed a commit that referenced this pull request May 22, 2025

ATLAS-4922: Atlas Async Import using Kafka (#307)

d776a87

Co-authored-by: jackhalfalltrades <[email protected]> (cherry picked from commit f3d302a)

ATLAS-4922: Atlas Async Import using Kafka #307

ATLAS-4922: Atlas Async Import using Kafka #307

Conversation

DishaTalreja3 commented Mar 12, 2025

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

mneethiraj left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mneethiraj Mar 13, 2025

Choose a reason for hiding this comment

Uh oh!

jackhalfalltrades Mar 14, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mneethiraj Mar 14, 2025

Choose a reason for hiding this comment

Uh oh!

DishaTalreja3 Mar 20, 2025

Choose a reason for hiding this comment

Uh oh!

mneethiraj Mar 25, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mneethiraj Mar 14, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mneethiraj Mar 25, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mneethiraj commented May 19, 2025

Uh oh!

Uh oh!

Uh oh!