[Feature][connector-elasticsearch] elasticsearch source support PIT #9150

CosmosNi · 2025-04-11T03:43:47Z

elasticsearch source support PIT

close #9149

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update plugin-mapping.properties and add new connector information in it
2. Update the pom file of seatunnel-dist
3. Add ci label in label-scope-conf
4. Add e2e testcase in seatunnel-e2e
5. Update connector plugin_config
Update the release-note.

Copilot

Copilot reviewed 11 out of 12 changed files in this pull request and generated 1 comment.

Files not reviewed (1)

seatunnel-e2e/seatunnel-connector-v2-e2e/connector-elasticsearch-e2e/src/test/resources/elasticsearch/elasticsearch_source_with_pit.conf: Language not supported

Copilot · 2025-04-11T12:25:41Z

...c/main/java/org/apache/seatunnel/connectors/seatunnel/elasticsearch/client/EsRestClient.java

+        Map<String, String> sortField = new HashMap<>();
+        sortField.put("order", "asc");
+        sort.add(Collections.singletonMap("_shard_doc", "asc"));


The variable 'sortField' is assigned but never used; consider removing it to clean up the code.

Suggested change

Map<String, String> sortField = new HashMap<>();

sortField.put("order", "asc");

sort.add(Collections.singletonMap("_shard_doc", "asc"));

hailin0 · 2025-04-11T12:26:29Z

Do you have more information on PIT vs scroll?

CosmosNi · 2025-04-11T12:46:50Z

Do you have more information on PIT vs scroll?
@hailin0
Here are the advantages of PIT compared with Scroll API:

Better data consistency: PIT creates a consistent data view, ensuring accurate results even when the index changes during pagination.
Lower resource consumption: It uses snapshots to save search results, only storing document changes after the write point, thus reducing memory consumption.
More flexible usage: Combined with Search After, PIT allows for more stable pagination and is less affected by the number of retrieval requests.

Hisoka-X · 2025-04-12T02:48:35Z

...g/apache/seatunnel/connectors/seatunnel/elasticsearch/config/ElasticsearchSourceOptions.java

@@ -87,4 +88,25 @@ public class ElasticsearchSourceOptions extends ElasticsearchBaseOptions {
                            Collections.singletonMap("match_all", new HashMap<String, String>()))
                    .withDescription(
                            "Elasticsearch query language. You can control the range of data read");
+
+    public static final Option<Boolean> USE_PIT =


How about add new enum type named search_api_type? Support scorll and pit and set scroll by default.

hailin0 · 2025-04-12T06:55:53Z

Do you have more information on PIT vs scroll?
@hailin0
Here are the advantages of PIT compared with Scroll API:

Better data consistency: PIT creates a consistent data view, ensuring accurate results even when the index changes during pagination.

Lower resource consumption: It uses snapshots to save search results, only storing document changes after the write point, thus reducing memory consumption.

More flexible usage: Combined with Search After, PIT allows for more stable pagination and is less affected by the number of retrieval requests.

Which versions of elasticsearch can be used?

CosmosNi · 2025-04-12T07:48:40Z

Do you have more information on PIT vs scroll?
@hailin0
Here are the advantages of PIT compared with Scroll API:

Better data consistency: PIT creates a consistent data view, ensuring accurate results even when the index changes during pagination.

Lower resource consumption: It uses snapshots to save search results, only storing document changes after the write point, thus reducing memory consumption.

More flexible usage: Combined with Search After, PIT allows for more stable pagination and is less affected by the number of retrieval requests.

Which versions of elasticsearch can be used?

7.10

Hisoka-X · 2025-04-14T02:28:48Z

docs/en/connector-v2/source/Elasticsearch.md

@@ -30,7 +30,7 @@ support version >= 2.x and <= 8.x.
 | index_list              | array   | no       | used to define a multiple table task                           |
 | source                  | array   | no       | -                                                              |
 | query                   | json    | no       | {"match_all": {}}                                              |
-| search_type             | json    | no       | Search method,sql or dsl,default dsl                           |
+| search_type             | json    | no       | Search method,SQL、PIT、SCROLL,default SCROLL                    |


search_type and search_api_type not same.
search_type : SQL, DSL
search_api_type: SCROLL, PIT

Hisoka-X · 2025-04-15T02:18:04Z

...c/main/java/org/apache/seatunnel/connectors/seatunnel/elasticsearch/client/EsRestClient.java

+        Map<String, String> sortField = new HashMap<>();
+        sortField.put("order", "asc");
+        sort.add(Collections.singletonMap("_shard_doc", "asc"));


Hisoka-X · 2025-04-15T02:24:48Z

...g/apache/seatunnel/connectors/seatunnel/elasticsearch/source/ElasticsearchSourceFactory.java

+                        PIT_KEEP_ALIVE,
+                        PIT_BATCH_SIZE,


Suggested change

PIT_KEEP_ALIVE,

PIT_BATCH_SIZE,

PIT_KEEP_ALIVE,

PIT_BATCH_SIZE,

SEARCH_API_TYPE,

Hisoka-X · 2025-04-15T02:25:14Z

...rg/apache/seatunnel/connectors/seatunnel/elasticsearch/source/ElasticsearchSourceReader.java

+        }
+        // DSL query
+        else {
+            // Check if we should use PIT API


Suggested change

}

// DSL query

else {

// Check if we should use PIT API

} else {

// Check if we should use PIT API

Hisoka-X · 2025-04-15T02:25:38Z

...rg/apache/seatunnel/connectors/seatunnel/elasticsearch/source/ElasticsearchSourceReader.java

+            }
+            // Default scroll API
+            else {


Suggested change

}

// Default scroll API

else {

} else {

…h_source_PIT

Hisoka-X

Thanks @CosmosNi . LGTM if ci passes.

[Feature][connector-elasticsearch] elasticsearch source support PIT

4050dcd

github-actions bot added connectors-v2 e2e elasticsearch labels Apr 11, 2025

[Feature][connector-elasticsearch] elasticsearch source support PIT

7d34224

github-actions bot added the document label Apr 11, 2025

hailin0 requested a review from Copilot April 11, 2025 12:25

Copilot AI reviewed Apr 11, 2025

View reviewed changes

Hisoka-X reviewed Apr 12, 2025

View reviewed changes

[Feature][connector-elasticsearch] elasticsearch source support PIT

25b647f

Hisoka-X reviewed Apr 14, 2025

View reviewed changes

CosmosNi added 2 commits April 14, 2025 11:45

[Feature][connector-elasticsearch] elasticsearch source support PIT

1bb296f

[Feature][connector-elasticsearch] elasticsearch source support PIT

3969e90

CosmosNi requested a review from Hisoka-X April 14, 2025 06:01

Hisoka-X reviewed Apr 15, 2025

View reviewed changes

CosmosNi added 2 commits April 15, 2025 11:08

[Feature][connector-elasticsearch] elasticsearch source support PIT

856cd10

Merge remote-tracking branch 'upstream/dev' into feature_elasticsearc…

9e05e92

…h_source_PIT

Hisoka-X approved these changes Apr 15, 2025

View reviewed changes

github-actions bot added approved reviewed labels Apr 15, 2025

hailin0 approved these changes Apr 16, 2025

View reviewed changes

hailin0 merged commit 948d588 into apache:dev Apr 16, 2025
4 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature][connector-elasticsearch] elasticsearch source support PIT #9150

[Feature][connector-elasticsearch] elasticsearch source support PIT #9150

CosmosNi commented Apr 11, 2025

Copilot AI left a comment

Copilot AI Apr 11, 2025

Hisoka-X Apr 15, 2025

hailin0 commented Apr 11, 2025

CosmosNi commented Apr 11, 2025

Hisoka-X Apr 12, 2025

CosmosNi Apr 12, 2025

hailin0 commented Apr 12, 2025

CosmosNi commented Apr 12, 2025

Hisoka-X Apr 14, 2025

Hisoka-X Apr 15, 2025

Hisoka-X Apr 15, 2025

Hisoka-X Apr 15, 2025

Hisoka-X Apr 15, 2025

Hisoka-X left a comment

	Map<String, String> sortField = new HashMap<>();
	sortField.put("order", "asc");
	sort.add(Collections.singletonMap("_shard_doc", "asc"));

[Feature][connector-elasticsearch] elasticsearch source support PIT #9150

[Feature][connector-elasticsearch] elasticsearch source support PIT #9150

Conversation

CosmosNi commented Apr 11, 2025

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot AI Apr 11, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hailin0 commented Apr 11, 2025

CosmosNi commented Apr 11, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hailin0 commented Apr 12, 2025

CosmosNi commented Apr 12, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Hisoka-X left a comment

Choose a reason for hiding this comment