You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Serverless mode automatically splits reads into parallel partitions using PIT + Slice. By default, the connector creates one partition per 50,000 documents in each index.
398
+
399
+
To customize the partition size, set `opensearch.input.max.docs.per.partition`:
The connector will count the documents in each index, divide by this value to determine the number of slices, and create one Spark partition per slice. For example, an index with 1,000,000 documents and `max.docs.per.partition=100000` will produce 10 parallel read tasks.
416
+
395
417
## Map/Reduce
396
418
397
419
For low-level Hadoop Map/Reduce jobs, opensearch-hadoop provides `OpenSearchInputFormat` and `OpenSearchOutputFormat`. Add `opensearch-hadoop-mr-2.0.0.jar` to your job classpath.
@@ -472,31 +494,3 @@ TBLPROPERTIES(
472
494
'opensearch.aws.sigv4.region'='us-east-1');
473
495
```
474
496
475
-
## Amazon OpenSearch Serverless
476
-
477
-
The connector supports Amazon OpenSearch Serverless (both Classic and NextGen architectures). Serverless collections do not expose shard information, so the connector uses PIT (Point in Time) with search_after for pagination instead of the scroll API.
Serverless mode automatically splits reads into parallel partitions using PIT + Slice. By default, the connector creates one partition per 50,000 documents in each index.
495
-
496
-
To customize the partition size, set `opensearch.input.max.docs.per.partition`:
497
-
498
-
```
499
-
opensearch.input.max.docs.per.partition=100000
500
-
```
501
-
502
-
The connector will count the documents in each index, divide by this value to determine the number of slices, and create one Spark partition per slice. For example, an index with 1,000,000 documents and `max.docs.per.partition=100000` will produce 10 parallel read tasks.
0 commit comments