Skip to content

Commit 3d06c38

Browse files
committed
update document
Signed-off-by: Sotaro Hikita <bering1814@gmail.com>
1 parent 76fcb0a commit 3d06c38

1 file changed

Lines changed: 22 additions & 28 deletions

File tree

USER_GUIDE.md

Lines changed: 22 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -392,6 +392,28 @@ df = spark.read.format("opensearch") \
392392
.load("my-index")
393393
```
394394

395+
### Parallel Reads
396+
397+
Serverless mode automatically splits reads into parallel partitions using PIT + Slice. By default, the connector creates one partition per 50,000 documents in each index.
398+
399+
To customize the partition size, set `opensearch.input.max.docs.per.partition`:
400+
401+
```python
402+
df = spark.read.format("opensearch") \
403+
.option("opensearch.nodes", "https://xxx.us-east-1.aoss.amazonaws.com") \
404+
.option("opensearch.port", "443") \
405+
.option("opensearch.net.ssl", "true") \
406+
.option("opensearch.nodes.wan.only", "true") \
407+
.option("opensearch.aws.sigv4.enabled", "true") \
408+
.option("opensearch.aws.sigv4.region", "us-east-1") \
409+
.option("opensearch.aws.sigv4.service.name", "aoss") \
410+
.option("opensearch.serverless", "true") \
411+
.option("opensearch.input.max.docs.per.partition", "100000") \
412+
.load("my-index")
413+
```
414+
415+
The connector will count the documents in each index, divide by this value to determine the number of slices, and create one Spark partition per slice. For example, an index with 1,000,000 documents and `max.docs.per.partition=100000` will produce 10 parallel read tasks.
416+
395417
## Map/Reduce
396418

397419
For low-level Hadoop Map/Reduce jobs, opensearch-hadoop provides `OpenSearchInputFormat` and `OpenSearchOutputFormat`. Add `opensearch-hadoop-mr-2.0.0.jar` to your job classpath.
@@ -472,31 +494,3 @@ TBLPROPERTIES(
472494
'opensearch.aws.sigv4.region' = 'us-east-1');
473495
```
474496

475-
## Amazon OpenSearch Serverless
476-
477-
The connector supports Amazon OpenSearch Serverless (both Classic and NextGen architectures). Serverless collections do not expose shard information, so the connector uses PIT (Point in Time) with search_after for pagination instead of the scroll API.
478-
479-
### Basic Configuration
480-
481-
```
482-
opensearch.serverless=true
483-
opensearch.nodes=https://<collection-id>.aoss.<region>.on.aws
484-
opensearch.port=443
485-
opensearch.net.ssl=true
486-
opensearch.nodes.wan.only=true
487-
opensearch.aws.sigv4.enabled=true
488-
opensearch.aws.sigv4.region=<region>
489-
opensearch.aws.sigv4.service.name=aoss
490-
```
491-
492-
### Parallel Reads
493-
494-
Serverless mode automatically splits reads into parallel partitions using PIT + Slice. By default, the connector creates one partition per 50,000 documents in each index.
495-
496-
To customize the partition size, set `opensearch.input.max.docs.per.partition`:
497-
498-
```
499-
opensearch.input.max.docs.per.partition=100000
500-
```
501-
502-
The connector will count the documents in each index, divide by this value to determine the number of slices, and create one Spark partition per slice. For example, an index with 1,000,000 documents and `max.docs.per.partition=100000` will produce 10 parallel read tasks.

0 commit comments

Comments
 (0)