Add support for Amazon OpenSearch Serverless

[lawofcycles]

Description

Add support for Amazon OpenSearch Serverless.

When opensearch.serverless=true is set, opensearch-hadoop works in serverless mode, connecting to serverless endpoints and calling APIs based on serverless specs.

How it works

Write path: Adapted to Serverless APIs. Cluster health checks, refresh calls, and node discovery are skipped since these endpoints are not available on Serverless. ClusterInfo is synthesized locally without calling GET /.

Read path: Uses PIT (Point in Time) + Search After API for pagination. Scroll API and Slice API are not available on Serverless, so all reads go through a single partition per index. PIT ensures snapshot consistency during reads, and _id is used as a sort tiebreaker to prevent missing documents.

Serverless API	Status	Notes
POST /_search/scroll	Not supported	Returns 404
POST /<index>/_search/point_in_time	Supported	Used for consistent reads
slice parameter	Not supported	Returns 500
search_after + pit	Supported	Primary read pagination method

Configuration

Setting	Default	Description
opensearch.serverless	false	Enable serverless mode
opensearch.pit.keepalive	5m	PIT keep alive duration for serverless reads
opensearch.nodes.wan.only	Must be true	Required for serverless

Example

Write

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

df = spark.createDataFrame([
  (1, "text1"),
  (2, "text2")
], ["id", "text"])

df.write \
  .format("org.opensearch.spark.sql") \
  .option("opensearch.nodes", "<opensearch serverless endpoint>") \
  .option("opensearch.port", "443") \
  .option("opensearch.resource", "<index name>") \
  .option("opensearch.aws.sigv4.enabled", "true") \
  .option("opensearch.aws.sigv4.region", "<region name>") \
  .option("opensearch.nodes.wan.only", "true") \
  .option("opensearch.serverless", "true") \
  .mode("append") \
  .save()

Read

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

df = spark.read \
  .format("org.opensearch.spark.sql") \
  .option("opensearch.nodes", "<opensearch serverless endpoint>") \
  .option("opensearch.port", "443") \
  .option("opensearch.resource", "<index name>") \
  .option("opensearch.aws.sigv4.enabled", "true") \
  .option("opensearch.aws.sigv4.region", "<region name>") \
  .option("opensearch.nodes.wan.only", "true") \
  .option("opensearch.serverless", "true") \
  .load()

df.show()

Future consideration

The current Search After implementation reuses ScrollReader and ScrollQuery with a serverless mode flag. Ideally, the pagination logic (ScrollQuery) should be split into separate ScrollPaginationQuery and SearchAfterPaginationQuery implementations, with the common hit parsing logic in ScrollReader extracted into a shared base. However, ScrollReader.read() contains complex parsing, raw JSON reconstruction, and metadata extraction that accounts for the majority of the code, while the Scroll/SearchAfter divergence is limited to response header handling and a few lines of state management. Restructuring this carries regression risk disproportionate to the benefit, so this refactoring is left for a future PR.

Testing

Tested on EMR 7.12 with Spark, Hive, and MapReduce against OpenSearch Serverless. Also verified that the standard (non serverless) read/write path is unaffected.

Co-authored-by: @dimorportheca47

Issues Resolved

fix #269

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[lawofcycles]

I have conducted the compatibility test for all OpenSearch APIs called from OpenSearch Hadoop.
Some of them are not supported on Serverless and we need further implementation to fill those gaps. I'm working on them.

API	Status	Implementation
Cluster Management
GET /	Not supported -> Handled	Returns dummy ClusterInfo (name="serverless-collection", version=V_2_X)
GET /_nodes/http	Not supported -> Skipped	Disabled via opensearch.nodes.discovery=false
GET /_cluster/health/{index}	Not supported -> Skipped	Returns immediately in serverless mode
Search
GET/POST /{index}/_search	Supported	Works as-is
POST /_search/scroll	Not supported (404)	Needs Search After API as replacement (see below)
DELETE /_search/scroll	Not supported -> Ignored	Scroll context expires automatically via timeout
GET /{index}/_search_shards	Not supported -> Handled	Creates single partition per index via findServerlessPartitions()
POST /{index}/_search with slice	Not supported (500)	No parallel read support on Serverless
POST /{index}/_search with search_after	Supported	Proposed replacement for Scroll API
POST /{index}/_count	Supported	Works as-is
Index Management
HEAD/PUT/DELETE /{index}	Supported	Works as-is
GET/PUT /{index}/_mapping	Supported	Works as-is
POST /{index}/_refresh	Not supported -> Skipped	No-op in serverless (auto-refresh)
GET /{index}/_aliases	Supported	Works as-is
Document
POST /_bulk	Supported	Works as-is
POST /{index}/_doc	Supported	Works as-is
POST /{index}/_delete_by_query	Not supported -> Fallback	Skipped, falls back to scroll + bulk delete

20260216Update

After further testing, I found that POST /_search/scroll returns 404 Not Found on OpenSearch Serverless. The initial search with ?scroll=1m succeeds and returns a _scroll_id, but the continuation endpoint does not exist. This means the current read implementation cannot paginate beyond the first batch (opensearch.scroll.size, default 1,000 docs). Note that opensearch.scroll.size can be increased up to 10,000, but that is the hard limit on Serverless since index.max_result_window is fixed at 10,000 and cannot be modified.

The slice parameter returns 500 Internal Server Error, so parallel partition reads via opensearch.input.max.docs.per.partition are not possible on Serverless. All reads go through a single partition.

The only viable pagination method on Serverless is PIT + Search After API (pit + sort + search_after), which provides consistent snapshot reads. The sort should include _id as a tiebreaker ("sort": ["_doc", "_id"]) to prevent missing documents caused by duplicate _doc sort values on Serverless.

[achaiban]

Thanks @lawofcycles for your work on this so far. I see you seem to have patched the sink related API request handling for AOSS mentioned here: #269 (comment)

If you could please provide me a build I'd love to test df sink with Glue 4.0 and report back, Gradle is new to me so I haven't been able to build the patch myself.

[lawofcycles]

@achaiban Thank you for your suggestion and sorry for the late response.
I was a bit busy recently so I couldn't work on this, but I'm back on it now.
Once my work reaches a testable stage, I'll be pleased to share it with you. Please bear with me a little longer.
Currently, I'm testing my patch on Spark on EMR.

[lawofcycles]

@Xtansia @harishbhakuni @nknize
@dimorportheca47 and I have finally developed serverless support. We tested it with Spark, Hive, and MapReduce on EMR 7.12.
Could you take a look?

[lawofcycles]

I pushed a new commit to replace Scroll API with Search After API for read pagination in serverless mode.

POST /_search/scroll returns 404 on Serverless, so the read path was broken for indices larger than opensearch.scroll.size (default 1,000 docs). The new commit uses the Search After API (sort + search_after) to paginate through all documents.

Tested on EMR 7.12 with a 100,000 doc index on OpenSearch Serverless. Both read and write operations work correctly. Also verified that the standard (non serverless) read/write path is unaffected.

Future consideration

The current Search After implementation reuses ScrollReader and ScrollQuery with a serverless mode flag. Ideally, the pagination logic (ScrollQuery) should be split into separate ScrollPaginationQuery and SearchAfterPaginationQuery implementations, with the common hit parsing logic in ScrollReader extracted into a shared base. However, ScrollReader.read() contains complex parsing, raw JSON reconstruction, and metadata extraction that accounts for the majority of the code, while the Scroll/SearchAfter divergence is limited to response header handling and a few lines of state management. Restructuring this carries regression risk disproportionate to the benefit, so this refactoring is left for a future PR.

[lawofcycles]

I pushed another commit to add PIT (Point in Time) support and a _id sort tiebreaker for serverless reads.

The previous Search After implementation had two issues.

1. Missing documents: sort: ["_doc"] alone produces duplicate sort values on Serverless (all docs in a batch share the same value), causing search_after to skip unread documents at batch boundaries. Adding _id as a tiebreaker (sort: ["_doc", "_id"]) ensures unique sort values and eliminates the data loss.

2. No snapshot consistency: Search After without PIT is stateless, so concurrent writes or segment merges during reads could cause duplicates or gaps. Each read session now creates a PIT at the start and deletes it on close, ensuring all pages are read from a consistent snapshot.

Tested on EMR 7.12 with OpenSearch Serverless and provisioned OpenSearch.

[lawofcycles]

@Xtansia @nknize @harshavamsi I've updated the PR description to reflect the latest changes. Ready for review.

[harshavamsi]

So we should definitely upgrade the client to support OS versions 3 and above, but it will be a breaking change that we can address later

[lawofcycles]

Good catch, I'll raise PR later.

[lawofcycles]

After looking into this further, the version branching in the codebase is mostly limited to the type/typeless API distinction (V_1_X vs V_2_X+), and V_3_X is already defined and parseable in OpenSearchMajorVersion. Since Serverless does not expose its internal OpenSearch version through GET /, it is unclear what version should be returned here.
The current V_2_X works correctly because all existing version branches treat V_2_X and above identically (typeless path). I think we can revisit this when a concrete incompatibility surfaces, either from Serverless behavior changes.

[harshavamsi]

does serverless not support delete?

[harshavamsi]

nvm, saw the description

[lawofcycles]

@harshavamsi
I've been testing various write/read patterns against OpenSearch Serverless and found a couple of issues. Submitted follow-up PRs:

Critical fixes:

• Skip refreshIndex in serverless mode to fix 404 on _refresh endpoint #692: Fix serverless mode writes failing with 404 on _refresh endpoint ([BUG] Serverless mode: writes fail with 404 on refresh endpoint #690)
• Use PIT + search_after for delete in serverless mode (#691) #693: Fix SaveMode.Overwrite failing when document count exceeds scroll size in serverless mode ([BUG] Serverless mode: SaveMode.Overwrite fails when document count exceeds scroll size #691)

Minor improvement:

• Add opensearch.search_after.size setting for serverless mode page size #695: Add dedicated opensearch.search_after.size setting for serverless mode page size ([FEATURE] Add dedicated page size setting for serverless mode (search_after pagination) #694)

All verified on EMR 7.12 + OpenSearch Serverless. Would appreciate a review when you get a chance.

[advaitpathak-cod]

Hi @harshavamsi,

Thanks a lot for adding support for OpenSearch Serverless - this is really appreciated. I just wanted to check when this change is expected to be included in a Maven release and which version it will be part of.

At the moment, the latest version available on Maven Central is 1.3.0:
https://mvnrepository.com/artifact/org.opensearch.client/opensearch-spark-30_2.12/1.3.0

It would be helpful to know when a new release including this PR might be published.

[lawofcycles]

@advaitpathak-cod
Hi, I'm planning the new release for opensearch-hadoop.
There is no fixed timeline yet, but I'm currently working on confirming the tasks and schedule.

#685
#699

If you need a version with serverless support before that, I can show you how to build the jar.

[advaitpathak-cod]

@advaitpathak-cod Hi, I'm planning the new release for opensearch-hadoop. There is no fixed timeline yet, but I'm currently working on confirming the tasks and schedule.

#685 #699

If you need a version with serverless support before that, I can show you how to build the jar.

Thanks @lawofcycles for the update.
I can wait for the serverless support as part of the regular release.