Skip to content

[BUG] Serverless mode: writes fail with 404 on refresh endpoint #690

Description

@lawofcycles

What is the bug?

When writing data to OpenSearch Serverless with opensearch.serverless=true, the write operation fails with a 404 error because BulkProcessor.close() calls RestClient.refreshIndex(String), which does not have the serverless mode skip logic.

`RestClient.refresh(Resource)` was correctly updated in #586 to skip the refresh call in serverless mode, but `RestClient.refreshIndex(String)` (a separate method called from `BulkProcessor`) was not updated. Since `opensearch.batch.write.refresh` defaults to `true`, all Spark writes to OpenSearch Serverless fail unless the user explicitly sets `opensearch.batch.write.refresh=false`.

How can one reproduce the bug?

Using PySpark on EMR with opensearch-hadoop 1.3.0-SNAPSHOT (commit 9807b25):

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

data = [{"id": i, "value": f"doc_{i}"} for i in range(10)]
df = spark.createDataFrame(data)
df.write \
 .format("org.opensearch.spark.sql") \
 .option("opensearch.nodes", "<serverless endpoint>") \
 .option("opensearch.port", "443") \
 .option("opensearch.nodes.wan.only", "true") \
 .option("opensearch.net.ssl", "true") \
 .option("opensearch.aws.sigv4.enabled", "true") \
 .option("opensearch.aws.sigv4.region", "<region>") \
 .option("opensearch.resource", "test-index") \
 .option("opensearch.nodes.resolve.hostname", "false") \
 .option("opensearch.serverless", "true") \
 .mode("append") \
 .save()

This fails with a 404 on _refresh. Adding .option("opensearch.batch.write.refresh", "false") works around the issue.

What is the expected behavior?

RestClient.refreshIndex(String) should skip the refresh call when opensearch.serverless=true, consistent with RestClient.refresh(Resource).

What is your host/environment?

EMR 7.12 (Spark 3.5, JDK 17), OpenSearch Serverless (ap-northeast-1)

Do you have any screenshots?

N/A

Do you have any additional context?

Introduced in #586. The fix is to add the same serverless mode check to RestClient.refreshIndex(String) that already exists in RestClient.refresh(Resource).

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions