What is the bug?
When writing data to OpenSearch Serverless with opensearch.serverless=true, the write operation fails with a 404 error because BulkProcessor.close() calls RestClient.refreshIndex(String), which does not have the serverless mode skip logic.
`RestClient.refresh(Resource)` was correctly updated in #586 to skip the refresh call in serverless mode, but `RestClient.refreshIndex(String)` (a separate method called from `BulkProcessor`) was not updated. Since `opensearch.batch.write.refresh` defaults to `true`, all Spark writes to OpenSearch Serverless fail unless the user explicitly sets `opensearch.batch.write.refresh=false`.
How can one reproduce the bug?
Using PySpark on EMR with opensearch-hadoop 1.3.0-SNAPSHOT (commit 9807b25):
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
data = [{"id": i, "value": f"doc_{i}"} for i in range(10)]
df = spark.createDataFrame(data)
df.write \
.format("org.opensearch.spark.sql") \
.option("opensearch.nodes", "<serverless endpoint>") \
.option("opensearch.port", "443") \
.option("opensearch.nodes.wan.only", "true") \
.option("opensearch.net.ssl", "true") \
.option("opensearch.aws.sigv4.enabled", "true") \
.option("opensearch.aws.sigv4.region", "<region>") \
.option("opensearch.resource", "test-index") \
.option("opensearch.nodes.resolve.hostname", "false") \
.option("opensearch.serverless", "true") \
.mode("append") \
.save()
This fails with a 404 on _refresh. Adding .option("opensearch.batch.write.refresh", "false") works around the issue.
What is the expected behavior?
RestClient.refreshIndex(String) should skip the refresh call when opensearch.serverless=true, consistent with RestClient.refresh(Resource).
What is your host/environment?
EMR 7.12 (Spark 3.5, JDK 17), OpenSearch Serverless (ap-northeast-1)
Do you have any screenshots?
N/A
Do you have any additional context?
Introduced in #586. The fix is to add the same serverless mode check to RestClient.refreshIndex(String) that already exists in RestClient.refresh(Resource).
What is the bug?
When writing data to OpenSearch Serverless with
opensearch.serverless=true, the write operation fails with a 404 error becauseBulkProcessor.close()callsRestClient.refreshIndex(String), which does not have the serverless mode skip logic.How can one reproduce the bug?
Using PySpark on EMR with opensearch-hadoop 1.3.0-SNAPSHOT (commit 9807b25):
This fails with a 404 on
_refresh. Adding.option("opensearch.batch.write.refresh", "false")works around the issue.What is the expected behavior?
RestClient.refreshIndex(String)should skip the refresh call whenopensearch.serverless=true, consistent withRestClient.refresh(Resource).What is your host/environment?
EMR 7.12 (Spark 3.5, JDK 17), OpenSearch Serverless (ap-northeast-1)
Do you have any screenshots?
N/A
Do you have any additional context?
Introduced in #586. The fix is to add the same serverless mode check to
RestClient.refreshIndex(String)that already exists inRestClient.refresh(Resource).