[BUG] Serverless mode: SaveMode.Overwrite fails when document count exceeds scroll size

[lawofcycles]

What is the bug?

When using SaveMode.Overwrite with OpenSearch Serverless, the overwrite operation fails if the existing index contains more documents than the internal scroll size (500).

RestRepository.delete() uses the Scroll API (_search?scroll=10m followed by POST _search/scroll) to iterate all documents for deletion. The initial search request succeeds because Serverless silently ignores the scroll parameter, but the subsequent scroll continuation call (POST _search/scroll) returns 404 since the Scroll API is not supported on Serverless.

When the document count is under 500 (the hardcoded batch size in delete()), all documents are returned in the initial response and no scroll continuation is needed, so the overwrite succeeds. With more than 500 documents, the scroll continuation is required and the operation fails.

org.opensearch.hadoop.rest.OpenSearchHadoopInvalidRequest:
[POST] on [_search/scroll] failed;
server[https://<collection>.aoss.amazonaws.com:443] returned [404|Not Found]

How can one reproduce the bug?

from pyspark.sql import SparkSession
import time

spark = SparkSession.builder.getOrCreate()

opts = {
   "opensearch.nodes": "<serverless endpoint>",
   "opensearch.port": "443",
   "opensearch.nodes.wan.only": "true",
   "opensearch.net.ssl": "true",
   "opensearch.aws.sigv4.enabled": "true",
   "opensearch.aws.sigv4.region": "<region>",
   "opensearch.resource": "overwrite-test",
   "opensearch.nodes.resolve.hostname": "false",
   "opensearch.serverless": "true",
   "opensearch.batch.write.refresh": "false",
}

# Write 600 records (exceeds internal scroll size of 500)
data = [{"id": i, "value": f"doc_{i}"} for i in range(600)]
df = spark.createDataFrame(data)
w = df.write.format("org.opensearch.spark.sql")
for k, v in opts.items():
   w = w.option(k, v)
w.mode("append").save()

time.sleep(15)  # wait for serverless refresh

# Overwrite triggers scroll-based delete, which fails
data2 = [{"id": i, "value": f"new_{i}"} for i in range(10)]
df2 = spark.createDataFrame(data2)
w2 = df2.write.format("org.opensearch.spark.sql")
for k, v in opts.items():
   w2 = w2.option(k, v)
w2.mode("overwrite").save()  # fails with 404 on search/scroll

What is the expected behavior?

RestRepository.delete() should use PIT + search_after pagination in serverless mode instead of the Scroll API, consistent with the read path implemented in #586

What is your host/environment?

EMR 7.12 (Spark 3.5, JDK 17), OpenSearch Serverless

Do you have any screenshots?

N/A

Do you have any additional context?

Introduced in #586. The delete() method in RestRepository was not updated for serverless mode. With 500 or fewer documents, the overwrite succeeds because all documents fit in the initial search response and no scroll continuation is needed.