[BUG] Error in Backfill migration from ES_6.7 to OS_2.17

[rudney-souza]

What is the bug?

When running the backfill migration from ES_6.7 to OS_2.17 i got an error like com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'Too': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false'), and was stuck in the same number of shards even after running all night long (at first were 50 workers and then ive added 130 to see if something changes, but none of remaining shards has finished yet)

How can one reproduce the bug?

running the backfill migration from ES_6.7 to OS_2.17

What is the expected behavior?

Run the migration without errors

What is your host/environment?

ES_6.7 to OS_2.17

Do you have any additional context?

2024-12-05 14:00:51,114 WARN o.o.m.b.c.OpenSearchClient [reactor-http-epoll-2] Unable to process bulk request for success com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'Too': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false') at [Source: REDACTED (StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION disabled); line: 1, column: 4] at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:2481) ~[jackson-core-2.16.2.jar:2.16.2] at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:762) ~[jackson-core-2.16.2.jar:2.16.2] at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._reportInvalidToken(ReaderBasedJsonParser.java:3042) ~[jackson-core-2.16.2.jar:2.16.2] at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:2085) ~[jackson-core-2.16.2.jar:2.16.2] at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:812) ~[jackson-core-2.16.2.jar:2.16.2] at org.opensearch.migrations.parsing.BulkResponseParser.findSuccessDocs(BulkResponseParser.java:39) ~[RFS-0.1.0-SNAPSHOT.jar:?] at org.opensearch.migrations.bulkload.common.OpenSearchClient$BulkResponse.getSuccessfulDocs(OpenSearchClient.java:531) ~[RFS-0.1.0-SNAPSHOT.jar:?] at org.opensearch.migrations.bulkload.common.OpenSearchClient.lambda$sendBulkRequest$28(OpenSearchClient.java:474) ~[RFS-0.1.0-SNAPSHOT.jar:?] at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:132) [reactor-core-3.6.5.jar:3.6.5] at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:180) [reactor-core-3.6.5.jar:3.6.5] at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:180) [reactor-core-3.6.5.jar:3.6.5] at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondComplete(MonoFlatMap.java:245) [reactor-core-3.6.5.jar:3.6.5] at reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:305) [reactor-core-3.6.5.jar:3.6.5] at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondComplete(MonoFlatMap.java:245) [reactor-core-3.6.5.jar:3.6.5] at reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:305) [reactor-core-3.6.5.jar:3.6.5] at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107) [reactor-core-3.6.5.jar:3.6.5] at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onNext(FluxDoFinally.java:113) [reactor-core-3.6.5.jar:3.6.5] at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onNext(FluxMap.java:224) [reactor-core-3.6.5.jar:3.6.5] at reactor.core.publisher.Operators$MonoInnerProducerBase.complete(Operators.java:2812) [reactor-core-3.6.5.jar:3.6.5] at reactor.core.publisher.MonoSingleOptional$SingleOptionalSubscriber.onNext(MonoSingleOptional.java:101) [reactor-core-3.6.5.jar:3.6.5] at reactor.core.publisher.FluxHandle$HandleSubscriber.onNext(FluxHandle.java:129) [reactor-core-3.6.5.jar:3.6.5] at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onNext(FluxMap.java:224) [reactor-core-3.6.5.jar:3.6.5] at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onNext(FluxDoFinally.java:113) [reactor-core-3.6.5.jar:3.6.5] at reactor.core.publisher.FluxHandleFuseable$HandleFuseableSubscriber.onNext(FluxHandleFuseable.java:194) [reactor-core-3.6.5.jar:3.6.5] at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107) [reactor-core-3.6.5.jar:3.6.5] at reactor.core.publisher.Operators$BaseFluxToMonoOperator.completePossiblyEmpty(Operators.java:2097) [reactor-core-3.6.5.jar:3.6.5] at reactor.core.publisher.MonoCollectList$MonoCollectListSubscriber.onComplete(MonoCollectList.java:118) [reactor-core-3.6.5.jar:3.6.5] at reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:260) [reactor-core-3.6.5.jar:3.6.5] at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:144) [reactor-core-3.6.5.jar:3.6.5] at reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:415) [reactor-netty-core-1.1.18.jar:1.1.18] at reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:446) [reactor-netty-core-1.1.18.jar:1.1.18] at reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:500) [reactor-netty-core-1.1.18.jar:1.1.18] at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:793) [reactor-netty-http-1.1.18.jar:1.1.18] at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:114) [reactor-netty-core-1.1.18.jar:1.1.18] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.108.Final.jar:4.1.108.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.108.Final.jar:4.1.108.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.108.Final.jar:4.1.108.Final] at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) [netty-transport-4.1.108.Final.jar:4.1.108.Final] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) [netty-codec-4.1.108.Final.jar:4.1.108.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) [netty-codec-4.1.108.Final.jar:4.1.108.Final] at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) [netty-transport-4.1.108.Final.jar:4.1.108.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.108.Final.jar:4.1.108.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.108.Final.jar:4.1.108.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.108.Final.jar:4.1.108.Final] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1475) [netty-handler-4.1.108.Final.jar:4.1.108.Final] at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1338) [netty-handler-4.1.108.Final.jar:4.1.108.Final] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1387) [netty-handler-4.1.108.Final.jar:4.1.108.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530) [netty-codec-4.1.108.Final.jar:4.1.108.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469) [netty-codec-4.1.108.Final.jar:4.1.108.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) [netty-codec-4.1.108.Final.jar:4.1.108.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.108.Final.jar:4.1.108.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.108.Final.jar:4.1.108.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.108.Final.jar:4.1.108.Final] at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) [netty-transport-4.1.108.Final.jar:4.1.108.Final] at org.opensearch.migrations.bulkload.netty.ReadMeteringHandler.channelRead(ReadMeteringHandler.java:26) [RFS-0.1.0-SNAPSHOT.jar:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.108.Final.jar:4.1.108.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.108.Final.jar:4.1.108.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.108.Final.jar:4.1.108.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.108.Final.jar:4.1.108.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.108.Final.jar:4.1.108.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.108.Final.jar:4.1.108.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.108.Final.jar:4.1.108.Final] at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:801) [netty-transport-classes-epoll-4.1.108.Final.jar:4.1.108.Final] at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:509) [netty-transport-classes-epoll-4.1.108.Final.jar:4.1.108.Final] at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:407) [netty-transport-classes-epoll-4.1.108.Final.jar:4.1.108.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.108.Final.jar:4.1.108.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.108.Final.jar:4.1.108.Final] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.108.Final.jar:4.1.108.Final] at java.base/java.lang.Thread.run(Thread.java:829) [?:?]

---

2024-12-05 14:00:48,025 ERROR r.c.s.Schedulers [parallel-3] Scheduler worker in group main failed with an uncaught exception java.lang.OutOfMemoryError: Java heap space at java.base/java.util.Arrays.copyOfRange(Arrays.java:4030) ~[?:?] at java.base/java.lang.StringLatin1.newString(StringLatin1.java:715) ~[?:?] at java.base/java.lang.StringBuilder.toString(StringBuilder.java:452) ~[?:?] at com.fasterxml.jackson.core.util.TextBuffer.contentsAsString(TextBuffer.java:498) ~[jackson-core-2.16.2.jar:2.16.2] at [com.fasterxml.jackson.core.io](http://com.fasterxml.jackson.core.io/).SegmentedStringWriter.getAndClear(SegmentedStringWriter.java:99) ~[jackson-core-2.16.2.jar:2.16.2] at org.opensearch.migrations.bulkload.common.BulkDocSection.convertToBulkRequestBody(BulkDocSection.java:72) ~[RFS-0.1.0-SNAPSHOT.jar:?] at org.opensearch.migrations.bulkload.common.OpenSearchClient.lambda$sendBulkRequest$29(OpenSearchClient.java:457) ~[RFS-0.1.0-SNAPSHOT.jar:?] at org.opensearch.migrations.bulkload.common.OpenSearchClient$$Lambda$1253/0x00000008007f5040.get(Unknown Source) ~[?:?] at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:45) ~[reactor-core-3.6.5.jar:3.6.5] at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.resubscribe(FluxRetryWhen.java:220) ~[reactor-core-3.6.5.jar:3.6.5] at reactor.core.publisher.FluxRetryWhen$RetryWhenOtherSubscriber.onNext(FluxRetryWhen.java:274) ~[reactor-core-3.6.5.jar:3.6.5] at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107) ~[reactor-core-3.6.5.jar:3.6.5] at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.innerNext(FluxConcatMapNoPrefetch.java:259) ~[reactor-core-3.6.5.jar:3.6.5] at reactor.core.publisher.FluxConcatMap$ConcatMapInner.onNext(FluxConcatMap.java:865) ~[reactor-core-3.6.5.jar:3.6.5] at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107) ~[reactor-core-3.6.5.jar:3.6.5] at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondComplete(MonoFlatMap.java:245) ~[reactor-core-3.6.5.jar:3.6.5] at reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:305) ~[reactor-core-3.6.5.jar:3.6.5] at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.complete(MonoIgnoreThen.java:294) ~[reactor-core-3.6.5.jar:3.6.5] at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onNext(MonoIgnoreThen.java:188) ~[reactor-core-3.6.5.jar:3.6.5] at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:237) ~[reactor-core-3.6.5.jar:3.6.5] at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51) ~[reactor-core-3.6.5.jar:3.6.5] at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:165) ~[reactor-core-3.6.5.jar:3.6.5] at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.complete(MonoIgnoreThen.java:294) ~[reactor-core-3.6.5.jar:3.6.5] at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onNext(MonoIgnoreThen.java:188) ~[reactor-core-3.6.5.jar:3.6.5] at reactor.core.publisher.MonoDelay$MonoDelayRunnable.propagateDelay(MonoDelay.java:270) ~[reactor-core-3.6.5.jar:3.6.5] at reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:285) ~[reactor-core-3.6.5.jar:3.6.5] at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68) ~[reactor-core-3.6.5.jar:3.6.5] at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28) ~[reactor-core-3.6.5.jar:3.6.5] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?] at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]

---

2024-12-05 14:00:47,618 WARN o.o.m.b.c.OpenSearchClient [reactor-http-epoll-4] After bulk request on index 'vindex', 0 more documents have succeed, 3614 remain

[peternied]

Thanks for reporting @rudney-souza

Unable to process bulk request for success com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'Too'...

I suspect that you were getting back a 429 - Too Many Requests and it didn't return a parseable body. You should be able to confirm this by looking at the cluster metrics.

You could quickly try reducing the backfill rate by using the console backfill scale command by dialing to 1 and see if the cluster handles the traffic better, see documentation.

There are some advanced options you can modify for RFS that could let you dial this in if you have especially large documents or expensive indexing operations. Use the reindexFromSnapshotExtraArgs field in the cdk.context.json to update your deployment, details. You'll need to do a redeployment from the bootstrap box to apply these settings.

Throughput impacting options

opensearch-migrations/DocumentsFromSnapshotMigration/src/main/java/org/opensearch/migrations/RfsMigrateDocuments.java

Lines 146 to 162 in 3d82d1f

	@Parameter(required = false,
	names = "--documents-per-bulk-request",
	description = "Optional. The number of documents to be included within each bulk request sent. " +
	"Default no max (controlled by documents size)")
	int numDocsPerBulkRequest = Integer.MAX_VALUE;
	@Parameter(required = false,
	names = "--documents-size-per-bulk-request",
	description = "Optional. The maximum aggregate document size to be used in bulk requests in bytes. " +
	"Note does not apply to single document requests. Default 10 MiB")
	long numBytesPerBulkRequest = 10 * 1024L * 1024L;
	@Parameter(required = false,
	names = "--max-connections",
	description = "Optional. The maximum number of connections to simultaneously " +
	"used to communicate to the target, default 10")
	int maxConnections = 10;

[rudney-souza]

ive reduced and the erro with 'Too' has gone, but the error with java heap and not migrating more indexes stayed

[peternied]

@rudney-souza Could you collect some stats about the documents in the index that you are migrating? Dialing down that number of documents --documents-per-bulk-request might help get unstuck while we figure out what is using up the memory.

[rudney-souza]

of course @peternied , when i dial down im going to let you know here!

[rudney-souza]

So when running 200 docs per bulk request i got this now (but it seems just to happen one time while running)
2024-12-06 15:27:29,184 ERROR o.o.m.RfsMigrateDocuments [main] Unexpected error running RfsWorker org.opensearch.migrations.bulkload.common.SnapshotShardUnpacker$CouldNotUnpackShard: Could not unpack shard: Index 8sawzFonQf675wuL8IUjfg, Shard 0 at org.opensearch.migrations.bulkload.common.SnapshotShardUnpacker.unpack(SnapshotShardUnpacker.java:90) ~[RFS-0.1.0-SNAPSHOT.jar:?] at org.opensearch.migrations.bulkload.worker.DocumentsRunner.doDocumentsMigration(DocumentsRunner.java:98) ~[RFS-0.1.0-SNAPSHOT.jar:?] at org.opensearch.migrations.bulkload.worker.DocumentsRunner$1.onAcquiredWork(DocumentsRunner.java:66) ~[RFS-0.1.0-SNAPSHOT.jar:?] at org.opensearch.migrations.bulkload.worker.DocumentsRunner$1.onAcquiredWork(DocumentsRunner.java:58) ~[RFS-0.1.0-SNAPSHOT.jar:?] at org.opensearch.migrations.bulkload.workcoordination.ScopedWorkCoordinator$1.onAcquiredWork(ScopedWorkCoordinator.java:49) ~[RFS-0.1.0-SNAPSHOT.jar:?] at org.opensearch.migrations.bulkload.workcoordination.IWorkCoordinator$WorkItemAndDuration.visit(IWorkCoordinator.java:191) ~[RFS-0.1.0-SNAPSHOT.jar:?] at org.opensearch.migrations.bulkload.workcoordination.ScopedWorkCoordinator.ensurePhaseCompletion(ScopedWorkCoordinator.java:33) ~[RFS-0.1.0-SNAPSHOT.jar:?] at org.opensearch.migrations.bulkload.worker.DocumentsRunner.migrateNextShard(DocumentsRunner.java:49) ~[RFS-0.1.0-SNAPSHOT.jar:?] at org.opensearch.migrations.RfsMigrateDocuments.run(RfsMigrateDocuments.java:381) ~[DocumentsFromSnapshotMigration-0.1.0-SNAPSHOT.jar:?] at org.opensearch.migrations.RfsMigrateDocuments.main(RfsMigrateDocuments.java:307) [DocumentsFromSnapshotMigration-0.1.0-SNAPSHOT.jar:?] Caused by: java.util.concurrent.CompletionException: software.amazon.awssdk.core.exception.SdkClientException: Failed to send request at java.base/java.util.concurrent.CompletableFuture.reportJoin(CompletableFuture.java:412) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.join(CompletableFuture.java:2050) ~[?:?] at org.opensearch.migrations.bulkload.common.S3Repo.prepBlobFiles(S3Repo.java:208) ~[RFS-0.1.0-SNAPSHOT.jar:?] at org.opensearch.migrations.bulkload.common.SourceRepoAccessor.prepBlobFiles(SourceRepoAccessor.java:49) ~[RFS-0.1.0-SNAPSHOT.jar:?] at org.opensearch.migrations.bulkload.common.SnapshotShardUnpacker.unpack(SnapshotShardUnpacker.java:44) ~[RFS-0.1.0-SNAPSHOT.jar:?] ... 9 more Caused by: software.amazon.awssdk.core.exception.SdkClientException: Failed to send request at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111) ~[sdk-core-2.25.19.jar:?] at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:47) ~[sdk-core-2.25.19.jar:?] at software.amazon.awssdk.transfer.s3.internal.DownloadDirectoryHelper.lambda$doDownloadDirectory$2(DownloadDirectoryHelper.java:121) ~[s3-transfer-manager-2.25.19.jar:?] at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?] at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2094) ~[?:?] at software.amazon.awssdk.transfer.s3.internal.AsyncBufferingSubscriber.onError(AsyncBufferingSubscriber.java:93) ~[s3-transfer-manager-2.25.19.jar:?] at software.amazon.awssdk.utils.async.DelegatingSubscriber.onError(DelegatingSubscriber.java:40) ~[utils-2.25.19.jar:?] at software.amazon.awssdk.core.internal.pagination.async.ItemsSubscription.lambda$fetchNextPage$0(ItemsSubscription.java:89) ~[sdk-core-2.25.19.jar:?] at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?] at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2094) ~[?:?] at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:79) ~[utils-2.25.19.jar:?] at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?] at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2094) ~[?:?] at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallMetricCollectionStage.lambda$execute$0(AsyncApiCallMetricCollectionStage.java:56) ~[sdk-core-2.25.19.jar:?] at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?] at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2094) ~[?:?] at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallTimeoutTrackingStage.lambda$execute$2(AsyncApiCallTimeoutTrackingStage.java:67) ~[sdk-core-2.25.19.jar:?] at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?] at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2094) ~[?:?] at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:79) ~[utils-2.25.19.jar:?] at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?] at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2094) ~[?:?] at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeAttemptExecute(AsyncRetryableStage.java:103) ~[sdk-core-2.25.19.jar:?] at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:184) ~[sdk-core-2.25.19.jar:?] at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:159) ~[sdk-core-2.25.19.jar:?] at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?] at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2094) ~[?:?] at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:79) ~[utils-2.25.19.jar:?] at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?] at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2094) ~[?:?] at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$execute$0(MakeAsyncHttpRequestStage.java:108) ~[sdk-core-2.25.19.jar:?] at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?] at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2094) ~[?:?] at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.completeResponseFuture(MakeAsyncHttpRequestStage.java:255) ~[sdk-core-2.25.19.jar:?] at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$executeHttpRequest$3(MakeAsyncHttpRequestStage.java:167) ~[sdk-core-2.25.19.jar:?] at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:930) ~[?:?] at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:907) ~[?:?] at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478) ~[?:?] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at java.base/java.lang.Thread.run(Thread.java:829) ~[?:?] Caused by: java.util.concurrent.CompletionException: software.amazon.awssdk.core.exception.SdkClientException: Unexpected exception occurred: s3metaRequest is not initialized yet at software.amazon.awssdk.utils.CompletableFutureUtils.errorAsCompletionException(CompletableFutureUtils.java:65) ~[utils-2.25.19.jar:?] at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.lambda$execute$0(AsyncExecutionFailureExceptionReportingStage.java:51) ~[sdk-core-2.25.19.jar:?] at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:930) ~[?:?] at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:907) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2094) ~[?:?] at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:79) ~[utils-2.25.19.jar:?] at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?] at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2094) ~[?:?] at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeAttemptExecute(AsyncRetryableStage.java:103) ~[sdk-core-2.25.19.jar:?] at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:184) ~[sdk-core-2.25.19.jar:?] at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:159) ~[sdk-core-2.25.19.jar:?] at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?] at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2094) ~[?:?] at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:79) ~[utils-2.25.19.jar:?] at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?] at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2094) ~[?:?] at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$execute$0(MakeAsyncHttpRequestStage.java:108) ~[sdk-core-2.25.19.jar:?] at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?] at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2094) ~[?:?] at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.completeResponseFuture(MakeAsyncHttpRequestStage.java:255) ~[sdk-core-2.25.19.jar:?] at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$executeHttpRequest$3(MakeAsyncHttpRequestStage.java:167) ~[sdk-core-2.25.19.jar:?] at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:930) ~[?:?] at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:907) ~[?:?] at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478) ~[?:?] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at java.base/java.lang.Thread.run(Thread.java:829) ~[?:?] Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unexpected exception occurred: s3metaRequest is not initialized yet at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111) ~[sdk-core-2.25.19.jar:?] at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:43) ~[sdk-core-2.25.19.jar:?] at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.lambda$onResponseBody$1(S3CrtResponseHandlerAdapter.java:114) ~[s3-2.25.19.jar:?] at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:883) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2257) ~[?:?] at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.onResponseBody(S3CrtResponseHandlerAdapter.java:105) ~[s3-2.25.19.jar:?] at software.amazon.awssdk.crt.s3.S3MetaRequestResponseHandlerNativeAdapter.onResponseBody(S3MetaRequestResponseHandlerNativeAdapter.java:19) ~[aws-crt-0.29.18.jar:0.29.18] Exception in thread "main" org.opensearch.migrations.bulkload.common.SnapshotShardUnpacker$CouldNotUnpackShard: Could not unpack shard: Index 8sawzFonQf675wuL8IUjfg, Shard 0 at org.opensearch.migrations.bulkload.common.SnapshotShardUnpacker.unpack(SnapshotShardUnpacker.java:90) at org.opensearch.migrations.bulkload.worker.DocumentsRunner.doDocumentsMigration(DocumentsRunner.java:98) at org.opensearch.migrations.bulkload.worker.DocumentsRunner$1.onAcquiredWork(DocumentsRunner.java:66) at org.opensearch.migrations.bulkload.worker.DocumentsRunner$1.onAcquiredWork(DocumentsRunner.java:58) at org.opensearch.migrations.bulkload.workcoordination.ScopedWorkCoordinator$1.onAcquiredWork(ScopedWorkCoordinator.java:49) at org.opensearch.migrations.bulkload.workcoordination.IWorkCoordinator$WorkItemAndDuration.visit(IWorkCoordinator.java:191) at org.opensearch.migrations.bulkload.workcoordination.ScopedWorkCoordinator.ensurePhaseCompletion(ScopedWorkCoordinator.java:33) at org.opensearch.migrations.bulkload.worker.DocumentsRunner.migrateNextShard(DocumentsRunner.java:49) at org.opensearch.migrations.RfsMigrateDocuments.run(RfsMigrateDocuments.java:381) at org.opensearch.migrations.RfsMigrateDocuments.main(RfsMigrateDocuments.java:307) Caused by: java.util.concurrent.CompletionException: software.amazon.awssdk.core.exception.SdkClientException: Failed to send request at java.base/java.util.concurrent.CompletableFuture.reportJoin(CompletableFuture.java:412) at java.base/java.util.concurrent.CompletableFuture.join(CompletableFuture.java:2050) at org.opensearch.migrations.bulkload.common.S3Repo.prepBlobFiles(S3Repo.java:208) at org.opensearch.migrations.bulkload.common.SourceRepoAccessor.prepBlobFiles(SourceRepoAccessor.java:49) at org.opensearch.migrations.bulkload.common.SnapshotShardUnpacker.unpack(SnapshotShardUnpacker.java:44) ... 9 more Caused by: software.amazon.awssdk.core.exception.SdkClientException: Failed to send request at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111) at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:47) at software.amazon.awssdk.transfer.s3.internal.DownloadDirectoryHelper.lambda$doDownloadDirectory$2(DownloadDirectoryHelper.java:121) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2094) at software.amazon.awssdk.transfer.s3.internal.AsyncBufferingSubscriber.onError(AsyncBufferingSubscriber.java:93) at software.amazon.awssdk.utils.async.DelegatingSubscriber.onError(DelegatingSubscriber.java:40) at software.amazon.awssdk.core.internal.pagination.async.ItemsSubscription.lambda$fetchNextPage$0(ItemsSubscription.java:89) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2094) at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:79) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2094) at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallMetricCollectionStage.lambda$execute$0(AsyncApiCallMetricCollectionStage.java:56) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2094) at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallTimeoutTrackingStage.lambda$execute$2(AsyncApiCallTimeoutTrackingStage.java:67) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2094) at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:79) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2094) at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeAttemptExecute(AsyncRetryableStage.java:103) at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:184) at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:159) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2094) at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:79) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2094) at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$execute$0(MakeAsyncHttpRequestStage.java:108) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2094) at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.completeResponseFuture(MakeAsyncHttpRequestStage.java:255) at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$executeHttpRequest$3(MakeAsyncHttpRequestStage.java:167) at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:930) at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:907) at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.util.concurrent.CompletionException: software.amazon.awssdk.core.exception.SdkClientException: Unexpected exception occurred: s3metaRequest is not initialized yet at software.amazon.awssdk.utils.CompletableFutureUtils.errorAsCompletionException(CompletableFutureUtils.java:65) at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.lambda$execute$0(AsyncExecutionFailureExceptionReportingStage.java:51) at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:930) at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:907) ... 32 more Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unexpected exception occurred: s3metaRequest is not initialized yet at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111) at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:43) at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.lambda$onResponseBody$1(S3CrtResponseHandlerAdapter.java:114) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) at java.base/java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:883) at java.base/java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2257) at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.onResponseBody(S3CrtResponseHandlerAdapter.java:105) at software.amazon.awssdk.crt.s3.S3MetaRequestResponseHandlerNativeAdapter.onResponseBody(S3MetaRequestResponseHandlerNativeAdapter.java:19)

---

And im still running backfill @peternied

[peternied]

@rudney-souza
Thanks for the latest error message, looks like the exception software.amazon.awssdk.core.exception.SdkClientException: Unexpected exception occurred: s3metaRequest is not initialized yet is the root cause related to the s3 client that is downloading data for the snapshot from S3.

I'll need to do more research for potential remedies, does it look like the backfill is still progressing forward or has it stalled?

[rudney-souza]

In my live tail it looks like its still progressing, but only with 200 docs per bulk!

[chelma]

@rudney-souza Some investigation seems to indicate that the s3metaRequest issue would be a transient, S3 server-side issue. Are you still encountering that issue, and is it impacting your migration?

[rudney-souza]

Its all running @chelma , but i got some of these errors like ERROR o.o.m.RfsMigrateDocuments [leaseWatchingProcessKillerThread-1-1] Terminating RfsMigrateDocuments because the lease has expired for index-x

[peternied]

@rudney-souza I don't see any specific action for us to take on this issue, please feel free to open up a bug report about a specific issue with a different behavior you'd expect.

We are tracking autoscaling MIGRATIONS-1622 which addresses the root cause of 429 retries.