[BUG] RFS - ElasticSearch 6.8 -> OpenSearch 2.x local testing fails

[shaun-scaling]

What is the bug?

I'm trying to follow the instructions here - https://github.com/opensearch-project/opensearch-migrations/blob/main/RFS/README.md, but they fail for ES6.8 -> OS2.x

From what I understand, the RFS process will allow jumping versions 6.8 -> OS 2.x, negating the normal upgrade restrictions of one major lucene version - so my expectations are that this should work.

What are your migration environments?

From main and using

./gradlew DocumentsFromSnapshotMigration:es68ComposeDown && ./gradlew DocumentsFromSnapshotMigration:es68ComposeUp

From the root endpoint, the domains show:
ES 6.8.23
OS 2.11.1

How can one reproduce the bug?

Follow the dev instructions (summarised below)

1. Create S3 bucket
2. Create + clean local /tmp dirs (s3 + lucene)
3. Export aws credentials with access to s3 bucket
4. ./gradlew DocumentsFromSnapshotMigration:es68ComposeDown && ./gradlew DocumentsFromSnapshotMigration:es68ComposeUp
5. Create test document curl "localhost:19200/test/_doc/1" --json '{"foo":true}'
6. Create a snapshot ./gradlew CreateSnapshot:run --args='--snapshot-name reindex-from-snapshot --s3-repo-uri s3://<redacted> --s3-region us-east-1 --source-host http://localhost:19200'
7. Migrate metadata ./gradlew MetadataMigration:run --args='migrate --source-version "ElasticSearch 6.8" --snapshot-name reindex-from-snapshot --s3-local-dir /tmp/s3_files --s3-repo-uri s3://<redacted> --s3-region us-east-1 --target-host http://localhost:29200'
8. Backfill from snapshot ./gradlew DocumentsFromSnapshotMigration:run --args='--source-version "ES 6.8" --snapshot-name reindex-from-snapshot --s3-local-dir /tmp/s3_files --s3-repo-uri s3://<redacted> --s3-region us-east-1 --lucene-dir /tmp/lucene_files --target-host http://localhost:29200'

What is the expected behavior?

I expected the test index to be migrated to the OS 2.x domain

However, the data is not migrated and the following logs are observed

> Task :DocumentsFromSnapshotMigration:run
2025-03-12 12:19:09,865 INFO o.o.m.u.ProcessHelpers [main] getNodeInstanceName()=generated_50f2b502-9bf0-4431-b33e-d50b993d2c6d
Starting program with: --source-version ES 6.8 --snapshot-name reindex-from-snapshot --s3-local-dir /tmp/s3_files --s3-repo-uri s3://<redacted> --s3-region us-east-1 --lucene-dir /tmp/lucene_files --target-host http://localhost:29200
2025-03-12 12:19:09,867 INFO o.o.m.RfsMigrateDocuments [main] Starting RfsMigrateDocuments with workerId=generated_50f2b502-9bf0-4431-b33e-d50b993d2c6d
2025-03-12 12:19:09,888 WARN o.o.m.t.RootOtelContext [main] Collector endpoint=null, so serviceName parameter 'documentMigration' is being ignored since a no-op OpenTelemetry object is being created
2025-03-12 12:19:10,231 ERROR i.n.r.d.DnsServerAddressStreamProviders [main] Unable to load io.netty.resolver.dns.macos.MacOSDnsServerAddressStreamProvider, fallback to system defaults. This may result in incorrect DNS resolutions on MacOS. Check whether you have a dependency on 'io.netty:netty-resolver-dns-native-macos'. Use DEBUG level to see the full stack: java.lang.UnsatisfiedLinkError: failed to load the required native library
2025-03-12 12:19:10,357 INFO o.o.m.RfsMigrateDocuments [main] Doc Transformations config string: [  {    "JsonTransformerForDocumentTypeRemovalProvider":""  }]
2025-03-12 12:19:10,359 INFO o.o.m.t.TransformationLoader [main] Adding IJsonTransfomerProvider: org.opensearch.migrations.transformation.JsonTransformerForDocumentTypeRemovalProvider@1686f0b4
2025-03-12 12:19:10,359 INFO o.o.m.t.TransformationLoader [main] Adding IJsonTransfomerProvider: org.opensearch.migrations.transform.JsonConditionalTransformerProvider@61e3cf4d
2025-03-12 12:19:10,360 INFO o.o.m.t.TransformationLoader [main] Adding IJsonTransfomerProvider: org.opensearch.migrations.transform.TypeMappingSanitizationTransformerProvider@1d61c6dc
2025-03-12 12:19:10,367 INFO o.o.m.t.TransformationLoader [main] Adding IJsonTransfomerProvider: org.opensearch.migrations.transform.JsonJMESPathTransformerProvider@50a691d3
2025-03-12 12:19:10,368 INFO o.o.m.t.TransformationLoader [main] Adding IJsonTransfomerProvider: org.opensearch.migrations.transform.JsonJinjavaTransformerProvider@87b5b49
2025-03-12 12:19:10,368 INFO o.o.m.t.TransformationLoader [main] Adding IJsonTransfomerProvider: org.opensearch.migrations.transform.JsonJoltTransformerProvider@128c502c
2025-03-12 12:19:10,368 INFO o.o.m.t.TransformationLoader [main] Adding IJsonTransfomerProvider: org.opensearch.migrations.transform.NoopTransformerProvider@3c4262d1
2025-03-12 12:19:10,369 INFO o.o.m.t.TransformationLoader [main] IJsonTransformerProviders loaded: class org.opensearch.migrations.transformation.JsonTransformerForDocumentTypeRemovalProvider; class org.opensearch.migrations.transform.JsonConditionalTransformerProvider; class org.opensearch.migrations.transform.TypeMappingSanitizationTransformerProvider; class org.opensearch.migrations.transform.JsonJMESPathTransformerProvider; class org.opensearch.migrations.transform.JsonJinjavaTransformerProvider; class org.opensearch.migrations.transform.JsonJoltTransformerProvider; class org.opensearch.migrations.transform.NoopTransformerProvider

2025-03-12 12:19:10,847 INFO o.o.m.c.ClusterProviderRegistry [main] Found snapshot resource reader for version: ELASTICSEARCH 6.8.0
2025-03-12 12:19:10,871 INFO o.o.m.b.w.OpenSearchWorkCoordinator [main] Creating .migrations_working_state because HEAD returned 404
2025-03-12 12:19:11,237 INFO o.o.m.b.w.ShardWorkPreparer [main] Acquired work to set the shard workitems
2025-03-12 12:19:11,237 INFO o.o.m.b.w.ShardWorkPreparer [main] Setting up the Documents Work Items...
2025-03-12 12:19:12,640 INFO o.o.m.b.w.ShardWorkPreparer [main] Index test has 5 shards
2025-03-12 12:19:12,640 INFO o.o.m.b.w.ShardWorkPreparer [main] Creating Documents Work Item for index: test, shard: 0
2025-03-12 12:19:12,651 INFO o.o.m.b.w.ShardWorkPreparer [main] Creating Documents Work Item for index: test, shard: 1
2025-03-12 12:19:12,658 INFO o.o.m.b.w.ShardWorkPreparer [main] Creating Documents Work Item for index: test, shard: 2
2025-03-12 12:19:12,665 INFO o.o.m.b.w.ShardWorkPreparer [main] Creating Documents Work Item for index: test, shard: 3
2025-03-12 12:19:12,671 INFO o.o.m.b.w.ShardWorkPreparer [main] Creating Documents Work Item for index: test, shard: 4
2025-03-12 12:19:12,678 INFO o.o.m.b.w.ShardWorkPreparer [main] Finished setting up the Documents Work Items.
2025-03-12 12:19:12,830 INFO o.o.m.b.w.OpenSearchWorkCoordinator [main] Returning work item and lease: OpenSearchWorkCoordinator.WorkItemWithPotentialSuccessors(workItemId=test__0__0, leaseExpirationTime=2025-03-12T12:29:12Z, successorWorkItemIds=[])
2025-03-12 12:19:12,830 INFO o.o.m.b.w.DocumentsRunner [main] Migrating docs for test__0__0
2025-03-12 12:19:13,220 INFO o.o.m.b.c.S3Repo [main] Downloading file from S3: s3://<redacted>/indices/f0IJl5kXSY-EupwULbpXww/0/snap-5IdHjmabR7KvvYUJCd74Bw.dat to /tmp/s3_files/indices/f0IJl5kXSY-EupwULbpXww/0/snap-5IdHjmabR7KvvYUJCd74Bw.dat
2025-03-12 12:19:13,657 INFO o.o.m.RfsMigrateDocuments [main] Shard size: 230
2025-03-12 12:19:13,664 INFO o.o.m.b.c.S3Repo [main] Downloading blob files from S3: s3://<redacted>/indices/f0IJl5kXSY-EupwULbpXww/0/ to /tmp/s3_files/indices/f0IJl5kXSY-EupwULbpXww/0
2025-03-12 12:19:15,797 INFO o.o.m.b.c.S3Repo [main] Blob file download(s) complete
2025-03-12 12:19:15,814 INFO o.o.m.b.c.SnapshotShardUnpacker [main] Unpacking - Blob Name: __gEj8wzYfSLuOauMNAcPpqw, Lucene Name: segments_2
2025-03-12 12:19:15,816 INFO o.o.m.b.l.LuceneIndexReader$Factory [main] Creating IndexReader7
2025-03-12 12:19:15,850 INFO o.o.m.b.l.LuceneReader [workFinishScheduler-1] 0 documents in 0 leaves found in the current Lucene index
2025-03-12 12:19:15,856 INFO o.o.m.b.w.DocumentsRunner [DocumentBatchReindexer-1] Reindexing completed for Index test, Shard 0
2025-03-12 12:19:15,884 WARN o.o.m.RfsMigrateDocuments [Cleanup-Hook-Thread] Received shutdown signal. Trying to mark progress and shutdown cleanly.
2025-03-12 12:19:15,884 INFO o.o.m.RfsMigrateDocuments [Cleanup-Hook-Thread] Clean shutdown already completed
2025-03-12 12:19:15,885 INFO o.o.m.RfsMigrateDocuments [Cleanup-Hook-Thread] Clean shutdown completed.

Do you have any additional context?

The backfill migration is successful if I use ES7 as source and OS2.11 as the target, the logs show reindexing complete for that setup

2025-03-12 11:28:29,131 INFO o.o.m.b.c.S3Repo [main] Downloading file from S3: s3://<redacted>/index-0 to /tmp/s3_files/index-0
2025-03-12 11:28:29,569 INFO o.o.m.b.c.S3Repo [main] Downloading file from S3: s3://<redacted>/indices/qx6vOXS9SRGTAM7Gg9Pmmw/meta-31gZipUBFGtiaci8vH19.dat to /tmp/s3_files/indices/qx6vOXS9SRGTAM7Gg9Pmmw/meta-31gZipUBFGtia
ci8vH19.dat
2025-03-12 11:28:30,120 INFO o.o.m.b.w.ShardWorkPreparer [main] Index test has 1 shards
2025-03-12 11:28:30,121 INFO o.o.m.b.w.ShardWorkPreparer [main] Creating Documents Work Item for index: test, shard: 0
2025-03-12 11:28:30,161 INFO o.o.m.b.w.ShardWorkPreparer [main] Finished setting up the Documents Work Items.
2025-03-12 11:28:30,321 INFO o.o.m.b.w.OpenSearchWorkCoordinator [main] Returning work item and lease: OpenSearchWorkCoordinator.WorkItemWithPotentialSuccessors(workItemId=test__0__0, leaseExpirationTime=2025-03-12T11:38:3
0Z, successorWorkItemIds=[])
2025-03-12 11:28:30,322 INFO o.o.m.b.w.DocumentsRunner [main] Migrating docs for test__0__0
2025-03-12 11:28:30,716 INFO o.o.m.b.c.S3Repo [main] Downloading file from S3: s3://<redacted>/indices/qx6vOXS9SRGTAM7Gg9Pmmw/0/snap-BFMfF4LPRS2Bpfcs_7hGDw.dat to /tmp/s3_files/indices/qx6vOXS9SRGTAM7Gg9Pmmw/0/snap-BFMfF4L
PRS2Bpfcs_7hGDw.dat
2025-03-12 11:28:31,312 INFO o.o.m.RfsMigrateDocuments [main] Shard size: 3329
2025-03-12 11:28:31,325 INFO o.o.m.b.c.S3Repo [main] Downloading blob files from S3: s3://<redacted>/indices/qx6vOXS9SRGTAM7Gg9Pmmw/0/ to /tmp/s3_files/indices/qx6vOXS9SRGTAM7Gg9Pmmw/0
2025-03-12 11:28:33,115 INFO o.o.m.b.c.S3Repo [main] Blob file download(s) complete
2025-03-12 11:28:33,129 INFO o.o.m.b.c.SnapshotShardUnpacker [main] Unpacking - Blob Name: __NKzQVRC9Rc6a-iP7XIjbfQ, Lucene Name: _0.cfe
2025-03-12 11:28:33,130 INFO o.o.m.b.c.SnapshotShardUnpacker [main] Unpacking - Blob Name: v__3KyiFr8JQCm5ztg7WQ7lGA, Lucene Name: _0.si
2025-03-12 11:28:33,131 INFO o.o.m.b.c.SnapshotShardUnpacker [main] Unpacking - Blob Name: __judMvv1jQbWdErTX5lc32Q, Lucene Name: _0.cfs
2025-03-12 11:28:33,131 INFO o.o.m.b.c.SnapshotShardUnpacker [main] Unpacking - Blob Name: v__513PFQGXRH-TbEO2WMWNWQ, Lucene Name: segments_3
2025-03-12 11:28:33,132 INFO o.o.m.b.l.LuceneIndexReader$Factory [main] Creating IndexReader9
2025-03-12 11:28:33,230 INFO o.o.m.b.l.LuceneReader [workFinishScheduler-1] 1 documents in 1 leaves found in the current Lucene index
2025-03-12 11:28:33,243 INFO o.o.m.t.TransformationLoader [DocumentBulkAggregator-2] Creating a transformer through provider=org.opensearch.migrations.transformation.JsonTransformerForDocumentTypeRemovalProvider@1686f0b4 w
ith configuration=
2025-03-12 11:28:33,258 INFO o.o.m.b.c.DocumentReindexer [DocumentBatchReindexer-1] Last doc is: Source Index test Lucene Doc Number 0
2025-03-12 11:28:33,260 INFO o.o.m.b.c.DocumentReindexer [DocumentBatchReindexer-1] Batch Id:3a58cf16-4046-43ab-b069-226e2bc2b671, 1 documents in current bulk request.
2025-03-12 11:28:33,348 INFO o.o.m.b.w.DocumentsRunner [DocumentBatchReindexer-3] Reindexing completed for Index test, Shard 0
2025-03-12 11:28:33,371 WARN o.o.m.RfsMigrateDocuments [Cleanup-Hook-Thread] Received shutdown signal. Trying to mark progress and shutdown cleanly.
2025-03-12 11:28:33,372 INFO o.o.m.RfsMigrateDocuments [Cleanup-Hook-Thread] Clean shutdown already completed
2025-03-12 11:28:33,372 INFO o.o.m.RfsMigrateDocuments [Cleanup-Hook-Thread] Clean shutdown completed.

[peternied]

@shaun-scaling Thanks for mentioning this issue, I'll do some investigating and I'll update this issue when I've got more details.

[peternied]

@shaun-scaling the fix for this issue should be merged. Give it another try. I'm also really curious what you are up to with our project, if you have some time reach out to me at petern@amazon.com or find me on the OpenSearch slack.

[peternied]

Spoke with @AndreKurait and he pointed out that the issue you were having is likely based on how the RFS worker was started and ran only once. After reviewing your comment, it looks like this was the case, our documentation is lacking in this space.

When running RFS, you should run it until it throws a no work left exception. This is handled when it runs inside its containerized environment, but isn't part of the run method in gradle.

opensearch-migrations/DocumentsFromSnapshotMigration/docker/entrypoint.sh

Lines 111 to 127 in 89b98c7

	[ -z "$RFS_COMMAND" ] && \
	{ echo "Warning: RFS_COMMAND is empty! Exiting."; exit 1; } || \
	while true; do
	echo "Running command $RFS_COMMAND"
	# Run command in background and get its PID
	eval "$RFS_COMMAND" &
	COMMAND_PID=$!
	# Wait for the command to finish
	wait $COMMAND_PID
	EXIT_CODE=$?
	if [ $EXIT_CODE -ne 0 ]; then
	echo "Command exited with code $EXIT_CODE"
	break
	fi
	echo "Cleaning up directories before the next run."
	cleanup_directories
	done

Each execution of this last tool moves the documents from a single shard of the source snapshot before exiting. To migrate all the shards of the source snapshot, keep running the tool until it throws a NoWorkLeftException. You can also look at the work coordination metadata stored in a special index created on the target cluster, but that requires specialized knowledge.

https://github.com/opensearch-project/opensearch-migrations/tree/main/RFS#migrate-the-documents

I'll make a separate changes to update the readme, but I don't think there is a product issue here.

[shaun-scaling]

Thanks @peternied for the quick response - I suspected user error, but wanted to document it as I was fairly confident I'd followed the documentation. It looks like some of my assumptions were wrong.