[BUG] Unusual behavior of RFS and Backfill

[jugal-chauhan]

What is the bug?

The bug involves unexpected behavior during the migration process from ElasticSearch 7.10 to OpenSearch 2.17, both hosted on AWS OpenSearch Service. Specifically:

• target cluster size (pri.store.size) unexpectedly drops by 5-10GB multiple times during backfill.
• document count on the target cluster shows periods of no increase during active backfill.
• RFS Reindexing Traffic graph shows discontinuities.
• number of RFS workers fluctuates dramatically and unexpectedly.

What are your migration environments?

• Managed Service to Managed Service
• Source: ElasticSearch 7.10 on AWS OpenSearch Service
• Target: OpenSearch 2.17 on AWS OpenSearch Service
• Data size: ~100GB
• Number of shards: 21 (with most data in 5 shards)

How can one reproduce the bug?

• Take a snapshot and perform console metadata migrate.
• Start backfill with 5 workers.
• Scale up to 21 workers.
• Further scale up to 42 workers.
• Monitor the following metrics during the process:

1. Target cluster size (pri.store.size on cat-indices)
2. TARGET CLUSTER Document SUM Count
3. RFS Reindexing Traffic
4. Number of RFS workers reporting in

What is the expected behavior?

• target cluster size should steadily increase or remain stable during backfill.
• document count should consistently increase during active backfill.
• RFS Reindexing Traffic graph should show a continuous line without cut-offs.
• number of RFS workers should remain stable at the set number (5, 21, or 42) without unexpected drops.

Do you have any additional context?

• target cluster size drops were verified through CloudWatch "Target Cluster Used Space" graph showing significant negative slopes.
• document count showed 4 instances of 0 slope, indicating no increase in total documents for extended periods.
• RFS Reindexing Traffic graph showed 3-4 instances of cut-offs and resumptions, with traffic ranging from approximately 500MB to 2GB.
• RFS worker count unexpectedly dropped from 21 to 5, 42 to 5, and 42 to 24 at various points.

Additional question: How can one better identify if the TARGET Cluster has been overloaded?

[chelma]

Thanks for reporting @jugal-chauhan. Can you clarify the impact you observed?

Just spitballing here, but:

• The number of searchable docs increases over the course of the migration, which is what I would expect.
• The cluster space used generally increases, with occasional dips. These dips appear in line with what I would expect with Lucene segment merges clearing out updated/deleted documents. These "extra" documents occur when RFS Workers retry a shard and overwrite the partial work of a previous attempt.
• The RFS worker count is also varying about as expected. We see the count staying around the desired numbers you indicated, with some noise, which can occur due to individual workers finishing shards at various rates and/or needing to retry due to a lease timeout.

[jugal-chauhan]

Thanks for your response and insights. After re-running the migration with a different data distribution (15GB across 55 shards instead of 100GB across 21 shards) and scaling RFS workers from 5 to 40, I did not observe the previously reported inconsistencies.

There were no gaps during my re-run for the 'RFS Reindexing Traffic'. The graph 'RFS workers reporting in' showed a consistent count of expected number of workers. No unexpected behavior noticed.

Given these findings, it appears that the unexpected behavior may have been influenced by the original shard distribution and workload characteristics. Since the issue did not reproduce under the new conditions, I am closing this issue for now.

If I encounter similar behavior again under controlled conditions, I'll reopen with additional details. Thanks again for the support!