[Feature] Guardrails in MetadataMigration Tool to Preserve Routing-Related Settings

[camerondurham]

TLDR: When using MetadataMigration tool with indices that have routing_partition_size configured, the routing distribution is not preserved due to missing number_of_routing_shards setting. Reproduction of the OpenSearch part of the issue: https://github.com/camerondurham/bug-repro-opensearch-routing

What is the issue/bug?

The configured routing partition size (5) is not being respected during bulk writes to the destination cluster. Documents are being distributed across fewer shards than configured (1-2 shards instead of 5), causing potential hotspotting issues. This is happening because the number_of_routing_shards doesn't appear to be set when copying metadata and creating the index on destination clusters.

What are your migration environments?

• AWS managed OpenSearch 1.3.x
• 6 x i3.4xlarge.search (Data nodes)
• 3 x c5.large.search (Master nodes)

Both cluster indexes configured with routing_partition_size: 5

How can one reproduce the issue/bug?

1. Create a snapshot repository
2. Use MetadataMigration tool to copy settings from source to destination cluster
3. Execute reindex-from-snapshot operation
4. Observe shard distribution patterns - documents will be concentrated in 1-2 shards instead of being distributed across 5 shards as configured

What is the expected behavior?

Documents should be evenly distributed across all 5 shards as specified by the routing_partition_size setting to prevent hotspotting. The shard distribution should match the source cluster's distribution pattern.

Do you have any additional context?

This caused errors for us when trying to migrate, since docs on routes previously spread across 5 shards were concentrated on 1, causing errors like:

"error": {
  "type": "illegal_argument_exception",
  "reason": "Number of documents in the index can't exceed [2147483519]"
}

Root Cause:
The MetadataMigration tool doesn't copy both required settings when routing_partition_size is present:

• settings.index.number_of_shards (copied)
• settings.index.number_of_routing_shards (not copied)

Resolution:
The issue can be resolved by explicitly setting both required settings during index creation, rather than relying on the MetadataMigration tool. This is related to some not well documented behavior inherited from Elasticsearch 7.x (elastic/elasticsearch #48863) that persists in OpenSearch 2.x (opensearch-project/OpenSearch #17472).

Verification Analysis:

Before fix: Source cluster shows 5-shard distribution (47,48,49,50,51), while target shows 2-shard distribution (5,6)
After fix: Both source and target show identical 5-shard distribution (47,48,49,50,51)

I reproduced the OpenSearch part of the issue here, in both OS 1.x and 2.x: https://github.com/camerondurham/bug-repro-opensearch-routing

[peternied]

@camerondurham Thanks for filing this issue. @mikaylathompson would you mind looking into this when you've got some cycles?

[mikaylathompson]

Thank you for your investigation and the very helpful write-up, @camerondurham ! I'm very curious why number_of_routing_shards wasn't copied from the original cluster by the metadata tool, and that's what I'm looking into now.

A few questions related to that:

• was the OpenSearch 1.3 cluster the source or the target for the metadata?
• do you happen to know whether the number_of_routing_shards property was explicitly set on the original cluster?

[camerondurham]

You're very welcome!

was the OpenSearch 1.3 cluster the source or the target for the metadata?

Yes, you're right, OS 1.3 was the source cluster.

do you happen to know whether the number_of_routing_shards property was explicitly set on the original cluster?

It was, we have a custom lambda that sets this value on index creation.

However, it looks like this setting isn't exposed to clients through the OpenSearch $INDEX/_settings API. (see: Reference: Source Cluster /_settings)

---

Thanks again for looking into the root cause. While investigating further, I found that this is related to an ongoing / unresolved problem within the OpenSearch project: opensearch-project/OpenSearch#17472 (comment).

Given that this caused our test migration to fail, I believe this may be an issue for other users who require custom routing behavior during migrations.

I have a few ideas about potential paths forward until the root cause is resolved in opensearch-project/OpenSearch#17472:

1. Add documentation to migrations tool warning about this limitation, suggesting clients create indexes manually to preserve routing behavior
2. Add option to metadata migration to explicitly set number_of_routing_shards when creating new OpenSearch index (defaulting to number_of_shards if not specified)
3. Add validation that warns when routing_partition_size is present but number_of_routing_shards isn't specified

Would love to hear your thoughts on which approach would be most aligned with the project's goals. I'm happy to contribute to the implementation of any of these solutions!

Reference: Source Cluster /_settings

For example, these are the only routing related settings when you hit the OS _settings API.

GET source_index/_settings
{
  "source_index" : {
    "settings" : {
      "index" : {
        ...
        "number_of_shards" : "90",
        "routing_partition_size" : "5",
        ...
      }
    }
  }
}

[peternied]

@camerondurham Really appreciate the depth you are going in on this issue. Ideally, if you could write a test and/or fix for OpenSearch core then this problem would go away once that version was more widely available. Please tag me on any PR and I'll help assist in that process in that repo.

More immediately, during Metadata's evaluation workflow we could detect if an index uses custom routing, then prompt a warning or/and supply a correction to the index setting to include number_of_routing_shards.