[BUG] autofollow replication fails with rolling over index #1579
Description
What is the bug?
Cluster B is auto following Cluster A.
Cluster A has policy managed indexes.
Policy has rollover action with index alias + write index.
When rollover happens, the following error is logged:
[2025-09-05T17:22:21,959][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [dev-om02-r2-vlp.*** Cancelling the migration process.
[2025-09-05T17:22:22,039][ERROR][o.o.r.t.i.IndexReplicationTask] [dev-om02-r2-vlp.mce.minint.fr] [mce-index-w-002742] Unable to initiate restore call for mce-index-w-0
02742 from my-connection-alias:mce-index-w-002742
java.lang.IllegalStateException: alias [mce-index-alias] has more than one write index [mce-index-w-002740,mce-index-w-002742]
This is because write index change is not replicated. The replicate keeps the previous index.
How can one reproduce the bug?
Steps to reproduce the behavior:
- Define policy on the leader:
# curl https://dev-om01-r2-vlp.****:9200/_plugins/_ism/policies/policy_1?pretty --cert /etc/
opensearch/admin.pem --key /etc/opensearch/admin.key
{
"_id" : "policy_1",
"_version" : 1,
"_seq_no" : 0,
"_primary_term" : 1,
"policy" : {
"policy_id" : "policy_1",
"description" : "hot warm delete workflow",
"last_updated_time" : 1756229501227,
"schema_version" : 24,
"error_notification" : null,
"default_state" : "hot",
"states" : [
{
"name" : "hot",
"actions" : [
{
"retry" : {
"count" : 3,
"backoff" : "exponential",
"delay" : "1m"
},
"rollover" : {
"min_doc_count" : 100,
"min_index_age" : "2h",
"min_primary_shard_size" : "3mb",
"copy_alias" : false
}
}
],
"transitions" : [
{
"state_name" : "warm"
}
]
},
{
"name" : "warm",
"actions" : [
{
"retry" : {
"count" : 3,
"backoff" : "exponential",
"delay" : "1m"
},
"replica_count" : {
"number_of_replicas" : 1
}
},
{
"retry" : {
"count" : 3,
"backoff" : "exponential",
"delay" : "1m"
},
"allocation" : {
"require" : {
"temp" : "warm"
},
"include" : { },
"exclude" : { },
"wait_for" : false
}
}
],
"transitions" : [
{
"state_name" : "delete",
"conditions" : {
"min_index_age" : "2h"
}
}
]
},
{
"name" : "delete",
"actions" : [
{
"retry" : {
"count" : 3,
"backoff" : "exponential",
"delay" : "1m"
},
"delete" : { }
}
],
"transitions" : [ ]
}
],
"ism_template" : [
{
"index_patterns" : [
"mce-index-w*"
],
"priority" : 100,
"last_updated_time" : 1756229501226
}
]
}
}
- Define alias on write index:
root@dev-om01-r2:/home/debian# curl https://dev-om01-r1-vlp.******:9200/_cat/aliases?v --cert /etc/opensearch/admin.pem --key /etc/opensearch/admin.key alias index filter routing.index routing.search is_write_index
mce-index-alias mce-index-w-002738 - - - false
mce-index-alias mce-index-w-002739 - - - false
.opendistro-ism-managed-index-history-write .opendistro-ism-managed-index-history-2025.09.05-000063 - - - -
mce-index-alias mce-index-w-002741 - - - false
mce-index-alias mce-index-w-002740 - - - false
mce-index-alias mce-index-w-002743 - - - true
mce-index-alias mce-index-w-002742 - - - false
.kibana .kibana_3 - - - -
.kibana_-947961014_elbazsi .kibana_-947961014_elbazsi_1 - - - -
root@dev-om01-r2:/home/debian# curl https://dev-om01-r2-vlp.****:9200/_cat/aliases?v --cert /etc/opensearch/admin.pem --key /etc/opensearch/admin.key
alias index filter routing.index routing.search is_write_index
mce-index-alias mce-index-w-002739 - - - false
.kibana .kibana_1 - - - -
mce-index-alias mce-index-w-002740 - - - true
- Setup autofollow replication:
root@dev-om01-r2:/home/debian# curl 'https://dev-om01-r2-vlp.******:9200/_cluster/settings?pretty' -XGET --cert /etc/opensearch/admin.pem --key /etc/opensearch/admin.key
{
"persistent" : {
"cluster" : {
"remote" : {
"my-connection-alias" : {
"seeds" : [
"dev-om01-r1-vlp.***:9300",
"dev-om02-r1-vlp.***:9300",
"dev-om03-r1-vlp.***:9300"
]
}
}
}
},
"transient" : { }
}
root@dev-om01-r2:/home/debian# curl https://dev-om01-r2.***:9200/_plugins/_replication/autofollow_stats?pretty --cert /etc/opensearch/admin.pem --key /etc/opensearch/admin.key
{
"num_success_start_replication" : 10,
"num_failed_start_replication" : 0,
"num_failed_leader_calls" : 0,
"failed_indices" : [
"mce-index-w-002741",
"mce-index-w-002742",
"mce-index-w-002743"
],
"autofollow_stats" : [
{
"name" : "my-replication-rule",
"pattern" : "mce-index-w-*",
"num_success_start_replication" : 10,
"num_failed_start_replication" : 0,
"num_failed_leader_calls" : 0,
"failed_indices" : [
"mce-index-w-002741",
"mce-index-w-002742",
"mce-index-w-002743"
],
"last_execution_time" : 1757086672352
}
]
}
- See error
Replicated indexes are red:
root@dev-om01-r2:/home/debian# curl https://dev-om01-r2-vlp.mce.minint.fr:9200/_cat/indices/mce-index-w*?v --cert /etc/opensearch/admin.pem --key /etc/opensearch/admin.key
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
red open mce-index-w-002341 7ynBz_gFTSGqCdGuWujhxA 2 1
red open mce-index-w-002738 d02C_wDdSLGPZkWyUaeNFw 2 1 899 0 822kb 411kb
red open mce-index-w-002737 44VuBLnXT1eUrwI3vSBELQ 2 1 312869 0 88.4mb 44.2mb
red open mce-index-w-002739 I3utC4FLS6iRvdGKBZ5n-Q 2 1 294325 0 82.9mb 41.4mb
red open mce-index-w-002728 -HbDOVNlTgSL1utYP4vY6g 2 1
red open mce-index-w-002734 VqV94DxXQ0OU8uWgVHerYg 2 1
red open mce-index-w-002723 sNSVAt3PSKGjpX34xFFiBg 2 1
red open mce-index-w-002733 K08N_l49QUmDp8-ezRfJJg 2 1
red open mce-index-w-002736 rt6YFW5MS-yfJHQCQh4UTg 2 1
red open mce-index-w-002735 rzDF-CPQQumIJwwcJ1C-jA 2 1
red open mce-index-w-002730 PA4qHllCQKS6R0UEk7h9Tw 2 1
red open mce-index-w-002740 HCOzL00pQXiO1TalZTOHfg 2 1 0 0 416b 208b
red open mce-index-w-002732 zFSjIkW-Qli-Oc0u_euWLg 2 1
red open mce-index-w-002731 v13AZlWTTDu_Ke8KySo-2g 2 1
What is the expected behavior?
- Replicated indexes should be green
- write index change should be replicated
What is your host/environment?
- OS: [Debian 11.11]
- Version [opensearch 3.2]
- Plugins
root@dev-om01-r2:/home/debian# /usr/share/opensearch/bin/opensearch-plugin list
opensearch-alerting
opensearch-anomaly-detection
opensearch-asynchronous-search
opensearch-cross-cluster-replication
opensearch-custom-codecs
opensearch-flow-framework
opensearch-geospatial
opensearch-index-management
opensearch-job-scheduler
opensearch-knn
opensearch-ltr
opensearch-ml
opensearch-neural-search
opensearch-notifications
opensearch-notifications-core
opensearch-observability
opensearch-reports-scheduler
opensearch-search-relevance
opensearch-security
opensearch-security-analytics
opensearch-skills
opensearch-sql
opensearch-system-templates
opensearch-ubi
query-insights
repository-s3
Do you have any screenshots?
If applicable, add screenshots to help explain your problem.
Do you have any additional context?
Add any other context about the problem.