Skip to content

[Bug]: MDR 4.22 deployment is broken. #14903

@petr-balogh

Description

@petr-balogh

Discussed here:
https://ibm-systems-storage.slack.com/archives/C06EEMBNKB4/p1776199081960959?thread_ts=1776152954.296099&cid=C06EEMBNKB4

@shylesh Kumar M I think this is related to recent changes to client provider rdr scenario. I see some changes also done from client provider job to propagate problematic config value dr_cluster_relations which failing MDR job.

b40f3a5f (Shylesh Kumar Mohan 2026-03-25 16:11:09 +0000  620)         // Example:
cb40f3a5f (Shylesh Kumar Mohan 2026-03-25 16:11:09 +0000  621)         // MULTICLUSTER:
cb40f3a5f (Shylesh Kumar Mohan 2026-03-25 16:11:09 +0000  622)         //  dr_cluster_relations:
cb40f3a5f (Shylesh Kumar Mohan 2026-03-25 16:11:09 +0000  623)         //    - ["cluster1", "cluster3"]
cb40f3a5f (Shylesh Kumar Mohan 2026-03-25 16:11:09 +0000  624)         //    - ["cluster3", "cluster4"]
cb40f3a5f (Shylesh Kumar Mohan 2026-03-25 16:11:09 +0000  625)         def config = readYaml text: yamlTextConfigStr
cb40f3a5f (Shylesh Kumar Mohan 2026-03-25 16:11:09 +0000  626)
cb40f3a5f (Shylesh Kumar Mohan 2026-03-25 16:11:09 +0000  627)         def pairs = config.MULTICLUSTER.dr_cluster_relations

And related ocs-ci code here: #14815
Done by @pratik Surve.

The issue is that in our config for MDR job - we don’t propagate anyhow this value to config config.MULTICLUSTER.get(“dr_cluster_relations”, [])

From traceback:

2026-04-14 01:02:13          dr_cluster_relations = config.MULTICLUSTER.get("dr_cluster_relations", [])
2026-04-14 01:02:13          if dr_cluster_relations:
2026-04-14 01:02:13              current_managed_clusters_list = [
2026-04-14 01:02:13                  (
2026-04-14 01:02:13                      f"{constants.HYPERSHIFT_ADDON_DISCOVERYPREFIX}-{item}"
2026-04-14 01:02:13                      if is_hosted_cluster(cluster_name=item)
2026-04-14 01:02:13                      else item
2026-04-14 01:02:13                  )
2026-04-14 01:02:13                  for item in dr_cluster_relations[0]
2026-04-14 01:02:13              ]
2026-04-14 01:02:13      
2026-04-14 01:02:13          # ignore local-cluster here
2026-04-14 01:02:13          for i in managed_clusters:
2026-04-14 01:02:13              if (
2026-04-14 01:02:13                  i["metadata"]["name"] != constants.ACM_LOCAL_CLUSTER
2026-04-14 01:02:13                  and i["metadata"]["name"] in current_managed_clusters_list
2026-04-14 01:02:13              ):
2026-04-14 01:02:13                  cluster_set.append(i["metadata"]["labels"][constants.ACM_CLUSTERSET_LABEL])
2026-04-14 01:02:13          if all(x == cluster_set[0] for x in cluster_set):
2026-04-14 01:02:13  >           logger.info(f"Found the unique clusterset {cluster_set[0]}")
2026-04-14 01:02:13  E           IndexError: list index out of range
2026-04-14 01:02:13  
2026-04-14 01:02:13  ocs_ci/helpers/dr_helpers.py:2579: IndexError

So what is happening that dr_cluster_relations is empty list as we don’t propagate it or set anyhow for MDR deployment. And current_managed_clusters_list is also empty list because of this because we don’t populate it here:

if dr_cluster_relations:
        current_managed_clusters_list = [
            (
                f"{constants.HYPERSHIFT_ADDON_DISCOVERYPREFIX}-{item}"
                if is_hosted_cluster(cluster_name=item)
                else item
            )
            for item in dr_cluster_relations[0]
        ]

Then we don’t appending anything to cluster_set which is also empty list defined earlier in function:

    for i in managed_clusters:
        if (
            i["metadata"]["name"] != constants.ACM_LOCAL_CLUSTER
            and i["metadata"]["name"] in current_managed_clusters_list
        ):
            cluster_set.append(i["metadata"]["labels"][constants.ACM_CLUSTERSET_LABEL])
And because of cluster_set is empty at this point we are failing with index error:
2026-04-14 01:02:13          if all(x == cluster_set[0] for x in cluster_set):
2026-04-14 01:02:13  >           logger.info(f"Found the unique clusterset {cluster_set[0]}")
2026-04-14 01:02:13  E           IndexError: list index out of range
2026-04-14 01:02:13  
2026-04-14 01:02:13  ocs_ci/helpers/dr_helpers.py:2579: IndexError

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions