Only 1 rack per DC with 3 AZ defined

[cezarmoise]

This does not happen in 2024.2 for some reason, I could not find code differences.
It's not related to rack_aware_loader, it also happens before that change.

Nodetool status

Datacenter: eu-west-2scylla_node_west
=====================================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address                                Load     Tokens Owns Host ID                              Rack
UN 2a05:d01c:964:7d02:44d8:fd94:6565:742  18.61 GB 256    ?    9a6f532f-4c6a-47f8-84f6-f091edd77207 2c  
UN 2a05:d01c:964:7d02:6617:7bc6:d20f:b39f 18.49 GB 256    ?    aa6338dd-bce3-41e7-a455-0453f39fee02 2c  
UN 2a05:d01c:964:7d02:aa84:9098:3df7:b67d 18.60 GB 256    ?    fb6aaf5b-edcf-434a-b311-3d3b2ac34051 2c  
UN 2a05:d01c:964:7d02:be0a:9f3e:36c2:711b 18.44 GB 256    ?    84fd6986-8b1f-4d73-a875-cdf1042a9396 2c  
UN 2a05:d01c:964:7d02:dfc3:ab90:5e6b:20e7 18.52 GB 256    ?    33c58964-05da-4e8d-86c0-60886b1796d0 2c  
DN 2a05:d01c:964:7d02:eff2:6989:7b97:cab  18.47 GB 256    ?    9cb7ae9f-3bf6-4986-9fe9-d75b58a55a83 2c  
Datacenter: eu-westscylla_node_west
===================================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address                                 Load     Tokens Owns Host ID                              Rack
UN 2a05:d018:12e3:f002:1276:503a:cc77:3a61 18.83 GB 256    ?    f89ae63b-8157-49fa-a996-d4208aeaa8a0 1c  
UN 2a05:d018:12e3:f002:5c59:8414:c097:aa26 18.70 GB 256    ?    6e129110-b84f-46c2-943f-cad33c35ce06 1c  
UN 2a05:d018:12e3:f002:786d:6a52:7477:dadb 18.62 GB 256    ?    6fd6e611-2f06-4db5-b63f-906a42698125 1c  
UN 2a05:d018:12e3:f002:a875:2aa0:f5e:2981  18.73 GB 256    ?    8faafe06-8bb9-422c-9c9a-3e215166a4e2 1c  
UN 2a05:d018:12e3:f002:b951:986:51e1:ef8   18.54 GB 256    ?    84cb395c-29c5-41e1-b2ec-31033e99155e 1c  
UN 2a05:d018:12e3:f002:c467:7827:f900:6b0c 18.88 GB 256    ?    cf416917-caed-4457-958a-5e369f062261 1c  

Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless

config

rack_aware_loader: true
availability_zone: 'a,b,c'
n_db_nodes: '6 6'
n_loaders: '2 1'
region_aware_loader: true
n_monitor_nodes: 1

https://argus.scylladb.com/tests/scylla-cluster-tests/fd7140c2-a358-4d4e-90db-154c5369ba79

[cezarmoise]

Looking back in time, even this run from 2024-05-11 has the same issue
https://argus.scylladb.com/tests/scylla-cluster-tests/073570cd-26dd-48aa-9601-5f141fd862ba

< t:2024-05-11 23:42:43,363 f:base.py         l:230  c:RemoteLibSSH2CmdRunner p:DEBUG > Datacenter: eu-west-2scylla_node_west
< t:2024-05-11 23:42:43,363 f:base.py         l:230  c:RemoteLibSSH2CmdRunner p:DEBUG > =====================================
< t:2024-05-11 23:42:43,364 f:base.py         l:230  c:RemoteLibSSH2CmdRunner p:DEBUG > Status=Up/Down
< t:2024-05-11 23:42:43,364 f:base.py         l:230  c:RemoteLibSSH2CmdRunner p:DEBUG > |/ State=Normal/Leaving/Joining/Moving
< t:2024-05-11 23:42:43,364 f:base.py         l:230  c:RemoteLibSSH2CmdRunner p:DEBUG > -- Address     Load    Tokens Owns Host ID                              Rack
< t:2024-05-11 23:42:43,364 f:base.py         l:230  c:RemoteLibSSH2CmdRunner p:DEBUG > UN 10.3.10.108 4.50 MB 256    ?    08d6395f-53a3-4769-8f80-9b3f230ccf46 2c  
< t:2024-05-11 23:42:43,364 f:base.py         l:230  c:RemoteLibSSH2CmdRunner p:DEBUG > UN 10.3.10.116 3.97 MB 256    ?    8c727408-bb7c-40d8-87ed-a6dd717276c3 2c  
< t:2024-05-11 23:42:43,364 f:base.py         l:230  c:RemoteLibSSH2CmdRunner p:DEBUG > UN 10.3.11.84  4.07 MB 256    ?    5edf27a6-4892-4e8f-8a18-ff7c1f5dfaa6 2c  
< t:2024-05-11 23:42:43,364 f:base.py         l:230  c:RemoteLibSSH2CmdRunner p:DEBUG > UN 10.3.8.100  4.29 MB 256    ?    19264265-ad15-4377-ba07-9d77bd05714f 2c  
< t:2024-05-11 23:42:43,364 f:base.py         l:230  c:RemoteLibSSH2CmdRunner p:DEBUG > UN 10.3.8.204  3.92 MB 256    ?    fc461748-1e18-4e58-ab87-caed88420c29 2c  
< t:2024-05-11 23:42:43,364 f:base.py         l:230  c:RemoteLibSSH2CmdRunner p:DEBUG > UN 10.3.9.139  3.83 MB 256    ?    209d08a7-1a07-4850-b4da-e6288eb58beb 2c  
< t:2024-05-11 23:42:43,364 f:base.py         l:230  c:RemoteLibSSH2CmdRunner p:DEBUG > Datacenter: eu-westscylla_node_west
< t:2024-05-11 23:42:43,364 f:base.py         l:230  c:RemoteLibSSH2CmdRunner p:DEBUG > ===================================
< t:2024-05-11 23:42:43,364 f:base.py         l:230  c:RemoteLibSSH2CmdRunner p:DEBUG > Status=Up/Down
< t:2024-05-11 23:42:43,364 f:base.py         l:230  c:RemoteLibSSH2CmdRunner p:DEBUG > |/ State=Normal/Leaving/Joining/Moving
< t:2024-05-11 23:42:43,364 f:base.py         l:230  c:RemoteLibSSH2CmdRunner p:DEBUG > -- Address     Load    Tokens Owns Host ID                              Rack
< t:2024-05-11 23:42:43,364 f:base.py         l:230  c:RemoteLibSSH2CmdRunner p:DEBUG > UN 10.4.10.241 3.83 MB 256    ?    6be17be8-b135-46ef-913a-fc2bfab4d79e 1c  
< t:2024-05-11 23:42:43,364 f:base.py         l:230  c:RemoteLibSSH2CmdRunner p:DEBUG > UN 10.4.11.145 3.70 MB 256    ?    9f90a048-8a30-421f-92b1-f442fd3ae23c 1c  
< t:2024-05-11 23:42:43,364 f:base.py         l:230  c:RemoteLibSSH2CmdRunner p:DEBUG > UN 10.4.8.209  3.65 MB 256    ?    185501b1-5f71-4787-a8cd-0b35b8faa783 1c  
< t:2024-05-11 23:42:43,364 f:base.py         l:230  c:RemoteLibSSH2CmdRunner p:DEBUG > UN 10.4.8.213  3.83 MB 256    ?    12c4bfa0-7213-4e5a-9ba5-5381a3b2aa4c 1c  
< t:2024-05-11 23:42:43,364 f:base.py         l:230  c:RemoteLibSSH2CmdRunner p:DEBUG > UN 10.4.8.4    3.75 MB 256    ?    70d54764-ed6d-470c-8d8e-f3c92210c71f 1c  
< t:2024-05-11 23:42:43,364 f:base.py         l:230  c:RemoteLibSSH2CmdRunner p:DEBUG > UN 10.4.9.104  3.92 MB 256    ?    72b467bb-9d71-4987-9476-298da46ee4ee 1c  
< t:2024-05-11 23:42:43,364 f:base.py         l:230  c:RemoteLibSSH2CmdRunner p:DEBUG > 
< t:2024-05-11 23:42:43,364 f:base.py         l:230  c:RemoteLibSSH2CmdRunner p:DEBUG > Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless

[fruch]

AZ: c is being passed for tier1 trigger, hence only c is use

we need to drop that from the tier1 trigger

[vponomaryov]

AZ: c is being passed for tier1 trigger, hence only c is use

we need to drop that from the tier1 trigger

@cezarmoise you looked at the wrong place for the configuration.
@fruch is right, the az config is defined to be only c.

From logs:

06:09:36  < t:2025-04-19 03:09:36,179 f:sct_config.py   l:2218 c:sdcm.sct_config      p:INFO  > availability_zone: c

And it is set in CI job parameters here: https://jenkins.scylladb.com/job/scylla-master/job/tier1/job/longevity-multidc-schema-topology-changes-12h-test/67/parameters/