enforce RF-rack-valid keyspaces for Xcloud clusters

[mykaul]

Exact same request as in https://github.com/scylladb/siren/issues/13035 - make sure that for X-cloud clusters, we enforce correct rf/rack topology by default.

CC @tzach , @dawmd , @pehala (who's been working on some of this already, I believe, making sure all SCT use valid topology)

[mykaul]

@pehala - @temichus is on PTO, please take it with someone in his team or just implement it:
rf_rack_valid_keyspaces: true in scylla.yaml , it's important for 2025.1.0.

[pehala]

@pehala - @temichus is on PTO, please take it with someone in his team or just implement it:
rf_rack_valid_keyspaces: true in scylla.yaml , it's important for 2025.1.0.

I will start working on this, for 2025.1.0 I would choose a few test to use it feature on, and for master I would include it everywhere we switched. Does that sound good?

[fruch]

@pehala - @temichus is on PTO, please take it with someone in his team or just implement it:
rf_rack_valid_keyspaces: true in scylla.yaml , it's important for 2025.1.0.

I will start working on this, for 2025.1.0 I would choose a few test to use it feature on, and for master I would include it everywhere we switched. Does that sound good?

FYI, while working on this in dtest, seems like the flow of decommission is not yet fully defined.

I.e. we can't decommission a node before RF is changed, and we can't change RF, without decommissioning a node (the only way to change the number of racks)

Enabling it on all SCT cases, would break all decommission nemesis

[pehala]

FYI, while working on this in dtest, seems like the flow of decommission is not yet fully defined.

I.e. we can't decommission a node before RF is changed, and we can't change RF, without decommissioning a node (the only way to change the number of racks)

Enabling it on all SCT cases, would break all decommission nemesis

That seems like it is big issue we should resolve, is it reported anywhere?

[dawmd]

FYI, while working on this in dtest, seems like the flow of decommission is not yet fully defined.
I.e. we can't decommission a node before RF is changed, and we can't change RF, without decommissioning a node (the only way to change the number of racks)
Enabling it on all SCT cases, would break all decommission nemesis

That seems like it is big issue we should resolve, is it reported anywhere?

It's not. We discussed some issues related to topology operations like adding a new DC in a meeting, but we focused mostly on how keyspaces will behave. I'll create an issue for topology operations so we can track problems people run into.

[aleksbykov]

The same and for adding new node with new rack. if ks configured with rf = 3 and cluster configured with 3 racks, bootstrap new node with rack4 will failed , and it also impossible to increase first rf to rf=4, because you will get error of cql query, that you don't have enough racks, the possible workarounds:

1. set rf =1, add new node with rack4 and then increase rf by 1 while it not reached rf = 4 (working)
2. add new node to existent rack. stop the node, change rack name, and then start it and increase rf to rf = 4(need to check)

[dawmd]

The same and for adding new node with new rack. if ks configured with rf = 3 and cluster configured with 3 racks, bootstrap new node with rack4 will failed , and it also impossible to increase first rf to rf=4, because you will get error of cql query, that you don't have enough racks, the possible workarounds:

1. set rf =1, add new node with rack4 and then increase rf by 1 while it not reached rf = 4 (working)

2. add new node to existent rack. stop the node, change rack name, and then start it and increase rf to rf = 4(need to check)

We discussed it today and the plan is to implement scylladb/scylladb#23426.

As for decommission, I think we can modify the plan in the mentioned issue to enforce rebinding during decommission if there's a rack that the keyspace isn't bound to yet. I'll try to provide an answer tomorrow. For now, keep track of which tests are affected by the problem and are impossible to adjust, but ignore them and continue with the rest. Worst-case scenario, we'll explicitly disable the configuration option in those tests and leave them be until there's a proper solution. CC @kostja @piodul

[pehala]

We discussed it today and the plan is to implement scylladb/scylladb#23426.

@mykaul Not sure how much sense it would make to test rack == #racks for 2025.1.0 if boostrap/decomission is limited with the switch. I would expect a lot of failures

[yarongilor]

what about alter keyspace RF?

[dawmd]

The same and for adding new node with new rack. if ks configured with rf = 3 and cluster configured with 3 racks, bootstrap new node with rack4 will failed , and it also impossible to increase first rf to rf=4, because you will get error of cql query, that you don't have enough racks, the possible workarounds:

1. set rf =1, add new node with rack4 and then increase rf by 1 while it not reached rf = 4 (working)

2. add new node to existent rack. stop the node, change rack name, and then start it and increase rf to rf = 4(need to check)

We discussed it today and the plan is to implement scylladb/scylladb#23426.

As for decommission, I think we can modify the plan in the mentioned issue to enforce rebinding during decommission if there's a rack that the keyspace isn't bound to yet. I'll try to provide an answer tomorrow. For now, keep track of which tests are affected by the problem and are impossible to adjust, but ignore them and continue with the rest. Worst-case scenario, we'll explicitly disable the configuration option in those tests and leave them be until there's a proper solution. CC @kostja @piodul

Unfortunately, decommission will most likely need to be handled as a separate task.