fix(disrupt_terminate_and_replace_node): raise critical event on failure #10403
Conversation
If the nemesis cannot leave the cluster in the topological state it was before it should raise a critical error so the test can be stopped. Add new event for topology failures `TopologyFailureEvent` refs: scylladb#9918
|
Why is it needed? |
By continuing, it leads to issues with other nemesis that affect topology, like GrowShrinkCluster. |
|
Another option is to refactor this nemesis to make sure we move to the part adding node, regardless of what was failing. Either way, we can't accept a nemesis removing a node and not adding a new node |
There was a problem hiding this comment.
@cezarmoise you look at wrong place and try to fix wrong issue.
In your test: https://argus.scylladb.com/tests/scylla-cluster-tests/0b0e042d-60a7-4dad-832d-4e38f2e5a5e9
see that longevity-parallel-topology-schema--db-node-0b0e042d-6 was already decommissioned during disrupt_decommission_streaming_err nemesis and we never got back to initial cluster node count - somehow SCT didn't detect problem with missing node. Please investigate this issue and propose fix for that.
If the nemesis cannot leave the cluster in the topological state it was before it should raise a critical error so the test can be stopped.
Add new event for topology failures
TopologyFailureEvent.refs: #9918
Testing
PR pre-checks (self review)
backportlabelsReminders
sdcm/sct_config.py)unit-test/folder)