fix(nemesis): add support ipv6 for refuse connection for banned node #10594
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Isn't it just a Scylla issue? Even under high load, shouldn't other nodes know that one is down within 10 minutes? |
aleksbykov
force-pushed
the
fix-10434-increase-timeout-to-wait-down
branch
8 times, most recently
from
912fb11 to
a06b7e1
Compare
aleksbykov
changed the title
i found the problem. It was not in timeout, it was related to ipv6. |
|
Additional staging job is running |
soyacz
reviewed
Apr 11, 2025
| target_node.log.debug("Send signal SIGSTOP to scylla process on node %s", target_node.name) | ||
| target_node.remoter.sudo("pkill --signal SIGSTOP -e scylla", timeout=60) | ||
| yield | ||
| target_node.log.debug("Send signal SIGCONT to scylla process on node %s", target_node.name) | ||
| target_node.remoter.sudo(cmd="pkill --signal SIGCONT -e scylla", timeout=60) | ||
|
|
||
|
|
||
| @contextlib.contextmanager | ||
| def block_loaders_payload_for_scylla_node(scylla_node: BaseNode, loader_nodes: list[BaseNode]): |
There was a problem hiding this comment.
please add docstring why this is needed
| @@ -48,4 +76,6 @@ def is_node_removed_from_cluster(removed_node: BaseNode, verification_node: Base | |||
|
|
|||
| def is_node_seen_as_down(down_node: BaseNode, verification_node: BaseNode) -> bool: | |||
| LOGGER.debug("Verification node %s", verification_node.name) | |||
| return down_node not in verification_node.parent_cluster.get_nodes_up_and_normal(verification_node) | |||
| nodes_status = verification_node.parent_cluster.get_nodetool_status(verification_node, dc_aware=False) | |||
| down_node_status = nodes_status.get(down_node.ip_address) | |||
There was a problem hiding this comment.
shouln't use down_node.listen_address?
There was a problem hiding this comment.
It will be the same as ip_address and we use it everywhere with nodetool_status
disrupt_refuse_connection_with_* nemesises doesn't support ipv6. - Added command for blocking ports for ipv6 stack. When node is banned and alive, c-s/s-b could connect to it and failed with critical error, because banned node return that other node cluster is down. - Added new node_operation: block_loader_workload_for_scyllanode. This allow to block connections to scylla node from loaders and aboid critical error of c-s/s-b if them connect to banned node and failed to run Fixes: scylladb#10434
aleksbykov
force-pushed
the
fix-10434-increase-timeout-to-wait-down
branch
from
a06b7e1 to
2c4e417
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
disrupt_refuse_connection_with_* nemesises doesn't support ipv6.
When node is banned and alive, c-s/s-b could connect to it
and failed with critical error, because banned node return
that other node cluster is down.
This allow to block connections to scylla node from loaders
and avoid critical error of c-s/s-b if them connect to
banned node and failed to run
Fixes: #10434
Testing
PR pre-checks (self review)
backportlabelsReminders
sdcm/sct_config.py)unit-test/folder)