ansible-scylla-node: use scylla_bootstrap_wait_time_sec for API port wait#539
Draft
cursor[bot] wants to merge 10 commits into
Draft
ansible-scylla-node: use scylla_bootstrap_wait_time_sec for API port wait#539cursor[bot] wants to merge 10 commits into
cursor[bot] wants to merge 10 commits into
Conversation
Check systemctl is-active scylla-server while polling nodetool during cluster health and replace-node UN waits, matching start_one_node.yml. Fixes scylladb/field-engineering#2109 Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Addresses Copilot review: systemctl is-active can return deactivating, unknown, etc. Fail for any state other than active/activating and include the actual state in the error message for diagnosis. Co-authored-by: Cursor <cursoragent@cursor.com>
* Fail fast when scylla-server stops during bootstrap health waits Check systemctl is-active scylla-server while polling nodetool during cluster health and replace-node UN waits, matching start_one_node.yml. Fixes scylladb/field-engineering#2109 Co-authored-by: Cursor <cursoragent@cursor.com> * ci: retrigger molecule after scylla-manager flake Co-authored-by: Cursor <cursoragent@cursor.com> * fix: allowlist active/activating states instead of blocklisting Addresses Copilot review: systemctl is-active can return deactivating, unknown, etc. Fail for any state other than active/activating and include the actual state in the error message for diagnosis. Co-authored-by: Cursor <cursoragent@cursor.com> * scylla-manager: automatically set Alternator credentials Newer Scylla Managers in combination with Alternator-enabled clusters, will attempt to retrieve the Alternator schema using Alternator credentials when performing a backup. In order to be able to perform a backup, we must set these credentials. Fixes https://scylladb.atlassian.net/browse/ANSROLES-17 Signed-off-by: Vincent Reniers <vincent.reniers@scylladb.com> Co-authored-by: Cursor <cursoragent@cursor.com> --------- Signed-off-by: Vincent Reniers <vincent.reniers@scylladb.com> Co-authored-by: manjots <manjot.singh@scylladb.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Manjot Singh <manjot@manjotsingh.net>
* Persist scylla-monitoring start/stop wrapper scripts on deploy.
Write /opt/scylla-monitoring/{start,stop}-monitoring.sh during monitoring
role deploy so operators can restart the stack without re-running ansible
or reconstructing start-all.sh arguments by hand.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Address Copilot review on monitoring wrapper scripts.
Quote paths and credentials, use conditional sudo for stop, move CQL
secrets to a 0600 env file sourced at runtime, and restore env vars for
ansible-driven starts.
Co-authored-by: Cursor <cursoragent@cursor.com>
* ci: retrigger molecule workflow
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
The assumption cannot be made that the hostname is DNS resolvable by all nodes in the cluster. It's more appropriate to require that the seed nodes are specified explicitly. Signed-off-by: Vincent Reniers <vincent.reniers@scylladb.com>
Removing the implicit scylla_seeds default meant the role would only surface an undefined-variable error deep into common.yml. Add an early explicit check so misconfiguration fails fast with a clear message instead. Also fix the commented example to use consistent groups['scylla'][0] bracket indexing and drop trailing whitespace. Co-authored-by: Cursor <cursoragent@cursor.com>
ansible-scylla-node: remove default scylla_seeds
* Persist iptables DROP rules via ansible-scylla-node variable Add iptables_drop_sources to ansible-scylla-node. When firewall_enabled is true, the role applies INPUT DROP rules for each listed source IP. Remove the ad-hoc iptables DROP task from replace_node.yml so blocking is owned by the role (and therefore by cluster vars), not by a one-off playbook side effect. Field Cloud replace jobs set iptables_drop_sources before replace and clear it after wiped nodes, then re-run ansible-scylla-node. * Address review: move iptables DROP into ansible-scylla-common Merge disable_firewall + iptables_drop_sources into handle_firewall.yml, gate DROP on not disable_firewall, drop block_replaced_node and stale comments. Co-authored-by: Cursor <cursoragent@cursor.com> * Clean up comments * ansible-scylla-node: bump API port wait timeout to 600s rockylinux9/10 Molecule hits Timeout when waiting for 127.0.0.1:10000 under loaded runners; same flaky class as prior readiness bumps. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com>
…wait The 'wait for the API port to come up on all nodes' task used a hardcoded 600s timeout, inconsistent with the surrounding bootstrap logic which allows scylla_bootstrap_wait_time_sec (default 7h) for node startup and cluster health checks. Under loaded CI runners the API port can take longer than 600s to come up even after the CQL port has already responded, causing flaky 'Timeout when waiting for 127.0.0.1:10000' failures unrelated to the change under test. Co-authored-by: Manjot Singh <ManjotS@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
wait for the API port to come up on all nodes(ansible-scylla-node/tasks/common.yml) used a hardcoded 600swait_fortimeout.scylla_bootstrap_wait_time_sec(default 25200s / 7h), matching the timeout philosophy already used by the neighboringWait for CQL portandwait for the cluster to become healthytasks in the same bootstrap flow.Context
PR #536's
build (debian12)Molecule job failed with:This happens in the
Deploy scylla nodesplay, entirely before theDeploy monitor nodesplay (which includes PR #536'sansible-scylla-monitoringchanges) ever runs, confirming it's pre-existing CI flakiness unrelated to PR #536's diff.The CQL port (9042) already responded for these nodes (via the separate
Wait for CQL porttask, which already tolerates up toscylla_bootstrap_wait_time_sec), but the API port (10000) check used a hardcoded 600s and timed out under a loaded CI runner. This same failure signature (previously at a 300s timeout) has recurred across several unrelated PRs; a prior fix bumped the hardcoded value from 300s to 600s, but that is apparently still insufficient under heavier CI load. This change removes the hardcoded value entirely in favor of the existing configurable bootstrap-wait variable, avoiding further timeout whack-a-mole.Test plan