ansible-scylla-node: increase API port wait timeout to reduce CI flakiness#535
Draft
cursor[bot] wants to merge 9 commits into
Draft
ansible-scylla-node: increase API port wait timeout to reduce CI flakiness#535cursor[bot] wants to merge 9 commits into
cursor[bot] wants to merge 9 commits into
Conversation
Check systemctl is-active scylla-server while polling nodetool during cluster health and replace-node UN waits, matching start_one_node.yml. Fixes scylladb/field-engineering#2109 Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Addresses Copilot review: systemctl is-active can return deactivating, unknown, etc. Fail for any state other than active/activating and include the actual state in the error message for diagnosis. Co-authored-by: Cursor <cursoragent@cursor.com>
* Fail fast when scylla-server stops during bootstrap health waits Check systemctl is-active scylla-server while polling nodetool during cluster health and replace-node UN waits, matching start_one_node.yml. Fixes scylladb/field-engineering#2109 Co-authored-by: Cursor <cursoragent@cursor.com> * ci: retrigger molecule after scylla-manager flake Co-authored-by: Cursor <cursoragent@cursor.com> * fix: allowlist active/activating states instead of blocklisting Addresses Copilot review: systemctl is-active can return deactivating, unknown, etc. Fail for any state other than active/activating and include the actual state in the error message for diagnosis. Co-authored-by: Cursor <cursoragent@cursor.com> * scylla-manager: automatically set Alternator credentials Newer Scylla Managers in combination with Alternator-enabled clusters, will attempt to retrieve the Alternator schema using Alternator credentials when performing a backup. In order to be able to perform a backup, we must set these credentials. Fixes https://scylladb.atlassian.net/browse/ANSROLES-17 Signed-off-by: Vincent Reniers <vincent.reniers@scylladb.com> Co-authored-by: Cursor <cursoragent@cursor.com> --------- Signed-off-by: Vincent Reniers <vincent.reniers@scylladb.com> Co-authored-by: manjots <manjot.singh@scylladb.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Manjot Singh <manjot@manjotsingh.net>
* Persist scylla-monitoring start/stop wrapper scripts on deploy.
Write /opt/scylla-monitoring/{start,stop}-monitoring.sh during monitoring
role deploy so operators can restart the stack without re-running ansible
or reconstructing start-all.sh arguments by hand.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Address Copilot review on monitoring wrapper scripts.
Quote paths and credentials, use conditional sudo for stop, move CQL
secrets to a 0600 env file sourced at runtime, and restore env vars for
ansible-driven starts.
Co-authored-by: Cursor <cursoragent@cursor.com>
* ci: retrigger molecule workflow
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
The assumption cannot be made that the hostname is DNS resolvable by all nodes in the cluster. It's more appropriate to require that the seed nodes are specified explicitly. Signed-off-by: Vincent Reniers <vincent.reniers@scylladb.com>
Removing the implicit scylla_seeds default meant the role would only surface an undefined-variable error deep into common.yml. Add an early explicit check so misconfiguration fails fast with a clear message instead. Also fix the commented example to use consistent groups['scylla'][0] bracket indexing and drop trailing whitespace. Co-authored-by: Cursor <cursoragent@cursor.com>
ansible-scylla-node: remove default scylla_seeds
…iness Co-authored-by: Manjot Singh <ManjotS@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
wait for the API port to come up on all nodestask timeout from 300s to 600s inansible-scylla-node/tasks/common.yml.Context
PR #532's Molecule
buildjobs (debian11/12, ubuntu2204/2404, rockylinux9/10) failed with:This happens during the
convergescenario, in thewait for the API port to come up on all nodestask inansible-scylla-node/tasks/common.yml, before any of PR #532's changes (which only touchansible-scylla-node/tasks/upgrade/post_upgrade.ymland are not exercised by this scenario) run. The CQL port already responded (after retries) but the scylla-server API port took longer than the hardcoded 300s under loaded CI runners. This confirms it's pre-existing CI flakiness, not a regression introduced by PR #532.Two other open PRs (#533, #534) independently identified and fixed the exact same flakiness for other unrelated PRs, but targeted
masterinstead ofdev. This PR applies the same fix againstdev.Test plan