ansible-scylla-node: increase API port wait timeout to reduce CI flakiness#534
Draft
cursor[bot] wants to merge 9 commits into
Draft
ansible-scylla-node: increase API port wait timeout to reduce CI flakiness#534cursor[bot] wants to merge 9 commits into
cursor[bot] wants to merge 9 commits into
Conversation
Check systemctl is-active scylla-server while polling nodetool during cluster health and replace-node UN waits, matching start_one_node.yml. Fixes scylladb/field-engineering#2109 Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Addresses Copilot review: systemctl is-active can return deactivating, unknown, etc. Fail for any state other than active/activating and include the actual state in the error message for diagnosis. Co-authored-by: Cursor <cursoragent@cursor.com>
* Fail fast when scylla-server stops during bootstrap health waits Check systemctl is-active scylla-server while polling nodetool during cluster health and replace-node UN waits, matching start_one_node.yml. Fixes scylladb/field-engineering#2109 Co-authored-by: Cursor <cursoragent@cursor.com> * ci: retrigger molecule after scylla-manager flake Co-authored-by: Cursor <cursoragent@cursor.com> * fix: allowlist active/activating states instead of blocklisting Addresses Copilot review: systemctl is-active can return deactivating, unknown, etc. Fail for any state other than active/activating and include the actual state in the error message for diagnosis. Co-authored-by: Cursor <cursoragent@cursor.com> * scylla-manager: automatically set Alternator credentials Newer Scylla Managers in combination with Alternator-enabled clusters, will attempt to retrieve the Alternator schema using Alternator credentials when performing a backup. In order to be able to perform a backup, we must set these credentials. Fixes https://scylladb.atlassian.net/browse/ANSROLES-17 Signed-off-by: Vincent Reniers <vincent.reniers@scylladb.com> Co-authored-by: Cursor <cursoragent@cursor.com> --------- Signed-off-by: Vincent Reniers <vincent.reniers@scylladb.com> Co-authored-by: manjots <manjot.singh@scylladb.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Manjot Singh <manjot@manjotsingh.net>
* Persist scylla-monitoring start/stop wrapper scripts on deploy.
Write /opt/scylla-monitoring/{start,stop}-monitoring.sh during monitoring
role deploy so operators can restart the stack without re-running ansible
or reconstructing start-all.sh arguments by hand.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Address Copilot review on monitoring wrapper scripts.
Quote paths and credentials, use conditional sudo for stop, move CQL
secrets to a 0600 env file sourced at runtime, and restore env vars for
ansible-driven starts.
Co-authored-by: Cursor <cursoragent@cursor.com>
* ci: retrigger molecule workflow
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
The assumption cannot be made that the hostname is DNS resolvable by all nodes in the cluster. It's more appropriate to require that the seed nodes are specified explicitly. Signed-off-by: Vincent Reniers <vincent.reniers@scylladb.com>
Removing the implicit scylla_seeds default meant the role would only surface an undefined-variable error deep into common.yml. Add an early explicit check so misconfiguration fails fast with a clear message instead. Also fix the commented example to use consistent groups['scylla'][0] bracket indexing and drop trailing whitespace. Co-authored-by: Cursor <cursoragent@cursor.com>
ansible-scylla-node: remove default scylla_seeds
…iness The 'wait for the API port to come up on all nodes' task used a hardcoded 300s timeout. Under loaded CI runners (multiple docker containers competing for resources) the scylla-server API can take longer than 300s to come up even though the CQL port is already responding, causing intermittent molecule build failures on rockylinux9/rockylinux10 (e.g. PR #531) unrelated to the code under test. Bump the timeout to 600s. Co-authored-by: Manjot Singh <ManjotS@users.noreply.github.com>
3 tasks
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
wait for the API port to come up on all nodestask timeout from 300s to 600s inansible-scylla-node/tasks/common.yml.Context
PR #531's
build (rockylinux9)andbuild (rockylinux10)Molecule jobs failed with:This was preceded by a transient
scylla-server has unexpectedly stoppedblip during the CQL-port wait retry loop, then the CQL port did come up, but the API port (10000) did not respond within the hardcoded 300s on rockylinux9/rockylinux10 runners under load.This is not caused by PR #531's iptables changes:
iptables_drop_sourcesdefaults to[], so the new DROP task is a no-op in this scenario, and the failure reproduced identically across every CI attempt on that same commit (debian11/12, ubuntu2204/2404 always pass; only rockylinux9/10 time out).devstill has the same 300s timeout, so this is pre-existing CI flakiness, not a regression.This mirrors the same class of flakiness already fixed once for a different PR (open PR #533,
ansible-scylla-node: increase API port wait timeout to reduce CI flakiness), and prior fixes for adjacent readiness checks (commits 34f0e15 and 10e2925).Test plan