Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

scylla_cluster,scylla_node: increase debug timeout for watch_rest_for_alive #480

Merged
merged 1 commit into from
Jul 25, 2023
Merged

scylla_cluster,scylla_node: increase debug timeout for watch_rest_for_alive #480

merged 1 commit into from
Jul 25, 2023

Conversation

fruch
Copy link
Contributor

@fruch fruch commented Jul 24, 2023

seems like the 360s default for debug for watch_rest_for_alive isn't enough as it was for watch_log_for_alive,
making it 600s (10min)

Fix: #477

@fruch fruch requested review from nyh and bhalevy July 24, 2023 20:33
@fruch fruch force-pushed the wait_other_notice_debug_timeout branch from b36471f to b80b7ae Compare July 24, 2023 20:33
@bhalevy
Copy link
Member

bhalevy commented Jul 25, 2023 •
edited
Loading

@fruch can you please test this change with update_cluster_layout_tests.py::TestLargeScaleCluster::test_add_many_nodes_under_load in debug mode?

@fruch
scylla_cluster,scylla_node: increase debug timeout for `watch_rest_fo…
730a518 
…r_alive`

seems like the 360s default for debug for `watch_rest_for_alive`
isn't enough as it was for `watch_log_for_alive`,
making it 600s (10min)

Fix: scylladb#477
@fruch fruch force-pushed the wait_other_notice_debug_timeout branch from b80b7ae to 730a518 Compare July 25, 2023 06:15
@fruch
Copy link
Contributor Author

fruch commented Jul 25, 2023

@fruch can you please test this change with update_cluster_layout_tests.py::TestLargeScaleCluster::test_add_many_nodes_under_load in debug mode?

I've sent it on:
https://jenkins.scylladb.com/view/staging/job/scylla-staging/job/fruch/job/new-dtest-pytest-parallel/327/

we'll need to check it out in few hours

@fruch
Copy link
Contributor Author

fruch commented Jul 25, 2023

@bhalevy

It passed:
https://jenkins.scylladb.com/view/staging/job/scylla-staging/job/fruch/job/new-dtest-pytest-parallel/327/artifact/logs-long.debug.000/dtest-gw0.log/*view*/

even that I was a bit surprised it took only 11m (and start only 5 nodes)

@bhalevy
Copy link
Member

bhalevy commented Jul 25, 2023

@bhalevy

It passed: https://jenkins.scylladb.com/view/staging/job/scylla-staging/job/fruch/job/new-dtest-pytest-parallel/327/artifact/logs-long.debug.000/dtest-gw0.log/*view*/

even that I was a bit surprised it took only 11m (and start only 5 nodes)

It was changed in https://github.com/scylladb/scylla-dtest/pull/3268 to stop adding nodes once stress is complete
since the point in running it in debug mode is to see that's everything is stable and doesn't trigger any sanitizer errors.

nyh
nyh approved these changes Jul 25, 2023
View reviewed changes
Copy link
Contributor

@nyh nyh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, but.. 10 minutes to boot a node? Wow, that's preposterous even for debug mode. Oh, well. 600 is just as arbitrary as 360, so I'm fine with that.

@fruch fruch merged commit 820776b into scylladb:next Jul 25, 2023
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nyh nyh nyh approved these changes

@bhalevy bhalevy bhalevy approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

self.watch_rest_for_alive(other_node) fails en-mass during bootstrap in debug mode
3 participants
@fruch @bhalevy @nyh