Description
Packages
Scylla version: 2025.2.0~dev-20250321.7646e1448a48 with build-id 09c0472dcbcb0062ea57d6f57f73687b4289463b
Kernel Version: 6.8.0-1024-aws
Issue description
- This issue is a regression.
- It is unknown if this issue is a regression.
The PR #10335 was not backported to branch-perf-v15, so we did not failed on this issue in official runs.
But when I ran the performance elasticity test (to test my branch), the tests failed with:
2025-03-27 14:46:20.505: (DisruptionEvent Severity.ERROR) period_type=end event_id=441dbb80-5738-41cd-9a78-de84a66796a4 duration=1h49m23s: nemesis_name=GrowShrinkCluster target_node=Node elasticity-test-ubuntu-db-node-00bc5952-3 [3.231.209.161 | 10.12.0.214] errors='list' object has no attribute 'get_results'
Traceback (most recent call last):
File "/home/ubuntu/scylla-cluster-tests/sdcm/nemesis.py", line 5688, in wrapper
result = method(*args[1:], **kwargs)
File "/home/ubuntu/scylla-cluster-tests/sdcm/nemesis.py", line 4384, in disrupt_grow_shrink_cluster
self._double_cluster_load(duration)
File "/home/ubuntu/scylla-cluster-tests/sdcm/utils/decorators.py", line 224, in wrapped
res = func(*args, **kwargs)
File "/home/ubuntu/scylla-cluster-tests/sdcm/nemesis.py", line 4370, in _double_cluster_load
self._nemesis_stress_failure_handler(results, errors)
File "/home/ubuntu/scylla-cluster-tests/sdcm/nemesis.py", line 2139, in _nemesis_stress_failure_handler
if len(errors) == len(stress_pool.get_results()):
AttributeError: 'list' object has no attribute 'get_results'
The problem is that sdcm.nemesis.Nemesis._nemesis_stress_failure_handler expects to receive stress pool, but actually receives list, when called from sdcm.nemesis.Nemesis._double_cluster_load functon:
@latency_calculator_decorator(legend="Doubling cluster load")
def _double_cluster_load(self, duration: int) -> None:
duration = 30
self.log.info("Doubling the load on the cluster for %s minutes", duration)
stress_queue = self.tester.run_stress_thread(
stress_cmd=self.tester.stress_cmd, stress_num=1, stats_aggregate_cmds=False, duration=duration)
results, errors = stress_queue.parse_results()
self._nemesis_stress_failure_handler(results, errors)
self.log.info(f"Double load results: {results}")
Impact
Describe the impact this issue causes to the user.
How frequently does it reproduce?
Describe the frequency with how this issue can be reproduced.
Installation details
Cluster size: 3 nodes (i4i.2xlarge)
Scylla Nodes used in this run:
- elasticity-test-ubuntu-db-node-00bc5952-6 (44.199.230.154 | 10.12.1.165) (shards: 7)
- elasticity-test-ubuntu-db-node-00bc5952-5 (34.200.232.246 | 10.12.0.30) (shards: 7)
- elasticity-test-ubuntu-db-node-00bc5952-4 (18.235.1.43 | 10.12.0.163) (shards: 7)
- elasticity-test-ubuntu-db-node-00bc5952-3 (3.231.209.161 | 10.12.0.214) (shards: 7)
- elasticity-test-ubuntu-db-node-00bc5952-2 (3.222.189.94 | 10.12.1.61) (shards: -1)
- elasticity-test-ubuntu-db-node-00bc5952-1 (44.199.188.174 | 10.12.2.35) (shards: 7)
OS / Image: ami-03794413c92ec8804 (aws: undefined_region)
Test: scylla-enterprise-perf-regression-latency-650gb-elasticity
Test id: 00bc5952-309b-4b6b-a223-8c6c89ff21a4
Test name: scylla-staging/yulia/performance/scylla-enterprise-perf-regression-latency-650gb-elasticity
Test method: performance_regression_test.PerformanceRegressionTest.test_latency_read_with_nemesis
Test config file(s):
Logs and commands
- Restore Monitor Stack command:
$ hydra investigate show-monitor 00bc5952-309b-4b6b-a223-8c6c89ff21a4 - Restore monitor on AWS instance using Jenkins job
- Show all stored logs command:
$ hydra investigate show-logs 00bc5952-309b-4b6b-a223-8c6c89ff21a4
Logs:
- db-cluster-00bc5952.tar.zst - https://cloudius-jenkins-test.s3.amazonaws.com/00bc5952-309b-4b6b-a223-8c6c89ff21a4/20250327_152331/db-cluster-00bc5952.tar.zst
- sct-runner-events-00bc5952.tar.zst - https://cloudius-jenkins-test.s3.amazonaws.com/00bc5952-309b-4b6b-a223-8c6c89ff21a4/20250327_152331/sct-runner-events-00bc5952.tar.zst
- sct-00bc5952.log.tar.zst - https://cloudius-jenkins-test.s3.amazonaws.com/00bc5952-309b-4b6b-a223-8c6c89ff21a4/20250327_152331/sct-00bc5952.log.tar.zst
- loader-set-00bc5952.tar.zst - https://cloudius-jenkins-test.s3.amazonaws.com/00bc5952-309b-4b6b-a223-8c6c89ff21a4/20250327_152331/loader-set-00bc5952.tar.zst
- monitor-set-00bc5952.tar.zst - https://cloudius-jenkins-test.s3.amazonaws.com/00bc5952-309b-4b6b-a223-8c6c89ff21a4/20250327_152331/monitor-set-00bc5952.tar.zst
- builder-00bc5952.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/00bc5952-309b-4b6b-a223-8c6c89ff21a4/upload_20250327_152457/builder-00bc5952.log.tar.gz