Add testcase that adds larger instance while having 90% storage utilization

[cezarmoise]

Create 3 node (i4i.large) cluster with rf=3.
After reaching 90% disk usage.
Wait until cluster stabilizes
Perform scale-out by adding 3 nodes of larger instances (i4i.4xlarge)

Results

[cezarmoise]

Results

https://argus.scylladb.com/tests/scylla-cluster-tests/3aa9e97b-a53d-48dd-a198-9ae9f97d4218
Run date: 2025-03-02
Scylla version: 2025.2.0~dev-20250301.0343235aa269
Initial cluster: 3 racks with one i4i.large per rack
Write to 90%, RF=3, initial tablets=1024
Run mixed workload at about 40%
Add one i4i.4xlarge to each rack

Steady latencies

	P90 write [ms]	P90 read [ms]	P99 write [ms]	P99 read [ms]	Throughput write [op/s]	Throughput read [op/s]	duration [HH:MM:SS]	start time	Overview	QA dashboard
Cycle #​0	$${\color{green}1.12}$$	$${\color{green}1.97}$$	$${\color{green}2.13}$$	$${\color{green}3.35}$$	4,924	4,926	00:30:00	23:41:58	view	view

Latencies during scale out

	P90 write [ms]	P90 read [ms]	P99 write [ms]	P99 read [ms]	Throughput write [op/s]	Throughput read [op/s]	duration [HH:MM:SS]	start time	Overview	QA dashboard
Cycle #​1	$${\color{green}1.39}$$	$${\color{green}2.33}$$	$${\color{green}2.54}$$	$${\color{green}4.39}$$	4,943	4,949	00:42:32	00:12:29	view	view

Older runs

4 x i4i.large + 3 x i4i.xlarge

Initial Cluster: 4 x i4i.large
Write to 70%; Sleep 30 min
Write to 90%; Sleep 30 min
Writes have RF=3
Add 3 nodes i4i.xlarge
Sleep 30 minutes

---

Usage % and P99 latencies

Usage in GB

Argus

https://argus.scylladb.com/tests/scylla-cluster-tests/7f77540e-48dc-4c25-a044-954c6448b7ba

3 x i4i.large + 1 x i4i.xlarge

Initial Cluster: 3 x i4i.large
Write to 70%; Sleep 30 min
Write to 90%; Sleep 30 min
Writes have RF=3
Add 1 node i4i.xlarge
Sleep 30 minutes

---

Usage % and P99 latencies

Usage in GB

Argus

https://argus.scylladb.com/tests/scylla-cluster-tests/22215e5f-3de2-4850-879f-327fb3efc48b

	Total [GB]	Used [GB]	Available [GB]	Usage %
node_1 [Soft Limit]	436.00	311.00	125.00	72.00
node_2 [Soft Limit]	436.00	309.00	128.00	71.00
node_3 [Soft Limit]	436.00	308.00	129.00	71.00
cluster [Soft Limit]	1308.00	928.00	382.00	71.00
node_1 [Hard Limit]	436.00	373.00	64.00	86.00
node_2 [Hard Limit]	436.00	369.00	68.00	85.00
node_3 [Hard Limit]	436.00	378.00	59.00	87.00
cluster [Hard Limit]	1308.00	1120.00	191.00	86.00
node_1 [Scale out]	436.00	268.00	169.00	62.00
node_2 [Scale out]	436.00	256.00	180.00	59.00
node_3 [Scale out]	436.00	258.00	179.00	60.00
node_4 [Scale out]	873.00	397.00	476.00	46.00
cluster [Scale out]	2181.00	1179.00	1004.00	56.00
node_1 [Final]	436.00	268.00	169.00	62.00
node_2 [Final]	436.00	256.00	180.00	59.00
node_3 [Final]	436.00	258.00	179.00	60.00
node_4 [Final]	873.00	397.00	476.00	46.00
cluster [Final]	2181.00	1179.00	1004.00	56.00

	duration [HH:MM:SS]
Soft Limit	01:21:27
Hard Limit	00:53:31
Add new node(s)	00:01:52
New node(s) ready	00:14:29

Performance run falied on 3rd added node

Setup

Initial Cluster: 3 x i4i.large
Write to 90%
Add 3 x i4i.4xlarge

Argus

Errors

2024-12-09 22:05:14.229 <2024-12-09 22:05:14.032>: (DatabaseLogEvent Severity.ERROR) period_type=one-time event_id=e2afc03b-be73-4a70-861f-de60a49318f6 during_nemesis=GrowCluster: type=DATABASE_ERROR regex=(^ERROR|!\s*?ERR).*\[shard.*\] line_number=1940 node=elasticity-test-ubuntu-db-node-7d99bf48-6
2024-12-09T22:05:14.032+00:00 elasticity-test-ubuntu-db-node-7d99bf48-6      !ERR | scylla[5596]:  [shard  0:main] init - Startup failed: service::raft_operation_timeout_error (group [d3c03611-b646-11ef-af05-6895dac0d512] raft operation [read_barrier] timed out)

2024-12-09 22:05:13.286 <2024-12-09 22:05:13.236>: (DatabaseLogEvent Severity.ERROR) period_type=one-time event_id=e2afc03b-be73-4a70-861f-de60a49318f6: type=DATABASE_ERROR regex=(^ERROR|!\s*?ERR).*\[shard.*\] line_number=33250 node=elasticity-test-ubuntu-db-node-7d99bf48-1
2024-12-09T22:05:13.236+00:00 elasticity-test-ubuntu-db-node-7d99bf48-1      !ERR | scylla[5313]:  [shard 0: gms] raft - [747802c4-360d-47b7-af08-e096b519f461] Transferring snapshot to 60338276-a6f9-43fe-9745-f63d8e2273a9 failed with: seastar::rpc::remote_verb_error (Abort requested while transferring snapshot from ID/IP: 747802c4-360d-47b7-af08-e096b519f461/optional(10.4.2.155), snapshot descriptor id: 04e15b4e-3b82-408c-8b83-9decf4fc28e0, snapshot index: 1)

Load

Disk Usage

mixed - Steady State - latencies

	P90 write [ms]	P90 read [ms]	P99 write [ms]	P99 read [ms]	Throughput write [op/s]	Throughput read [op/s]	duration [HH:MM:SS]	start time	Overview	QA dashboard
Cycle #​0	$${\color{red}2353642.08}$$	$${\color{red}2353642.08}$$	$${\color{red}2542620.64}$$	$${\color{red}2542620.64}$$	7,987	7,990	$${\color{green}00:30:00}$$	20:28:50	view	view

mixed - Steady State - stalls - REACTOR_STALLED

	total [count]	10ms [count]
Cycle #​0	$${\color{green}1}$$	$${\color{green}1}$$

mixed - add_new_nodes - latencies

	P90 write [ms]	P90 read [ms]	P99 write [ms]	P99 read [ms]	Throughput write [op/s]	Throughput read [op/s]	duration [HH:MM:SS]	start time	Overview	QA dashboard
Cycle #​1	$${\color{red}4221952.85}$$	$${\color{red}4221952.85}$$	$${\color{red}4256312.59}$$	$${\color{red}4256312.59}$$	11,064	11,060	$${\color{green}00:41:21}$$	20:59:23	view	view
Cycle #​2	$${\color{red}4591320.04}$$	$${\color{red}4591320.04}$$	$${\color{red}4595615.01}$$	$${\color{red}4595615.01}$$	28,276	28,274	$${\color{green}00:19:44}$$	21:41:44	view	view

mixed - add_new_nodes - stalls - REACTOR_STALLED

	total [count]	10ms [count]
Cycle #​1	$${\color{green}1}$$	$${\color{green}1}$$
Cycle #​2	$${\color{green}2}$$	$${\color{green}2}$$

Performance run @ 80% load

Setup

Initial Cluster: 3 x i4i.large
Write to 90%
Mixed workload @ 'threads=300 fixed=6000/s'
Add 3 x i4i.4xlarge

Argus

Load

Disk Usage

mixed - Steady State - latencies

	P90 write [ms]	P90 read [ms]	P99 write [ms]	P99 read [ms]	Throughput write [op/s]	Throughput read [op/s]	duration [HH:MM:SS]	start time	Overview	QA dashboard
Cycle #​0	$${\color{red}394600.12}$$	$${\color{red}394600.12}$$	$${\color{red}410437.81}$$	$${\color{red}410437.81}$$	9,391	9,399	$${\color{green}00:30:00}$$	19:41:39	view	view

mixed - add_new_nodes - latencies

	P90 write [ms]	P90 read [ms]	P99 write [ms]	P99 read [ms]	Throughput write [op/s]	Throughput read [op/s]	duration [HH:MM:SS]	start time	Overview	QA dashboard
Cycle #​1	$${\color{red}706522.12}$$	$${\color{red}706522.12}$$	$${\color{red}718870.15}$$	$${\color{red}718870.15}$$	13,334	13,325	$${\color{green}00:31:45}$$	20:12:10	view	view
Cycle #​2	$${\color{red}80195.09}$$	$${\color{red}80060.87}$$	$${\color{red}131868.92}$$	$${\color{red}132070.24}$$	13,775	13,794	$${\color{green}00:14:58}$$	20:44:51	view	view
Cycle #​3	$${\color{green}0.73}$$	$${\color{green}2.04}$$	$${\color{green}2.03}$$	$${\color{green}4.49}$$	11,899	11,909	$${\color{green}00:09:12}$$	21:00:42	view	view

mixed - add_new_nodes - stalls - REACTOR_STALLED

	total [count]	10ms [count]
Cycle #​1	$${\color{green}1}$$	$${\color{green}1}$$
Cycle #​3	$${\color{green}1}$$	$${\color{green}1}$$

Performance run @ 70% load

Setup

Initial Cluster: 3 x i4i.large
Write to 90%
Mixed workload @ 'threads=300 fixed=4500/s'
Add 3 x i4i.4xlarge

Argus

Load

Disk Usage

mixed - Steady State - latencies

	P90 write [ms]	P90 read [ms]	P99 write [ms]	P99 read [ms]	Throughput write [op/s]	Throughput read [op/s]	duration [HH:MM:SS]	start time	Overview	QA dashboard
Cycle #​0	$${\color{green}1.75}$$	$${\color{green}3.67}$$	$${\color{green}2.26}$$	$${\color{green}7.45}$$	8,981	8,989	$${\color{green}00:30:00}$$	15:05:22	view	view

mixed - add_new_nodes - latencies

	P90 write [ms]	P90 read [ms]	P99 write [ms]	P99 read [ms]	Throughput write [op/s]	Throughput read [op/s]	duration [HH:MM:SS]	start time	Overview	QA dashboard
Cycle #​1	$${\color{red}67477.96}$$	$${\color{red}67578.63}$$	$${\color{red}80195.09}$$	$${\color{red}80396.42}$$	8,993	8,974	$${\color{green}00:27:12}$$	15:36:03	view	view
Cycle #​2	$${\color{green}0.69}$$	$${\color{green}2.17}$$	$${\color{green}2.10}$$	$${\color{green}3.81}$$	8,940	8,956	$${\color{green}00:12:54}$$	16:04:17	view	view
Cycle #​3	$${\color{green}0.54}$$	$${\color{green}1.65}$$	$${\color{green}1.87}$$	$${\color{green}4.51}$$	8,945	8,928	$${\color{green}00:07:01}$$	16:18:03	view	view

mixed - add_new_nodes - stalls - REACTOR_STALLED

	total [count]	10ms [count]
Cycle #​1	$${\color{green}3}$$	$${\color{green}3}$$
Cycle #​2	$${\color{green}1}$$	$${\color{green}1}$$
Cycle #​3	$${\color{green}1}$$	$${\color{green}1}$$

Performance run @ 50% load

Setup

Initial Cluster: 3 x i4i.large
Write to 90%
Mixed workload @ 'threads=300 fixed=3750/s'
Add 3 x i4i.4xlarge

Argus

Load

Disk Usage

mixed - Steady State - latencies

	P90 write [ms]	P90 read [ms]	P99 write [ms]	P99 read [ms]	Throughput write [op/s]	Throughput read [op/s]	duration [HH:MM:SS]	start time	Overview	QA dashboard
Cycle #​0	$${\color{green}1.24}$$	$${\color{green}2.67}$$	$${\color{green}2.14}$$	$${\color{green}4.65}$$	7,488	7,487	$${\color{green}00:30:00}$$	02:07:50	view	view

mixed - add_new_nodes - latencies

	P90 write [ms]	P90 read [ms]	P99 write [ms]	P99 read [ms]	Throughput write [op/s]	Throughput read [op/s]	duration [HH:MM:SS]	start time	Overview	QA dashboard
Cycle #​1	$${\color{green}1.46}$$	$${\color{red}10.96}$$	$${\color{red}699.40}$$	$${\color{red}858.78}$$	7,473	7,463	$${\color{green}00:23:25}$$	02:38:28	view	view
Cycle #​2	$${\color{green}0.57}$$	$${\color{green}2.36}$$	$${\color{green}2.02}$$	$${\color{green}3.99}$$	7,471	7,465	$${\color{green}00:11:26}$$	03:02:51	view	view
Cycle #​3	$${\color{green}0.49}$$	$${\color{green}1.71}$$	$${\color{green}1.81}$$	$${\color{green}4.61}$$	7,450	7,443	$${\color{green}00:07:02}$$	03:14:46	view	view

Performance run @ 50% load (add nodes in parallel)

Setup

Initial Cluster: 3 x i4i.large
Write to 90%
Mixed workload @ 'threads=150 fixed=3400/s'
Add 3 x i4i.4xlarge

Argus

Load

Disk Usage

mixed - Steady State - latencies

	P90 write [ms]	P90 read [ms]	P99 write [ms]	P99 read [ms]	Throughput write [op/s]	Throughput read [op/s]	duration [HH:MM:SS]	start time	Overview	QA dashboard
Cycle #​0	$${\color{green}1.49}$$	$${\color{green}2.96}$$	$${\color{green}2.20}$$	$${\color{green}4.92}$$	6,790	6,788	$${\color{green}00:30:00}$$	14:47:02	view	view

mixed - add_new_nodes - latencies

	P90 write [ms]	P90 read [ms]	P99 write [ms]	P99 read [ms]	Throughput write [op/s]	Throughput read [op/s]	duration [HH:MM:SS]	start time	Overview	QA dashboard
Cycle #​1	$${\color{green}0.91}$$	$${\color{green}3.75}$$	$${\color{green}2.16}$$	$${\color{green}9.42}$$	6,779	6,782	$${\color{green}00:34:46}$$	15:17:47	view	view

Performance run w\ 2025.1

Setup

Scylla version: 2025.1.0~dev-20250119.1ef2d9d07692
Initial Cluster: 3 x i4i.large
Write to 90%
Mixed workload @ 'threads=150 fixed=3400/s'
Add 3 x i4i.4xlarge

Argus

Load

Disk Usage

mixed - Steady State - latencies

	P90 write[ms]	P90 read[ms]	P99 write[ms]	P99 read[ms]	Throughput write[op/s]	Throughput read[op/s]	duration[HH:MM:SS]	start time	Overview	QA dashboard
Cycle #​0	$${\color{green}1.61}$$	$${\color{green}4.76}$$	$${\color{green}2.25}$$	$${\color{red}10.19}$$	6,583	6,588	00:30:00	02:06:26	view	view

mixed - add_new_nodes - latencies

	P90 write[ms]	P90 read[ms]	P99 write[ms]	P99 read[ms]	Throughput write[op/s]	Throughput read[op/s]	duration[HH:MM:SS]	start time	Overview	QA dashboard
Cycle #​1	$${\color{green}2.57}$$	$${\color{red}5.31}$$	$${\color{green}3.08}$$	$${\color{green}8.34}$$	6,591	6,596	01:06:35	02:37:06	view	view

mixed - add_new_nodes - stalls - REACTOR_STALLED

	total[count]	20ms[count]	50ms[count]
Cycle #​1	4	3	1

---

Performance run w\ 2025.2

Setup

Scylla version: 2025.2.0~dev-20250130.000791ad5c84
Initial Cluster: 3 x i4i.large
Write to 90%
Mixed workload @ 'threads=150 fixed=3400/s'
Add 3 x i4i.4xlarge

Argus

Load

Disk Usage

mixed - Steady State - latencies

	P90 write [ms]	P90 read [ms]	P99 write [ms]	P99 read [ms]	Throughput write [op/s]	Throughput read [op/s]	duration [HH:MM:SS]	start time	Overview	QA dashboard
Cycle #​0	$${\color{green}1.99}$$	$${\color{green}3.20}$$	$${\color{green}2.48}$$	$${\color{green}4.67}$$	6,587	6,591	00:30:00	18:40:24	view	view

mixed - add_new_nodes - latencies

	P90 write [ms]	P90 read [ms]	P99 write [ms]	P99 read [ms]	Throughput write [op/s]	Throughput read [op/s]	duration [HH:MM:SS]	start time	Overview	QA dashboard
Cycle #​1	$${\color{green}1.00}$$	$${\color{green}2.06}$$	$${\color{green}2.21}$$	$${\color{green}4.02}$$	6,586	6,589	00:26:22	19:11:00	view	view

[pehala]

What was the time to scaleout?

[cezarmoise]

What was the time to scaleout?

Didn't calculate exactly, but from graph aprox 15 minutes.

Updated: #9257 (comment)

[cezarmoise]

Updated results. Failed on adding a 3rd nodes and had huge latencies. Might be caused by high load. Will run again with lower load.

[cezarmoise]

Updated results with a run @ 80% load, will run a new one with something closer to 50%

[cezarmoise]

Updated results with runs at 70% and 50% load.

[pehala]

Updated results with runs at 70% and 50% load.

Thanks,

• Did you add the nodes in the parallel?
• Regarding latencies, it seems like way too much, do you have any explanation for it, or do you believe it is an issue?
• Is the 50% load really that bad for i4i.large? Not sure how much less compute it has compared to our standard instances but I think we are usually running around 30K for those. Does it match proportionally?