Add testcase for data being deleted using Drop while having 90% storage utilization

[pehala]

• Create 3 node cluster with rf=3.
• Reaching 90% disk usage.
• Keep it running for an hour, in-between we should clean up some data by dropping tables and keyspace.
• Wait for compaction

[cezarmoise]

Results

https://argus.scylladb.com/tests/scylla-cluster-tests/a22d7bbd-4e14-4a94-b9ef-c6460a50b465
Run date: 2025-03-02
Scylla version: 2025.2.0~dev-20250301.0343235aa269
Initial cluster: 3 racks with one i4i.2xlarge per rack
Write to 90%, RF=3, initial tablets=1024
Drop a big table

Older runs

Exploratory Run

Initial Cluster: 4 x i4i.large with auto_snapshot: false
Write to 70%; Sleep 30 min
Write to 90%; Sleep 30 min
Writes have RF=3
Drop some tables
Sleep 30 minutes

---

reached 91% disk usage and started waiting for 30mins, no write or read.

16:59:51  < t:2024-11-14 14:59:50,996 f:full_storage_utilization_test.py l:166  c:FullStorageUtilizationTest2 p:INFO  > Current max disk usage after writing to keyspace14: 91% (395 GB / 392.40000000000003 GB)
16:59:53  < t:2024-11-14 14:59:53,007 f:full_storage_utilization_test.py l:134  c:FullStorageUtilizationTest2 p:INFO  > Wait for 1800 seconds

After 30min idle time, started dropping some tables.

17:31:02  < t:2024-11-14 15:31:02,190 f:full_storage_utilization_test_2.py l:25   c:FullStorageUtilizationTest2 p:INFO  > Executing DROP TABLE keyspace_large3.standard1
17:41:05  < t:2024-11-14 15:41:03,922 f:full_storage_utilization_test_2.py l:25   c:FullStorageUtilizationTest2 p:INFO  > Executing DROP TABLE keyspace_large4.standard1
17:51:08  < t:2024-11-14 15:51:05,916 f:full_storage_utilization_test_2.py l:25   c:FullStorageUtilizationTest2 p:INFO  > Executing DROP TABLE keyspace_small7.standard1
18:01:11  < t:2024-11-14 16:01:08,301 f:full_storage_utilization_test_2.py l:25   c:FullStorageUtilizationTest2 p:INFO  > Executing DROP TABLE keyspace_small5.standard1

Disk usage drops to about 56%

---

Usage % and P99 latencies

Usage in GB

Argus

Argus link https://argus.scylladb.com/tests/scylla-cluster-tests/00e90f4b-46ae-489d-b3a4-12b42e540a5e

Performance run at 70% workload

Setup

Initial cluster: 3 x i4i.2xlarge
Write to 90%
Drop a big keyspace

Argus

Load

Disk Usage

mixed - disrupt_drop - latencies

	P90 write [ms]	P90 read [ms]	P99 write [ms]	P99 read [ms]	Throughput write [op/s]	Throughput read [op/s]	duration[HH:MM:SS]	start time	Overview	QA dashboard
Cycle #​1	$${\color{green}2.21}$$	$${\color{red}5.18}$$	$${\color{green}3.72}$$	$${\color{red}14.70}$$	26,034	25,895	$${\color{green}00:00:12}$$	17:48:45	view	view

mixed - disrupt_drop - stalls - REACTOR_STALLED

	total [count]	10ms [count]
Cycle #​1	$${\color{green}2}$$	$${\color{green}2}$$

Performance run 2

Setup

Scylla version: 2025.1.0~dev-20250119.1ef2d9d07692
Initial cluster: 3 x i4i.2xlarge
Write to 90%
Drop a big keyspace

Argus

Load

Disk Usage

mixed - Steady State - latencies

	P90 write[ms]	P90 read[ms]	P99 write[ms]	P99 read[ms]	Throughput write[op/s]	Throughput read[op/s]	duration[HH:MM:SS]	start time	Overview	QA dashboard
Cycle #​0	$${\color{green}2.07}$$	$${\color{green}3.27}$$	$${\color{green}2.76}$$	$${\color{green}5.54}$$	33,678	33,673	00:30:00	19:22:55	view	view

mixed - disrupt_drop - latencies

	P90 write [ms]	P90 read [ms]	P99 write [ms]	P99 read[ms]	Throughput write[op/s]	Throughput read[op/s]	duration[HH:MM:SS]	start time	Overview	QA dashboard
Cycle #​1	$${\color{green}2.09}$$	$${\color{green}3.32}$$	$${\color{green}3.01}$$	$${\color{green}5.54}$$	32,764	32,722	00:02:06	19:53:34	view	view

[cezarmoise]

Updated #9166 (comment) with better graph.

[cezarmoise]

Updated results with run at 70% workload

[cezarmoise]

Added run with latest 2025.1