Add testcase for data being deleted using Truncate while having 90% storage utilization

[pehala]

• Create 3 node (i4i.2xlarge) cluster with rf=3.
• After reaching 90% disk usage.
• Keep it running for an hour, in-between we should remove some data by truncating tables and keyspace.

[cezarmoise]

Results

https://argus.scylladb.com/tests/scylla-cluster-tests/4c7a618f-1ff2-450c-928c-57c5766aba97
Run date: 2025-03-02
Scylla version: 2025.2.0~dev-20250301.0343235aa269
Initial cluster: 3 racks with one i4i.2xlarge per rack
Write to 90%, RF=3, initial tablets=1024
Truncate a big table

Older Results

Run without performance metrics

Initial Cluster: 4 x i4i.large, with auto_snapshot: false
Write to 70%; Sleep 30 min
Write to 90%; Sleep 30 min
Writes have RF=3
Truncate 4 keyspaces
Sleep 30 minutes

---

reached 92% disk usage and started waiting for 30mins, no write or read.

17:02:18  < t:2024-11-14 15:02:18,419 f:full_storage_utilization_test.py l:166  c:FullStorageUtilizationTest2 p:INFO  > Current max disk usage after writing to keyspace15: 92% (399 GB / 392.40000000000003 GB)
17:02:20  < t:2024-11-14 15:02:20,431 f:full_storage_utilization_test.py l:134  c:FullStorageUtilizationTest2 p:INFO  > Wait for 1800 seconds

After 30min idle time, started truncating some tables.

17:33:51  < t:2024-11-14 15:33:51,240 f:full_storage_utilization_test_2.py l:25   c:FullStorageUtilizationTest2 p:INFO  > Executing TRUNCATE TABLE keyspace_large3.standard1
17:43:54  < t:2024-11-14 15:43:51,796 f:full_storage_utilization_test_2.py l:25   c:FullStorageUtilizationTest2 p:INFO  > Executing TRUNCATE TABLE keyspace_large4.standard1
17:53:58  < t:2024-11-14 15:53:52,851 f:full_storage_utilization_test_2.py l:25   c:FullStorageUtilizationTest2 p:INFO  > Executing TRUNCATE TABLE keyspace_small7.standard1
18:04:01  < t:2024-11-14 16:03:53,030 f:full_storage_utilization_test_2.py l:25   c:FullStorageUtilizationTest2 p:INFO  > Executing TRUNCATE TABLE keyspace_small5.standard1

Disk usage drops to about 57%

---

Usage % and P99 latencies

Usage in GB

Argus link https://argus.scylladb.com/tests/scylla-cluster-tests/3a9fe52b-24f3-41da-abf8-22dc6f5d9efc

Performance run

Setup

Scylla version: 2024.3.0~dev-20241206.c09ce641d153
Initial cluster: 3 x i4i.2xlarge
Write to 90%
Mixed workload at about 60%
Truncate a big keyspace

Argus

Load

Disk Usage

mixed - disrupt_truncate - latencies

	P90 write [ms]	P90 read [ms]	P99 write [ms]	P99 read [ms]	Throughput write [op/s]	Throughput read [op/s]	duration [HH:MM:SS]	start time	Overview	QA dashboard
Cycle #​1	$${\color{green}2.25}$$	$${\color{green}4.87}$$	$${\color{green}3.36}$$	$${\color{green}8.48}$$	31,421	31,370	$${\color{green}00:00:37}$$	18:18:12	view	view

Longevity run

Setup

Scylla version: 2025.1.0~dev-20250119.1ef2d9d0769
Initial cluster: 3 x i4i.2xlarge
Write to 90%
Mixed workload at about 60%
Truncate a big keyspace

Argus

Load

Disk Usage

mixed - Steady State - latencies

	P90 write[ms]	P90 read[ms]	P99 write[ms]	P99 read[ms]	Throughput write[op/s]	Throughput read[op/s]	duration[HH:MM:SS]	start time	Overview	QA dashboard
Cycle #​0	$${\color{green}2.06}$$	$${\color{green}2.99}$$	$${\color{green}2.75}$$	$${\color{green}4.76}$$	33,698	33,690	00:30:00	19:49:21	view	view

mixed - disrupt_truncate - latencies

	P90 write[ms]	P90 read[ms]	P99 write[ms]	P99 read[ms]	Throughput write[op/s]	Throughput read[op/s]	duration[HH:MM:SS]	start time	Overview	QA dashboard
Cycle #​1	$${\color{green}2.06}$$	$${\color{green}3.13}$$	$${\color{green}2.85}$$	$${\color{green}5.42}$$	32,006	32,119	00:02:24	20:19:59	view	view

[cezarmoise]

Updated #9169 (comment) with better graph.

[cezarmoise]

Updated results with performance run at 60% load

[pehala]

@cezarmoise Did truncate really run only for 37seconds and in that timeframe it deleted over 100gb of data?

[cezarmoise]

@cezarmoise Did truncate really run only for 37seconds and in that timeframe it deleted over 100gb of data?

Yes, see the disk graph, it drops down in a straight line.