Add 90% storage utilization tests

[pehala]

Overview

Add both test & testcases to verify that Scylla can run with 90% storage utilization with tablets

Test Requirements

Workload

• If not specified otherwise, all workloads are assumed to be 50% reads / 50% writes.

Latency

• Operation latency measured with 50% of the max Throughput.
• P99 should be under 10ms while scaling both, for read and write

Storage utilization

• Maximum storage utilization before a scale out: 90%
• Minimum storage utilization before a scale in: 75%.
  • Make sure that after scale in we will be below maximum storage utilization

Scale Time (Duration):

• Base on 100MB/s BW per core
• Scaling up operation require 2.5 extra minutes for node boot time
• Validating scale time in the worst case

Tasks

• Create test function for testing 90% storage utilization #9130
• run-longevity-tests, run-nemesis-at-90-percent: Run existing nemesis with 90% storage utilization test function #9155

Scaling

• 3node-scale-out-scale-in: Add testcase for scale-out/scale-in while having 90% storage utilization #9156
• multidc-cluster-scaleout: Add multi-dc testcase for 90% storage utilization #9157

Data deletion

• data-removal-drop-compaction: Add testcase for data being deleted using Drop while having 90% storage utilization #9166
• data-removal-truncate-compaction: Add testcase for data being deleted using Truncate while having 90% storage utilization #9169
• data-removal-ttl-compaction Add testcase for data being deleted using TTL while having 90% storage utilization #9167

Scaling time

• time-to-double-cluster: Add testcase to measure scale-out time for doubling cluster size while having 90% storage utilization #9220

Mixed instances

• add-larger-instance: Add testcase that adds larger instance while having 90% storage utilization #9257
• add-small-instance: Add testcase that adds smaller instance while having 90% storage utilization #9256

Extras

• multiple-small-tables: Create testcase for having 90% utilization with a lot of small tables #9309
• change-instance-type:
• double-cluster-smaller-instances: Add testcase for doubling cluster with smaller instances while keeping 90% utilization #9597

[mykaul]

@pehala - we are missing (or I could not find) mixed nodes support - we add a different instance type, same family to the cluster - say you had several i4i.8xlarge, now you add an additional i4i.2xlarge. This will happen often.
CC @swasik

[pehala]

@pehala - we are missing (or I could not find) mixed nodes support - we add a different instance type, same family to the cluster - say you had several i4i.8xlarge, now you add an additional i4i.2xlarge. This will happen often.
CC @swasik

We have #9257 (3 L -> +XL) and #9256 (3 XL -> +L), which add different instance sizes to the cluster, does that cover the mixed node use?