Cluster disk utilization raised from ~65% to >90% while doing backup with Manager

[mikliapko]

Environment:

• Scylla Cloud deployment
• MultiDC cluster (3+3)
• AWS, i4i.2xlarge (1875 GB NVMe SSD)
• ~1TB dataset

Description:

1. Initial disk utilization before backup is around 65%
2. Right after cluster population with data (~1TB) using C-S trigger Manager backup
3. Monitor disk utilization by cluster

Disk utilization by cluster starts to grow right after backup process start, reaches some peak values (~99%) and then drops down to initial state of 65% by the end of the backup.

Important point here is that the utilization reaches 90% and since we are about to have an automatic cluster scale out (X-Cloud) this particular scenario would lead to undesired scale out.

Logs

Jenkins job - https://jenkins.scylladb.com/job/manager-master/job/1-1-restore/job/prepare-snapshot-aws-multi-dc/13/

[mikliapko]

cc: @Michal-Leszczynski @karol-kokoszka

[mikliapko]

Triggered an extra test with some sleep (2 hours) between data population and backup to prove the fact that the compaction is a reason of disk utilization growth and the issue is not related to Manager itself.

[Michal-Leszczynski]

Ref https://docs.google.com/document/d/1q6IfjjR8fwXhXjyPiE_ksDjY8WAfN2VzynJlQkySwlI/edit?tab=t.0

[Michal-Leszczynski]

@mikliapko was the cluster under load during the backup?
This is more realistic and also more problematic from the disk space consumption and latency POV.

I wonder if it would make sense for SM to:

• trigger major compaction
• disable compactions
• snapshot
• upload
• enable compactions

Triggering major compaction and then disabling them for the duration of the backup could (those speculations based on intuition):

• decrease backup size
• decrease disk space utilization during backup caused by rewriting sstables during compaction
• not degrade latency during backup too much

There are a few problems with this approach:

• snapshot is delayed - backup scheduled at 18:00 might be snapshot-ed at 18:15 (it's not like SM ensures the exact data from 18:00 right now, but it should be close to that)
• how to handle pause/resume/retry - normally SM keeps the snapshot on Scylla nodes during pause, as we want to resume backing up the data snapshoted at give point in time, but this seems really problematic for 90% disk utilization
• how does disabling compaction impact latency

[mikliapko]

@mikliapko was the cluster under load during the backup?

Nope, the backup has started only after the completion of the C-S write operation.

[mikliapko]

Triggered an extra test with some sleep (2 hours) between data population and backup to prove the fact that the compaction is a reason of disk utilization growth and the issue is not related to Manager itself.

So, for this test I went with smaller dataset (256Gb) and less productive machines trying to save on resources during the testing.

Here is a graph of average disk usage and compaction running:

Perhaps, the idea to use different cluster instances, loaders and dataset size was wrong and didn't allow to load the cluster at the same extend as in original scenario (the cluster had less data for compaction comparing to original case what resulted in no disk utilization peaks).

@karol-kokoszka @Michal-Leszczynski WDYT about it, guys?

[Michal-Leszczynski]

@mikliapko so you wanted to reproduce the high disk utilization caused by compactions on smaller machines, but you also included the 2h sleep which made it so no compactions were running during the backup?
I guess that you want to rerun it without the sleep to see if it is a smaller reproducer.

[mikliapko]

@mikliapko so you wanted to reproduce the high disk utilization caused by compactions on smaller machines, but you also included the 2h sleep which made it so no compactions were running during the backup? I guess that you want to rerun it without the sleep to see if it is a smaller reproducer.

The original idea was to add 2 hours sleep to prove that the backup process doesn't lead to any disk utilization increase when there is no interfering compaction running.

[Michal-Leszczynski]

Could you re-explain what is the problem?
Adding the 2h sleep is the thing that prevents compactions from causing disk utilization spike during backup.
Smaller machines and dataset size should still experience disk utilization spike without the 2h sleep, but the test run pasted above did both 2h sleep and smaller machines.

[mikliapko]

Could you re-explain what is the problem? Adding the 2h sleep is the thing that prevents compactions from causing disk utilization spike during backup. Smaller machines and dataset size should still experience disk utilization spike without the 2h sleep, but the test run pasted above did both 2h sleep and smaller machines.

Idea was to separate compaction from backup to prove the disk utilization spike is not related to backup in any way.

Plus, for this case I was expecting to see the similar spike because of still running compaction when the C-S write has finished (but the backup not started yet). But I haven't seen it and now I wonder if it is because of dataset size and the rate at which the data was written or because of no backup triggered right after C-S write.

[Michal-Leszczynski]

Idea was to separate compaction from backup to prove the disk utilization spike is not related to backup in any way.

Well, it is related, as disk utilization spike happens when both backup and compactions are running at the same time.
Compactions on their own shouldn't cause big spikes (at least in the regular scenario).

To be more specific, disk utilization spikes happen because:

• SM makes a snapshot which consists of hard link to sstables (hard link basically adds to file's reference counter and the file can be deleted from the disk only after nothing is pointing to it)
• no disk util spike yet
• compaction start rewriting the sstables with hard links
• even though compaction removes old sstables replaced by the rewritten ones, they can't be removed from the disk until they have a hard link attached to them
• this causes disk utilization spike, as both old and rewritten sstables are stored on the disk
• as the upload of the sstables proceeds, they are removed from the snapshot dir, so they can finally also be deleted from the disk

That's why it's expected that you didn't see any disk util spikes if you separated compactions from the backup (snapshot).
If you want to test if smaller machines and dataset size is a good reproducer, you should run the backup right after c-s write like in the original case.

[mikliapko]

@Michal-Leszczynski
I see now... Thanks for detailed explanation.

It turns out that the initial scenario describes the issue pretty well and we have an explanation for such a behavior.

What will be the next steps? Should we wait for the any solution addressing that from Scylla base team?
Is there any way to mitigate possible risks (getting above 90% utilization) for such a scenario on Manager side?

[Michal-Leszczynski]

There is a doc about out-of-space prevention: https://docs.google.com/document/d/1q6IfjjR8fwXhXjyPiE_ksDjY8WAfN2VzynJlQkySwlI/edit?tab=t.0
There are also efforts to fix it with native Scylla backup API: scylladb/scylladb#23241

So it is a know problem and it's analyzed and worked on.

[mikliapko]

I just wonder what are the next steps...

We may close this ticket since the problem has been already taken care of.

But I also see your suggestion above #4321 (comment)

Is it something you'd like to test in Manager?

[Michal-Leszczynski]

Let's keep it opened as a reminder that such problem exists - we will probably get back to it because of the XCLoud.