More consistent behaviour for delta backup snapshots

[Samuel-BF]

Summary

• from a user point of view, we want to define a retention schema using "backup schedules" but it fails as we can't predict which schedule will apply when they are conflicting.
• for delta backups (and for other jobs such as CR/DR i guess), there is one snapshot made per retention period, which is pointless (you only need one snapshot per backup job).

Context

• XO origin: the sources
• Versions:
  • Node: v8.11.4
  • xo-web: 5.26
  • xo-server: 5.26

Expected behavior

Let's suppose you want to keep 1 backup/month for 6 month, 1 backup/week for 4 weeks and 1 backup/day for 8 days. For efficiency reasons, you use delta backups. With backup-NG, you create a backup job with 3 schedules :

• monthly : retention = 6 / cron = 0 0 1 * *
• weekly : retention = 4 / cron = 0 0 * * 1
• daily : retention = 8 / cron = 0 0 * * *

With this scheme, you expect that a delta backup occurs daily and that one or several merges occurs depending on which day of week or month you are. The schedules are defining only the merges. So I expect to have only one snapshot on the hypervisor.

And after a while, I should have backups for : 04-01, 05-01, 06-01, 07-01, 08-01 (monthly backups), 09-01 (monthly and daily backups), 08-13, 08-20, 08-27 (weekly backups), 09-03 (weekly & daily backup), 08-31, 09-02, 09-04, 09-05, 09-06, 09-07 (daily backups). I will have at least 16 backups and at most 17 backups (I can't have 18 backups, as there will always be one backup which is both weekly and daily).

And, more important, I will have a backup per week and a backup per month.

Current behavior

As schedules have the same time (0:0), the schedule running monday D is either the weekly or the daily one. And the schedule running the 1st of month is either the monthly either the daily one (this doesn't seems predictable). This has several consequences :

• there is 3 snapshots kept on the hypervisor (1/schedule), which is useless (and I can remove the all-but-the-youngest snapshots without damage for future backups). This represents wasted storage.
• If a daily snapshot occurs rather than the monthly one, it will not be kept more than 8 days. I may have holes in my backup timetables.

See for example what I got for a similar scheme with more frequent backups (7 x 0,10,20,30,40,50 * * * *, 4 x 10,40 * * * *, 20 x 10 * * * * = 7 x 1 backup per 10 min, 4 x 1 backup at *:{10,40} and 20 x 1 backup at *:10) :

$ jq '.scheduleId + " " + .vmSnapshot.snapshot_time' *.json -r
73be1820-c879-49b5-a716-8e9569db5325 20180905T16:10:15Z
73be1820-c879-49b5-a716-8e9569db5325 20180905T19:12:48Z
73be1820-c879-49b5-a716-8e9569db5325 20180905T22:10:21Z
73be1820-c879-49b5-a716-8e9569db5325 20180906T00:10:07Z
73be1820-c879-49b5-a716-8e9569db5325 20180906T01:10:19Z
73be1820-c879-49b5-a716-8e9569db5325 20180906T03:10:06Z
73be1820-c879-49b5-a716-8e9569db5325 20180906T05:10:05Z
73be1820-c879-49b5-a716-8e9569db5325 20180906T08:10:06Z
73be1820-c879-49b5-a716-8e9569db5325 20180906T09:10:06Z
73be1820-c879-49b5-a716-8e9569db5325 20180906T12:10:05Z
73be1820-c879-49b5-a716-8e9569db5325 20180906T14:10:07Z
73be1820-c879-49b5-a716-8e9569db5325 20180906T16:10:32Z
73be1820-c879-49b5-a716-8e9569db5325 20180906T19:14:43Z
73be1820-c879-49b5-a716-8e9569db5325 20180906T22:10:14Z
73be1820-c879-49b5-a716-8e9569db5325 20180907T00:10:11Z
73be1820-c879-49b5-a716-8e9569db5325 20180907T02:10:07Z
73be1820-c879-49b5-a716-8e9569db5325 20180907T05:10:05Z
12737e38-9763-48f0-be04-3b9ba80e05d3 20180907T06:10:06Z
73be1820-c879-49b5-a716-8e9569db5325 20180907T07:10:07Z
12737e38-9763-48f0-be04-3b9ba80e05d3 20180907T07:40:07Z
73be1820-c879-49b5-a716-8e9569db5325 20180907T08:10:11Z
12737e38-9763-48f0-be04-3b9ba80e05d3 20180907T08:40:07Z
76bdde01-9dda-4085-97be-abb3791fa6bc 20180907T08:50:07Z
76bdde01-9dda-4085-97be-abb3791fa6bc 20180907T09:10:17Z
76bdde01-9dda-4085-97be-abb3791fa6bc 20180907T09:20:12Z
76bdde01-9dda-4085-97be-abb3791fa6bc 20180907T09:30:06Z
12737e38-9763-48f0-be04-3b9ba80e05d3 20180907T09:40:06Z
76bdde01-9dda-4085-97be-abb3791fa6bc 20180907T09:50:08Z
76bdde01-9dda-4085-97be-abb3791fa6bc 20180907T10:00:08Z
73be1820-c879-49b5-a716-8e9569db5325 20180907T10:10:12Z
76bdde01-9dda-4085-97be-abb3791fa6bc 20180907T10:20:24Z

There are 31 backups (20+4+7) and holes of several hours between "hourly" backups.

Workarounds

1. To prevent holes, schedules should not occur at the same time (but this is a waste of resources, since you have to uselessly snapshot and backup twice on mondays)
2. Manually remove extra snapshots when needed (there is no 'snapshot.remove' task in jobs)

[Jarli01]

Your proposal is to have a single snapshot for multiple jobs that are run at any given time,, and to retain that snapshot until the next scheduled version for that job is run.

I can't imagine how you could have just a single snapshot for multiple distinct jobs and to retain a specific snapshot over any period.

XCP-ng and XS will coalescence you're snaps on a random schedule when the SR has the free time, which will get rid of the old snapshots automatically.

In cases like this where you need multiple retention cycles like this, you will have to sacrifice some storage.

Think of this a lot like Microsoft's Shadow Copies, if you want to keep a copy of the data around for any period of time so you can restore to it, you have to consume the storage. . .

[Jarli01]

You can perform an SR scan, which should clean up the old snapshots by performing the coalescence operation, but this also puts added stress on your SR.

[Jarli01]

I believe the biggest reason this won't work is that each snapshot has a unique UUID and multiple snapshots makeup the chain. Removing a link breaks the chain (different retention periods would be links getting removed), thus you are required to have multiple chains with snapshots when you have multiple retention periods in cases like this.

[Samuel-BF]

The point is to have only one delta backup job but several schedules and no snapshot retention. In this case, you don't need to keep several snapshots on the machine, as XO is currently doing. In fact, even if there are several snapshots (all named "[XO Backup job_name] vm_name", 1 per schedule), the delta backup job will snapshot and retrieve the delta only with the last snapshot and will not use the previous one.

Example: suppose you have already done 4 backups using the same job but with 3 different schedules (the things would be different if there were several backup jobs) :

1. M1 (monthly one, the first) -> full backup
2. M2 (monthly one, the second) -> delta backup, parent = M1
3. W (weekly one) -> delta backup, parent = M2
4. D (daily one) -> delta backup, parent = W

Now, you have 3 snapshots on your server, corresponding to backups M2, W and D.
Now, you make a new monthly backup. XO will :

1. snapshot the VM
2. take the delta between this snapshot and the previous one (D) and make a delta backup M3.
3. delete the snapshot corresponding to M2
4. (if needed) merge M1 and M2 VHD files.

The VHD chain is then : M2 (full) <- W (delta) <- D (delta) <- M3 (delta). And to get this result, you didn't need to keep M2 and W snapshots on server. The step 3 in in the scheme could have been done far earlier, as soon as you had made the weekly backup. One snapshot is enough to compute the next link of the chain (you keep the VHD chain in your backups, not in the snapshots on the server).

Note : previously, with backup-not-NG, you needed to have 3 different jobs : in your backups you had 3 times the full backup and in the server you had 3 snapshots.
Backup-NG sharply decreases the need of place in these scenarios.

And i encourage you to test what i say : deleting unneeded snapshots doesn't prevent future delta snapshot to complete without harm (no new full backup and file recovery works well).

[olivierlambert]

@julien-f not sure what to do with this? Isn't the case already? (to use snapshots efficiently)

[olivierlambert]

passing the thread to @fbeauchamp