Add Filler field for XVA exported backups to improve deduplication of backups

[andrey-podko]

XVA begins from XML of metadata, this XML has random size and do a random shift from beginning of backup file to beginning of VM disks.
That means that disk blocks are not aligned with real blocks on storage, they are shifted
so attempt to do deduplication for these blocks will fail due this shift.
That is not so good if you store XVA backups on ZFS or XFS and you have tens of very similar VMs.

Could you add an option like --align-header-to=8KB(16KB.......1M) to xe vm-export ?
This could be something as simple as an extra field in XML that will be ignored on import but will give the required size of header.

And these so big alignments up to 1MB is not a joke! At least if you use ZFS on backup storage.
I got a best results on deduplication of backup with recordsize=256k ! I mean between multiple daily backups.
And would be very nice to see a normal deduplication inside one daily set to ;)

I would be happy to just send you a patch but OCAML.... unfortunately it makes it too complicated for me

Thanks in advance

[psafont]

This looks like it's done on ocaml/xapi/export.ml, around line 666:

  let hdr =
    Tar.Header.make ~mod_time:now Xapi_globs.ova_xml_filename
      (Int64.of_int @@ String.length ova_xml)
  in
  Tar_helpers.write_block hdr (fun s -> Unixext.really_write_string s ova_xml) s ;

So it would be a matter of adding padding to the tarball, or the file itself.

Is there any minimal alignment that should be done? How big are the blocks, does it depend on filesystem?

[andrey-podko]

Is there any minimal alignment that should be done? How big are the blocks, does it depend on filesystem?

I think 4k is minimum alignment for today as it is minimum block size for modern filesystems.
And yes, it depends on filesystem and underlayer , because popular filesystems (include XFS) allow to tune their behavior for best utilization of RAID with very different stripe size (32KB-1MB). Not so many people do backups to single drive, right ?
Similar thing with ZFS that has its own RAIDs and recordsize configurable in range "512 to 1M, power of 2"

That is a reason why I think about configurable option. XFS users may want to use 32KB-128KB but ZFS users may want to use 128KB-1M

[psafont]

Could tar be used in another way, where this xml is stored at the end of the tarball and the header is fixed length, allowing most of the archives be deduplicated?

[andrey-podko]

That could the good way - all disks will have same shift,
But I am not sure about compatibility... It depends on how tools read and process XVA files.