Skip to content

partially_populate_device: align span positions to block size for better performance - #509

Merged
glehmann merged 1 commit into
masterfrom
gln/align-spans-to-block-size-rupt
Jun 2, 2026
Merged

partially_populate_device: align span positions to block size for better performance#509
glehmann merged 1 commit into
masterfrom
gln/align-spans-to-block-size-rupt

Conversation

@glehmann

@glehmann glehmann commented Apr 29, 2026

Copy link
Copy Markdown
Member

Add an optional align parameter (default: 4KiB) to align span positions
to a given block size. Only positions are aligned (rounded down); span sizes
are reduced if needed to avoid overlapping the next span's position. When
config.write_volume_cap is large enough, spans are contiguous and cover the
full device with no gaps. Without alignment, write throughput drops from
~700MiB/s to ~200MiB/s on large disks.

This PR is part of a tree containing 19 PRs:

  1. master
  2. storage: avoid removing the xva/vdi before going in the debugger #436master
  3. storage: test large volumes #437storage: avoid removing the xva/vdi before going in the debugger #436
  4. host: include mdadm RAID devices in disk detection #447storage: test large volumes #437
  5. Update randstream to 0.5.0 #446host: include mdadm RAID devices in disk detection #447
  6. storage: Avoid writing the whole device in coalesce tests #449Update randstream to 0.5.0 #446
  7. storage: avoid writing the whole device in migration tests #450storage: Avoid writing the whole device in coalesce tests #449
  8. xva/vdi: only write a small data amount in large volumes for faster tests #452storage: avoid writing the whole device in migration tests #450
  9. storage: test full device write #453xva/vdi: only write a small data amount in large volumes for faster tests #452
  10. storage: test that we can't create a vdi over its max allowed size #454storage: test full device write #453
  11. storage: add jobs for large volume tests #461storage: test that we can't create a vdi over its max allowed size #454
  12. skip large volume tests for zvol and nfsv4 #464storage: add jobs for large volume tests #461
  13. storage: add comprehensive tests for lvmohba storage repositories #470skip large volume tests for zvol and nfsv4 #464
  14. storage: free space for XVA import by destroying source VM first #471storage: add comprehensive tests for lvmohba storage repositories #470
  15. storage: limit data written per VDI with --write-volume-cap #481storage: free space for XVA import by destroying source VM first #471
  16. Add Packer configuration to build a minimal Alpine 3.23 UEFI VM for XCP-ng tests #523storage: limit data written per VDI with --write-volume-cap #481
  17. migration: create a xfs sr on the second host for intra/cross-pool migration #497storage: limit data written per VDI with --write-volume-cap #481
  18. Enhance block device management #498migration: create a xfs sr on the second host for intra/cross-pool migration #497
  19. Add VHD_MAX and QCOW2_MAX symbolic size constants #500Enhance block device management #498
  20. "partially_populate_device: align spans to block size for better performance" (this PR) → Add VHD_MAX and QCOW2_MAX symbolic size constants #500

@glehmann
glehmann requested review from a team as code owners April 29, 2026 17:39
@glehmann
glehmann force-pushed the gln/align-spans-to-block-size-rupt branch 3 times, most recently from cd4e983 to 5b46ad7 Compare May 1, 2026 12:09
@glehmann
glehmann force-pushed the gln/add-max-volume-size-constants-npxp branch from 0a82603 to b73bcc9 Compare May 11, 2026 15:48
@glehmann
glehmann force-pushed the gln/align-spans-to-block-size-rupt branch from 5b46ad7 to 03aaba9 Compare May 11, 2026 15:49
@glehmann glehmann changed the title partially_populate_device: align spans to block size for better performance partially_populate_device: align span positions to block size for better performance May 11, 2026
@glehmann
glehmann force-pushed the gln/add-max-volume-size-constants-npxp branch from b73bcc9 to 5df0dc2 Compare May 12, 2026 13:16
@glehmann
glehmann force-pushed the gln/align-spans-to-block-size-rupt branch from 03aaba9 to 2039225 Compare May 12, 2026 13:16
@glehmann
glehmann force-pushed the gln/add-max-volume-size-constants-npxp branch from 5df0dc2 to ff50780 Compare May 12, 2026 13:23
@glehmann
glehmann force-pushed the gln/align-spans-to-block-size-rupt branch from 2039225 to 9f79300 Compare May 12, 2026 13:23
Comment thread tests/storage/storage.py Outdated
expected_flags = f'--expected-checksum {self.checksum}' if self.checksum is not None else ''
randstream(vm, f'validate {expected_flags} --position {self.position} --size {self.size} {dev}')

def compute_span_layout(dev_size: int, total_size: int, num_spans: int, align: int) -> list[tuple[int, int]]:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd rather have the argument called block_size. Align makes it ambiguous whether it's a flag or a value.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done!

@glehmann
glehmann force-pushed the gln/align-spans-to-block-size-rupt branch from 34456d1 to 81137d3 Compare May 13, 2026 13:56
@glehmann
glehmann force-pushed the gln/add-max-volume-size-constants-npxp branch from c70e171 to fbb8227 Compare May 13, 2026 13:58
@glehmann
glehmann force-pushed the gln/align-spans-to-block-size-rupt branch 2 times, most recently from 9d503d2 to d05f1ec Compare May 18, 2026 16:22
Comment thread conftest.py
@glehmann
glehmann force-pushed the gln/align-spans-to-block-size-rupt branch from d05f1ec to 8fe8f4a Compare May 19, 2026 15:29
@glehmann
glehmann force-pushed the gln/add-max-volume-size-constants-npxp branch from 3e8fe23 to 44d95e4 Compare May 19, 2026 15:29
@glehmann
glehmann force-pushed the gln/align-spans-to-block-size-rupt branch from 8fe8f4a to c538815 Compare May 19, 2026 15:31
@glehmann
glehmann force-pushed the gln/add-max-volume-size-constants-npxp branch from 44d95e4 to c097695 Compare May 19, 2026 15:55
@glehmann
glehmann force-pushed the gln/align-spans-to-block-size-rupt branch from c538815 to 4eb8490 Compare May 19, 2026 15:55
@glehmann
glehmann force-pushed the gln/add-max-volume-size-constants-npxp branch from c097695 to 937fa6d Compare May 20, 2026 19:02
@glehmann
glehmann force-pushed the gln/align-spans-to-block-size-rupt branch from 4eb8490 to 0da90f0 Compare May 20, 2026 19:02
@glehmann
glehmann force-pushed the gln/add-max-volume-size-constants-npxp branch from 937fa6d to 27533d0 Compare May 21, 2026 13:43
@glehmann
glehmann force-pushed the gln/align-spans-to-block-size-rupt branch 2 times, most recently from 2308266 to 84d7873 Compare May 21, 2026 14:07
@glehmann
glehmann force-pushed the gln/add-max-volume-size-constants-npxp branch 2 times, most recently from 2a7ce58 to 4ca7988 Compare May 21, 2026 14:16
@glehmann
glehmann force-pushed the gln/align-spans-to-block-size-rupt branch from 84d7873 to 7f13042 Compare May 21, 2026 14:16
@glehmann
glehmann force-pushed the gln/add-max-volume-size-constants-npxp branch from 4ca7988 to ce36044 Compare May 21, 2026 14:29
@glehmann
glehmann force-pushed the gln/align-spans-to-block-size-rupt branch from 7f13042 to 7d3fc3e Compare May 21, 2026 14:29
@glehmann
glehmann force-pushed the gln/add-max-volume-size-constants-npxp branch from ce36044 to ff961ea Compare May 22, 2026 15:29
@glehmann
glehmann force-pushed the gln/align-spans-to-block-size-rupt branch from 7d3fc3e to b88f1f8 Compare May 22, 2026 15:29
@glehmann
glehmann force-pushed the gln/add-max-volume-size-constants-npxp branch from ff961ea to b882ed3 Compare May 29, 2026 11:37
@glehmann
glehmann force-pushed the gln/align-spans-to-block-size-rupt branch from b88f1f8 to 597a422 Compare May 29, 2026 11:37
…ter performance

Add an optional `align` parameter (default: 1) to align span positions
to a given block size. Only positions are aligned (rounded down); span sizes
are reduced if needed to avoid overlapping the next span's position. When
config.write_volume_cap is large enough, spans are contiguous and cover the
full device with no gaps. Without alignment, write throughput drops from
~700MiB/s to ~200MiB/s on large disks.

Signed-off-by: Gaëtan Lehmann <gaetan.lehmann@vates.tech>
@glehmann
glehmann force-pushed the gln/align-spans-to-block-size-rupt branch from 597a422 to 5076ce0 Compare June 1, 2026 08:48
Base automatically changed from gln/add-max-volume-size-constants-npxp to master June 2, 2026 11:38
@glehmann
glehmann merged commit 1835135 into master Jun 2, 2026
11 checks passed
@glehmann
glehmann deleted the gln/align-spans-to-block-size-rupt branch June 2, 2026 11:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants