Skip to content

storage: limit data written per VDI with --write-volume-cap - #481

Merged
Lankou66 merged 1 commit into
masterfrom
gln/limit-vdi-write-cap-umro
May 12, 2026
Merged

storage: limit data written per VDI with --write-volume-cap#481
Lankou66 merged 1 commit into
masterfrom
gln/limit-vdi-write-cap-umro

Conversation

@glehmann

@glehmann glehmann commented Apr 16, 2026

Copy link
Copy Markdown
Member

Refactor stream management to support arbitrary span distributions.
The new StreamSpan dataclass encapsulates stream metadata and operations,
while partially_populate_device() now supports configurable span counts
and skip patterns.

This required to upgrade randstream to the version 0.6.0 which supports
arbitrary position.

The default cap is 2GiB.

Signed-off-by: Gaëtan Lehmann gaetan.lehmann@vates.tech

This PR is part of a tree containing 19 PRs:

  1. master
  2. storage: avoid removing the xva/vdi before going in the debugger #436master
  3. storage: test large volumes #437storage: avoid removing the xva/vdi before going in the debugger #436
  4. host: include mdadm RAID devices in disk detection #447storage: test large volumes #437
  5. Update randstream to 0.5.0 #446host: include mdadm RAID devices in disk detection #447
  6. storage: Avoid writing the whole device in coalesce tests #449Update randstream to 0.5.0 #446
  7. storage: avoid writing the whole device in migration tests #450storage: Avoid writing the whole device in coalesce tests #449
  8. xva/vdi: only write a small data amount in large volumes for faster tests #452storage: avoid writing the whole device in migration tests #450
  9. storage: test full device write #453xva/vdi: only write a small data amount in large volumes for faster tests #452
  10. storage: test that we can't create a vdi over its max allowed size #454storage: test full device write #453
  11. storage: add jobs for large volume tests #461storage: test that we can't create a vdi over its max allowed size #454
  12. skip large volume tests for zvol and nfsv4 #464storage: add jobs for large volume tests #461
  13. storage: add comprehensive tests for lvmohba storage repositories #470skip large volume tests for zvol and nfsv4 #464
  14. storage: free space for XVA import by destroying source VM first #471storage: add comprehensive tests for lvmohba storage repositories #470
  15. "storage: limit data written per VDI with --write-volume-cap" (this PR) → storage: free space for XVA import by destroying source VM first #471
  16. Add Packer configuration to build a minimal Alpine 3.23 UEFI VM for XCP-ng tests #523storage: limit data written per VDI with --write-volume-cap #481
  17. migration: create a xfs sr on the second host for intra/cross-pool migration #497storage: limit data written per VDI with --write-volume-cap #481
  18. Enhance block device management #498migration: create a xfs sr on the second host for intra/cross-pool migration #497
  19. Add VHD_MAX and QCOW2_MAX symbolic size constants #500Enhance block device management #498
  20. partially_populate_device: align span positions to block size for better performance #509Add VHD_MAX and QCOW2_MAX symbolic size constants #500

@glehmann
glehmann requested review from a team as code owners April 16, 2026 13:56
@glehmann
glehmann force-pushed the gln/limit-vdi-write-cap-umro branch from 6ee5b4d to fc1e701 Compare April 16, 2026 18:00
@glehmann
glehmann force-pushed the gln/fix-xva-import-space-pslw branch from 81ce353 to d273947 Compare April 16, 2026 18:00
@glehmann
glehmann force-pushed the gln/limit-vdi-write-cap-umro branch from fc1e701 to af6499e Compare April 16, 2026 18:38
@glehmann
glehmann force-pushed the gln/fix-xva-import-space-pslw branch 2 times, most recently from f244985 to e3515aa Compare April 17, 2026 13:52
@glehmann
glehmann force-pushed the gln/limit-vdi-write-cap-umro branch 3 times, most recently from 6ddff5e to ef998eb Compare April 21, 2026 06:08
@glehmann
glehmann force-pushed the gln/fix-xva-import-space-pslw branch from e3515aa to 3806fe9 Compare April 21, 2026 06:08
@glehmann
glehmann force-pushed the gln/limit-vdi-write-cap-umro branch from ef998eb to 2c3ab59 Compare April 21, 2026 12:47
@glehmann
glehmann force-pushed the gln/fix-xva-import-space-pslw branch from 3806fe9 to 150f448 Compare April 21, 2026 12:47
@glehmann
glehmann force-pushed the gln/limit-vdi-write-cap-umro branch from 2c3ab59 to 261ee8d Compare April 21, 2026 13:24
@glehmann
glehmann force-pushed the gln/fix-xva-import-space-pslw branch from 150f448 to 067a161 Compare April 21, 2026 13:24
@glehmann
glehmann force-pushed the gln/limit-vdi-write-cap-umro branch from 261ee8d to 36f7634 Compare April 21, 2026 14:03
@glehmann
glehmann force-pushed the gln/fix-xva-import-space-pslw branch from 067a161 to 352e955 Compare April 21, 2026 14:03
Comment thread lib/config.py
ignore_ssh_banner = False
ssh_output_max_lines = 20
volume_size = 1 * GiB
write_volume_cap = 2 * GiB

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it twice volume size ?

if "NFS4" in dispatch_nfs.vdis[0].sr.get_name_label() and config.volume_size > 20 * GiB:
def test_xva_export_import(self, dispatch_nfs: VM, compression: XVACompression, temp_large_dir: str,
defer: Defer) -> None:
if "NFS4" in dispatch_nfs.vdis[0].sr.get_name_label() and config.write_volume_cap > 20 * GiB:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry if asked previously but why magic numbers are duplicated, and not a constant introduced?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because I didn't feel a need for it, I suppose.

Indeed, I think we've already had this discussion :)

Comment thread tests/storage/storage.py
integrity_check = not vm.is_windows
dev = ""
checksums = ('', '', '')
spans: list[StreamSpan] = []

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about using sets in place of list ?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's the type returned by the function that creates them.
And it's convenient to use a list as return type because they are not all equivalent when using skip_span.

Refactor stream management to support arbitrary span distributions.
The new StreamSpan dataclass encapsulates stream metadata and operations,
while partially_populate_device() now supports configurable span counts
and skip patterns.

This required to upgrade randstream to the version 0.6.0 which supports
arbitrary position.
 
The default cap is 2GiB.

Signed-off-by: Gaëtan Lehmann <gaetan.lehmann@vates.tech>
@glehmann
glehmann force-pushed the gln/fix-xva-import-space-pslw branch from 3ee475f to 7a8a2d7 Compare May 12, 2026 13:23
@glehmann
glehmann force-pushed the gln/limit-vdi-write-cap-umro branch from f2fef78 to fdd3cbf Compare May 12, 2026 13:23
Base automatically changed from gln/fix-xva-import-space-pslw to master May 12, 2026 13:29
@Lankou66
Lankou66 merged commit 0d1ec34 into master May 12, 2026
9 checks passed
@Lankou66
Lankou66 deleted the gln/limit-vdi-write-cap-umro branch May 12, 2026 13:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants