storage: test large volumes by glehmann · Pull Request #437 · xcp-ng/xcp-ng-tests

glehmann · 2026-03-26T08:21:09Z

This PR is part of a tree containing 19 PRs:

master
storage: avoid removing the xva/vdi before going in the debugger #436 → master
"storage: test large volumes" (this PR) → storage: avoid removing the xva/vdi before going in the debugger #436
host: include mdadm RAID devices in disk detection #447 → storage: test large volumes #437
Update randstream to 0.5.0 #446 → host: include mdadm RAID devices in disk detection #447
storage: Avoid writing the whole device in coalesce tests #449 → Update randstream to 0.5.0 #446
storage: avoid writing the whole device in migration tests #450 → storage: Avoid writing the whole device in coalesce tests #449
xva/vdi: only write a small data amount in large volumes for faster tests #452 → storage: avoid writing the whole device in migration tests #450
storage: test full device write #453 → xva/vdi: only write a small data amount in large volumes for faster tests #452
storage: test that we can't create a vdi over its max allowed size #454 → storage: test full device write #453
storage: add jobs for large volume tests #461 → storage: test that we can't create a vdi over its max allowed size #454
skip large volume tests for zvol and nfsv4 #464 → storage: add jobs for large volume tests #461
storage: add comprehensive tests for lvmohba storage repositories #470 → skip large volume tests for zvol and nfsv4 #464
storage: free space for XVA import by destroying source VM first #471 → storage: add comprehensive tests for lvmohba storage repositories #470
storage: limit data written per VDI with --write-volume-cap #481 → storage: free space for XVA import by destroying source VM first #471
Add Packer configuration to build a minimal Alpine 3.23 UEFI VM for XCP-ng tests #523 → storage: limit data written per VDI with --write-volume-cap #481
migration: create a xfs sr on the second host for intra/cross-pool migration #497 → storage: limit data written per VDI with --write-volume-cap #481
Enhance block device management #498 → migration: create a xfs sr on the second host for intra/cross-pool migration #497
Add VHD_MAX and QCOW2_MAX symbolic size constants #500 → Enhance block device management #498
partially_populate_device: align span positions to block size for better performance #509 → Add VHD_MAX and QCOW2_MAX symbolic size constants #500

Add pytest command-line option to configure default test VDI sizes, enabling testing of large volumes supported by the new qcow2 feature. The default remains 1GiB to avoid hardware requirements; larger sizes will be added as new jobs in jobs.py. Changes: - Add parse_size() to handle size string parsing (e.g., '2.5TiB', '1GiB') - Remove hardcoded integrity checksums to adapt to any volume size - Introduce randstream() helper to keep code compact and extract checksums - Update all storage fixtures to use configurable volume_size Signed-off-by: Gaëtan Lehmann <gaetan.lehmann@vates.tech>

At module or package scope, we may have a vdi still hanging around, using the space we may need for another test Signed-off-by: Gaëtan Lehmann <gaetan.lehmann@vates.tech>

With larger volume sizes, the exported images may not fit in /tmp. The large_temp_dir fixture still defaults to /tmp when no nfs configuration is available, to keep the tests runnable, with small volume sizes, without a nfs server. Signed-off-by: Gaëtan Lehmann <gaetan.lehmann@vates.tech>

Resize test VM's VDI to match the --volume-size parameter to support testing with different volume sizes. Dynamically calculate expected XVA sizes and handle filesystem resizing on Alpine systems. Signed-off-by: Gaëtan Lehmann <gaetan.lehmann@vates.tech>

Remove hardcoded 'xvdb' device name and use the device name returned by vm.connect_vdi() to support flexible device naming. Signed-off-by: Gaëtan Lehmann <gaetan.lehmann@vates.tech>

stormi · 2026-05-12T12:43:02Z

-    #       to ensure we can export and import large xva images
-    randstream(vm, 'generate --size 500MiB /root/data')
-    randstream(vm, 'validate --expected-checksum 24e905d6 /root/data')
+    is_alpine = vm.ssh_with_result('apk --version').returncode == 0


You could use the method exposed by VM to get the package manager.

stormi · 2026-05-12T12:44:41Z

+    if is_alpine:
+        # growpart is not available in alpine 3.12
+        # vm.ssh('apk add cloud-utils-growpart e2fsprogs-extra')
+        vm.ssh('apk add gawk util-linux e2fsprogs-extra')
+        vm.ssh('wget https://raw.githubusercontent.com/canonical/cloud-utils/main/bin/growpart -O /usr/bin/growpart')
+        vm.ssh('chmod +x /usr/bin/growpart')
+        # TODO: maybe use `findmnt -no SOURCE /` from util-linux to get the blockdevice mounted on /
+        growpart_returncode = vm.ssh_with_result('growpart /dev/xvda 3').returncode
+        assert growpart_returncode in [0, 1] # growpart returns 1 if the size is already the expected one
+        vm.ssh('resize2fs /dev/xvda3')
+        stream_size = volume_size // 2
+    else:
+        stream_size = 500 * MiB


Let's improve the VM object so that it can grow partitions on a wider variety of OSes (RHEL-like, debian...)

Lankou66

LGTM

glehmann requested review from a team as code owners March 26, 2026 08:21

glehmann force-pushed the gln/storage-volume-size-support-vzxu branch from 01c5e43 to 859fc5c Compare March 26, 2026 08:26

glehmann force-pushed the gln/storage-debugger-xva-retention-uxrt branch from 12bb98b to a57c109 Compare March 26, 2026 08:26

glehmann force-pushed the gln/storage-volume-size-support-vzxu branch from 859fc5c to 4cd93f5 Compare March 26, 2026 08:28

last-genius reviewed Mar 27, 2026

View reviewed changes

Comment thread tests/storage/lvm/test_lvm_sr.py

glehmann force-pushed the gln/storage-volume-size-support-vzxu branch from 4cd93f5 to 346eff1 Compare March 27, 2026 10:06

glehmann mentioned this pull request Mar 27, 2026

storage: Snapshot the VM multiple times before exporting/importing #438

Open

glehmann force-pushed the gln/storage-volume-size-support-vzxu branch 2 times, most recently from 5d3fa9b to d96b2a9 Compare March 30, 2026 11:27

glehmann force-pushed the gln/storage-debugger-xva-retention-uxrt branch 2 times, most recently from 080ee74 to c130ebc Compare March 31, 2026 18:27

glehmann force-pushed the gln/storage-volume-size-support-vzxu branch from d96b2a9 to a11ecb2 Compare March 31, 2026 18:27

glehmann changed the title ~~storage: test large volumes~~ gln/storage-volume-size-support-vzxu Mar 31, 2026

glehmann force-pushed the gln/storage-volume-size-support-vzxu branch 2 times, most recently from 9b0818e to 5800d41 Compare March 31, 2026 20:36

glehmann changed the title ~~gln/storage-volume-size-support-vzxu~~ storage: test large volumes Apr 6, 2026

glehmann mentioned this pull request Apr 7, 2026

storage: add jobs for large volume tests #461

Merged

glehmann force-pushed the gln/storage-volume-size-support-vzxu branch 2 times, most recently from a6a78f3 to 0f07c9b Compare April 7, 2026 13:32

glehmann mentioned this pull request Apr 8, 2026

skip large volume tests for zvol and nfsv4 #464

Merged

glehmann force-pushed the gln/storage-volume-size-support-vzxu branch from f4248cf to eb0eac0 Compare April 15, 2026 16:18

stormi approved these changes Apr 15, 2026

View reviewed changes

glehmann mentioned this pull request Apr 16, 2026

storage: limit data written per VDI with --write-volume-cap #481

Merged

glehmann force-pushed the gln/storage-volume-size-support-vzxu branch 2 times, most recently from 9ade0b0 to a63e581 Compare April 17, 2026 13:52

glehmann force-pushed the gln/storage-debugger-xva-retention-uxrt branch from cd1c5d4 to 27d715c Compare April 17, 2026 13:52

glehmann force-pushed the gln/storage-volume-size-support-vzxu branch from a63e581 to d7826f6 Compare April 21, 2026 06:08

glehmann force-pushed the gln/storage-debugger-xva-retention-uxrt branch from 27d715c to aeaca44 Compare April 24, 2026 14:57

glehmann force-pushed the gln/storage-volume-size-support-vzxu branch from d7826f6 to a29798c Compare April 24, 2026 14:57

This was referenced Apr 24, 2026

Enhance block device management #498

Open

migration: create a xfs sr on the second host for intra/cross-pool migration #497

Merged

Add VHD_MAX and QCOW2_MAX symbolic size constants #500

Open

glehmann force-pushed the gln/storage-debugger-xva-retention-uxrt branch from aeaca44 to b2250b7 Compare April 29, 2026 17:39

glehmann force-pushed the gln/storage-volume-size-support-vzxu branch from a29798c to 9d399c6 Compare April 29, 2026 17:39

glehmann mentioned this pull request Apr 29, 2026

partially_populate_device: align span positions to block size for better performance #509

Open

glehmann added 5 commits May 11, 2026 17:48

storage: move the vdi fixtures on thick provisioning to function scope

a305fb5

At module or package scope, we may have a vdi still hanging around, using the space we may need for another test Signed-off-by: Gaëtan Lehmann <gaetan.lehmann@vates.tech>

storage: use dynamic device names instead of hardcoded device names

2d28eea

Remove hardcoded 'xvdb' device name and use the device name returned by vm.connect_vdi() to support flexible device naming. Signed-off-by: Gaëtan Lehmann <gaetan.lehmann@vates.tech>

glehmann force-pushed the gln/storage-debugger-xva-retention-uxrt branch from b2250b7 to 3470670 Compare May 11, 2026 15:49

glehmann force-pushed the gln/storage-volume-size-support-vzxu branch from 9d399c6 to 2d28eea Compare May 11, 2026 15:49

glehmann mentioned this pull request May 11, 2026

Add Packer configuration to build a minimal Alpine 3.23 UEFI VM for XCP-ng tests #523

Draft

Base automatically changed from gln/storage-debugger-xva-retention-uxrt to master May 12, 2026 12:32

stormi reviewed May 12, 2026

View reviewed changes

Lankou66 approved these changes May 12, 2026

View reviewed changes

Millefeuille42 approved these changes May 12, 2026

View reviewed changes

stormi merged commit a650506 into master May 12, 2026
9 checks passed

stormi deleted the gln/storage-volume-size-support-vzxu branch May 12, 2026 12:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

storage: test large volumes#437

storage: test large volumes#437
stormi merged 5 commits into
masterfrom
gln/storage-volume-size-support-vzxu

glehmann commented Mar 26, 2026 •

edited

Loading

Uh oh!

Uh oh!

stormi May 12, 2026

Uh oh!

stormi May 12, 2026

Uh oh!

Lankou66 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

glehmann commented Mar 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

stormi May 12, 2026

Choose a reason for hiding this comment

Uh oh!

stormi May 12, 2026

Choose a reason for hiding this comment

Uh oh!

Lankou66 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

glehmann commented Mar 26, 2026 •

edited

Loading