Test execution ordering: regroup by axis/package to reduce fixture churn - #544
Merged
Conversation
stormi
commented
May 20, 2026
stormi
force-pushed
the
stormi/ordering_constraints
branch
from
May 20, 2026 16:45
d945be8 to
9568250
Compare
ydirson
requested changes
May 21, 2026
ydirson
left a comment
ydirson
left a comment
Collaborator
There was a problem hiding this comment.
Quick test of the overall impact, by comparing before and after the output of:
uv run pytest --setup-plan --image-format=vhd,qcow2 tests/ | sort | uniq -c
$ wc -l *cnt
1026 after.cnt
1304 before.cnt
Diffing shows a few things:
- difference in skipped tests
+ 1 SKIPPED [1] tests/storage/zfsvol/test_zfsvol_sr.py:77: zfsvol doesn't provide vhd-parent
+ 1 tests/storage/zfsvol/test_zfsvol_sr.py::TestZfsvolVm::test_coalesce[None-snapshot] SKIPPED
- investigating those skipped tests, we can se many more
TestZfsvolVmtests scheduled than originally - fixtures
dispatch_nfs['nfs_device_config']anddispatch_nfs['nfs4_device_config']were actually not scheduled before the change, but now are - ...
Actually, the set of tests scheduled is really different. There seems to be both tests that were not scheduled but now are (which seems good(?), but is scary): zfsvol stuff, all MooseFS tests, some glusterfs tests, TestTLSVerification stuff... and tests which were but are not any more (even more scary), most prominently any qcow2 test
glehmann
approved these changes
May 21, 2026
| Check pytest.ini or pytest --markers for marker descriptions. | ||
| - Regroup tests by axis (image_format) and by package (leaf directory) | ||
| """ | ||
| def get_axis(item: pytest.Item) -> Optional[str]: |
Member
There was a problem hiding this comment.
get_axis could be moved out of pytest_collection_modifyitems, next SCHEDULING_AXES, to keep the pytest_collection_modifyitems shorter
Member
Author
There was a problem hiding this comment.
I hesitated, since it's really purely a help function inside the other function, but I don't mind changing.
| # List the items in the order that pytest initially determined, and add extra grouping criteria. | ||
| for item in items: | ||
| axis_order = axis_ordering[get_axis(item)] | ||
| package = item.getparent(pytest.Package) |
Member
There was a problem hiding this comment.
I don't think we have tests outside a package, and I don't think we want that to happen, so you could add
assert package is not Noneand simplify the grouped type a bit to
dict[int, dict[pytest.Package, list[pytest.Item]]]
Member
|
@ydirson I see the same tests skipped with that PR and the commit before, and the comparison with the old and new plan is quite similar (except it looks better in many places). --- /tmp/old.cnt 2026-05-21 17:56:49.893897471 +0200
+++ /tmp/new.cnt 2026-05-21 17:56:56.971822220 +0200
@@ -1,9 +1,9 @@
2
- 1 ============================== 6 skipped in 0.51s ==============================
+ 1 ============================== 6 skipped in 0.46s ==============================
1 collected 671 items
1 configfile: pytest.ini
1 -------------------------------- live log setup --------------------------------
- 1 May 21 17:54:28.983 INFO skip test_chain because it depends on TestManualChaining::test_create
+ 1 May 21 17:54:41.690 INFO skip test_chain because it depends on TestManualChaining::test_create
1 platform linux -- Python 3.11.11, pytest-9.0.2, pluggy-1.6.0
1 plugins: dependency-0.6.1
1 rootdir: /home/glehmann/src/xcp-ng/xcp-ng-tests
@@ -85,45 +85,45 @@
1 SETUP M unix_vm_with_vtpm (fixtures used: snapshotted_halted_uefi_unix_vm)
2 SETUP M unsealed_windows_vm_and_snapshot (fixtures used: running_windows_vm_without_tools)
1 SETUP M vdi_on_cephfs_sr (fixtures used: cephfs_sr)
- 6 SETUP M vdi_on_ext_sr (fixtures used: ext_sr)
+ 4 SETUP M vdi_on_ext_sr (fixtures used: ext_sr)
1 SETUP M vdi_on_glusterfs_sr (fixtures used: glusterfs_sr)
2 SETUP M vdi_on_largeblock_sr (fixtures used: largeblock_sr)
1 SETUP M vdi_on_linstor_sr (fixtures used: linstor_sr)
1 SETUP M vdi_on_moosefs_sr (fixtures used: moosefs_sr)
2 SETUP M vdi_on_nfs_sr (fixtures used: nfs_sr)
- 7 SETUP M vdi_on_xfs_sr (fixtures used: xfs_sr)
- 7 SETUP M vdi_on_zfs_sr (fixtures used: zfs_sr)
+ 4 SETUP M vdi_on_xfs_sr (fixtures used: xfs_sr)
+ 4 SETUP M vdi_on_zfs_sr (fixtures used: zfs_sr)
1 SETUP M vdi_on_zfsvol_sr (fixtures used: zfsvol_sr)
1 SETUP M vdis (fixtures used: host, local_sr_on_hostA1)
6 SETUP M vm_on_cephfs_sr (fixtures used: cephfs_sr, host, vm_ref)
- 12 SETUP M vm_on_ext_sr (fixtures used: ext_sr, host, vm_ref)
+ 8 SETUP M vm_on_ext_sr (fixtures used: ext_sr, host, vm_ref)
6 SETUP M vm_on_glusterfs_sr (fixtures used: glusterfs_sr, host, vm_ref)
6 SETUP M vm_on_largeblock_sr (fixtures used: host, largeblock_sr, vm_ref)
- 5 SETUP M vm_on_linstor_sr (fixtures used: host, linstor_sr, vm_ref)
- 12 SETUP M vm_on_lvmohba_sr (fixtures used: host, lvmohba_sr, vm_ref)
- 12 SETUP M vm_on_lvmoiscsi_sr (fixtures used: host, lvmoiscsi_sr, vm_ref)
- 12 SETUP M vm_on_lvm_sr (fixtures used: host, lvm_sr, vm_ref)
- 4 SETUP M vm_on_moosefs_sr (fixtures used: host, moosefs_sr, vm_ref)
+ 6 SETUP M vm_on_linstor_sr (fixtures used: host, linstor_sr, vm_ref)
+ 8 SETUP M vm_on_lvmohba_sr (fixtures used: host, lvmohba_sr, vm_ref)
+ 8 SETUP M vm_on_lvmoiscsi_sr (fixtures used: host, lvmoiscsi_sr, vm_ref)
+ 8 SETUP M vm_on_lvm_sr (fixtures used: host, lvm_sr, vm_ref)
+ 6 SETUP M vm_on_moosefs_sr (fixtures used: host, moosefs_sr, vm_ref)
2 SETUP M vm_on_nfs_sr (fixtures used: host, nfs_sr, vm_ref)
- 12 SETUP M vm_on_xfs_sr (fixtures used: host, vm_ref, xfs_sr)
- 12 SETUP M vm_on_zfs_sr (fixtures used: host, vm_ref, zfs_sr)
- 3 SETUP M vm_on_zfsvol_sr (fixtures used: host, vm_ref, zfsvol_sr)
- 118 SETUP M vm_ref[None]
+ 8 SETUP M vm_on_xfs_sr (fixtures used: host, vm_ref, xfs_sr)
+ 8 SETUP M vm_on_zfs_sr (fixtures used: host, vm_ref, zfs_sr)
+ 5 SETUP M vm_on_zfsvol_sr (fixtures used: host, vm_ref, zfsvol_sr)
+ 126 SETUP M vm_ref[None]
1 SETUP M vm_with_vbds (fixtures used: host, imported_vm, vdis)
4 SETUP M windows_vm (fixtures used: imported_vm)
- 3 SETUP P cephfs_device_config
- 3 SETUP P cephfs_sr (fixtures used: cephfs_device_config, host, pool_with_ceph)
- 9 SETUP P ext_sr (fixtures used: host, image_format, unused_512B_disks)
+ 2 SETUP P cephfs_device_config
+ 2 SETUP P cephfs_sr (fixtures used: cephfs_device_config, host, pool_with_ceph)
+ 2 SETUP P ext_sr (fixtures used: host, image_format, unused_512B_disks)
3 SETUP P formatted_and_mounted_ext4_disk (fixtures used: host, unused_512B_disks)
1 SETUP P fsp_config (fixtures used: host_with_fsp)
1 SETUP P fsp_sr (fixtures used: fsp_config, host_with_fsp)
- 3 SETUP P gluster_disk (fixtures used: _glusterfs_config, pool_with_unused_512B_disk, unused_512B_disks)
- 3 SETUP P _glusterfs_config
- 3 SETUP P glusterfs_device_config (fixtures used: host)
- 3 SETUP P glusterfs_sr (fixtures used: _glusterfs_config, gluster_volume_started, glusterfs_device_config, host, pool_with_glusterfs)
- 3 SETUP P gluster_volume_started (fixtures used: _glusterfs_config, gluster_disk, host, hostA2)
- 21 SETUP P hostA2_with_xfsprogs (fixtures used: _xfs_config_on_hostA2, hostA2, image_format)
- 22 SETUP P hostB1_with_xfsprogs (fixtures used: _xfs_config_on_hostB1, hostB1, image_format)
+ 2 SETUP P gluster_disk (fixtures used: _glusterfs_config, pool_with_unused_512B_disk, unused_512B_disks)
+ 2 SETUP P _glusterfs_config
+ 2 SETUP P glusterfs_device_config (fixtures used: host)
+ 2 SETUP P glusterfs_sr (fixtures used: _glusterfs_config, gluster_volume_started, glusterfs_device_config, host, pool_with_glusterfs)
+ 2 SETUP P gluster_volume_started (fixtures used: _glusterfs_config, gluster_disk, host, hostA2)
+ 4 SETUP P hostA2_with_xfsprogs (fixtures used: _xfs_config_on_hostA2, hostA2, image_format)
+ 4 SETUP P hostB1_with_xfsprogs (fixtures used: _xfs_config_on_hostB1, hostB1, image_format)
1 SETUP P host_no_sdn_controller (fixtures used: host)
1 SETUP P host_with_dom0_tests (fixtures used: host_with_saved_yum_state)
1 SETUP P host_with_dynamically_disabled_ept_sp (fixtures used: host)
@@ -133,51 +133,51 @@
1 SETUP P host_with_netdata (fixtures used: host_with_saved_yum_state)
1 SETUP P host_without_mlx_card (fixtures used: host)
1 SETUP P host_without_mlx_compat_loaded (fixtures used: host_with_saved_yum_state)
- 7 SETUP P host_without_zfs (fixtures used: host)
+ 3 SETUP P host_without_zfs (fixtures used: host)
1 SETUP P host_with_perf (fixtures used: host_at_least_8_3, host_with_saved_yum_state)
1 SETUP P host_with_ring0_tests (fixtures used: host_with_saved_yum_state)
1 SETUP P host_with_runx_repo (fixtures used: host_with_saved_yum_state_toolstack_restart)
- 14 SETUP P host_with_saved_yum_state (fixtures used: host)
- 4 SETUP P host_with_saved_yum_state_toolstack_restart (fixtures used: host)
- 16 SETUP P host_with_xfsprogs (fixtures used: _xfs_config, host, image_format)
+ 10 SETUP P host_with_saved_yum_state (fixtures used: host)
+ 3 SETUP P host_with_saved_yum_state_toolstack_restart (fixtures used: host)
+ 2 SETUP P host_with_xfsprogs (fixtures used: _xfs_config, host, image_format)
1 SETUP P host_with_zfs (fixtures used: host_with_saved_yum_state, host_without_zfs)
- 16 SETUP P host_with_zfs (fixtures used: host_with_saved_yum_state, host_without_zfs, image_format)
- 3 SETUP P host_with_zfsvol (fixtures used: host_with_saved_yum_state_toolstack_restart)
+ 2 SETUP P host_with_zfs (fixtures used: host_with_saved_yum_state, host_without_zfs, image_format)
+ 2 SETUP P host_with_zfsvol (fixtures used: host_with_saved_yum_state_toolstack_restart)
2 SETUP P largeblock_sr (fixtures used: host, image_format, unused_4k_disks)
- 1 SETUP P _linstor_config
- 1 SETUP P linstor_redundancy (fixtures used: pool_with_linstor)
- 1 SETUP P linstor_sr (fixtures used: _linstor_config, linstor_redundancy, lvm_disks, pool_with_linstor, provisioning_type, storage_pool_name)
- 1 SETUP P lvm_disks (fixtures used: pool_with_unused_512B_disk, provisioning_type, unused_512B_disks)
- 6 SETUP P lvmohba_device_config
- 16 SETUP P lvmohba_sr (fixtures used: host, image_format, lvmohba_device_config)
- 6 SETUP P lvmoiscsi_device_config
- 16 SETUP P lvmoiscsi_sr (fixtures used: host, image_format, lvmoiscsi_device_config)
- 16 SETUP P lvm_sr (fixtures used: host, image_format, unused_512B_disks)
+ 2 SETUP P _linstor_config
+ 2 SETUP P linstor_redundancy (fixtures used: pool_with_linstor)
+ 2 SETUP P linstor_sr (fixtures used: _linstor_config, linstor_redundancy, lvm_disks, pool_with_linstor, provisioning_type, storage_pool_name)
+ 2 SETUP P lvm_disks (fixtures used: pool_with_unused_512B_disk, provisioning_type, unused_512B_disks)
+ 2 SETUP P lvmohba_device_config
+ 2 SETUP P lvmohba_sr (fixtures used: host, image_format, lvmohba_device_config)
+ 2 SETUP P lvmoiscsi_device_config
+ 2 SETUP P lvmoiscsi_sr (fixtures used: host, image_format, lvmoiscsi_device_config)
+ 2 SETUP P lvm_sr (fixtures used: host, image_format, unused_512B_disks)
2 SETUP P moosefs_device_config
2 SETUP P moosefs_sr (fixtures used: moosefs_device_config, pool_with_moosefs_enabled)
3 SETUP P nfs_device_config
4 SETUP P nfs_sr (fixtures used: host, image_format)
- 6 SETUP P nfs_sr (fixtures used: host, image_format, nfs_device_config)
- 3 SETUP P pool_with_ceph (fixtures used: pool_with_saved_yum_state, pool_without_ceph)
- 3 SETUP P pool_with_glusterfs (fixtures used: _glusterfs_config, pool_with_saved_yum_state, pool_without_glusterfs)
- 1 SETUP P pool_with_linstor (fixtures used: _linstor_config, hostA2, lvm_disks, pool_with_saved_yum_state)
+ 4 SETUP P nfs_sr (fixtures used: host, image_format, nfs_device_config)
+ 2 SETUP P pool_with_ceph (fixtures used: pool_with_saved_yum_state, pool_without_ceph)
+ 2 SETUP P pool_with_glusterfs (fixtures used: _glusterfs_config, pool_with_saved_yum_state, pool_without_glusterfs)
+ 2 SETUP P pool_with_linstor (fixtures used: _linstor_config, hostA2, lvm_disks, pool_with_saved_yum_state)
2 SETUP P pool_with_moosefs_enabled (fixtures used: pool_with_moosefs_installed)
2 SETUP P pool_with_moosefs_installed (fixtures used: pool_with_saved_yum_state)
- 3 SETUP P pool_without_ceph (fixtures used: host)
- 3 SETUP P pool_without_glusterfs (fixtures used: host)
- 9 SETUP P pool_with_saved_yum_state (fixtures used: host)
- 10 SETUP P sr_disk_wiped (fixtures used: host, unused_512B_disks)
- 1 SETUP P storage_pool_name (fixtures used: provisioning_type)
- 6 SETUP P _xfs_config
- 3 SETUP P _xfs_config_on_hostA2
- 3 SETUP P _xfs_config_on_hostB1
- 16 SETUP P xfs_sr (fixtures used: _xfs_config, host_with_xfsprogs, image_format, unused_512B_disks)
- 21 SETUP P xfs_sr_on_hostA2 (fixtures used: _xfs_config_on_hostA2, hostA2_with_xfsprogs, image_format, unused_512B_disks)
- 22 SETUP P xfs_sr_on_hostB1 (fixtures used: _xfs_config_on_hostB1, hostB1_with_xfsprogs, image_format, unused_512B_disks)
+ 2 SETUP P pool_without_ceph (fixtures used: host)
+ 2 SETUP P pool_without_glusterfs (fixtures used: host)
+ 8 SETUP P pool_with_saved_yum_state (fixtures used: host)
+ 5 SETUP P sr_disk_wiped (fixtures used: host, unused_512B_disks)
+ 2 SETUP P storage_pool_name (fixtures used: provisioning_type)
+ 2 SETUP P _xfs_config
+ 4 SETUP P _xfs_config_on_hostA2
+ 4 SETUP P _xfs_config_on_hostB1
+ 2 SETUP P xfs_sr (fixtures used: _xfs_config, host_with_xfsprogs, image_format, unused_512B_disks)
+ 4 SETUP P xfs_sr_on_hostA2 (fixtures used: _xfs_config_on_hostA2, hostA2_with_xfsprogs, image_format, unused_512B_disks)
+ 4 SETUP P xfs_sr_on_hostB1 (fixtures used: _xfs_config_on_hostB1, hostB1_with_xfsprogs, image_format, unused_512B_disks)
1 SETUP P xtf_runner (fixtures used: host_with_git_and_gcc_and_py3)
- 16 SETUP P zfs_sr (fixtures used: host, image_format, zpool_vol0)
- 3 SETUP P zfsvol_sr (fixtures used: host, host_with_zfsvol, sr_disk_wiped)
- 17 SETUP P zpool_vol0 (fixtures used: host_with_zfs, sr_disk_wiped)
+ 2 SETUP P zfs_sr (fixtures used: host, image_format, zpool_vol0)
+ 2 SETUP P zfsvol_sr (fixtures used: host, host_with_zfsvol, sr_disk_wiped)
+ 3 SETUP P zpool_vol0 (fixtures used: host_with_zfs, sr_disk_wiped)
1 SETUP S disks (fixtures used: pools_hosts_by_name_or_ip, pytestconfig)
1 SETUP S enabled_pci_uuid (fixtures used: enabled_pgpu_uuid, host)
1 SETUP S enabled_pgpu_uuid (fixtures used: host)
@@ -193,8 +193,8 @@
1 SETUP S hosts (fixtures used: pytestconfig)
1 SETUP S hosts_with_xo (fixtures used: hosts, registered_xo_cli)
1 SETUP S host_with_hsts (fixtures used: host)
- 44 SETUP S image_format['qcow2']
- 44 SETUP S image_format['vhd']
+ 1 SETUP S image_format['qcow2']
+ 1 SETUP S image_format['vhd']
1 SETUP S local_sr_on_hostA1 (fixtures used: hostA1)
1 SETUP S local_sr_on_hostA2 (fixtures used: hostA2)
1 SETUP S pools_hosts_by_name_or_ip (fixtures used: hosts)
@@ -292,45 +292,45 @@
1 TEARDOWN M unix_vm_with_vtpm
2 TEARDOWN M unsealed_windows_vm_and_snapshot
1 TEARDOWN M vdi_on_cephfs_sr
- 6 TEARDOWN M vdi_on_ext_sr
+ 4 TEARDOWN M vdi_on_ext_sr
1 TEARDOWN M vdi_on_glusterfs_sr
2 TEARDOWN M vdi_on_largeblock_sr
1 TEARDOWN M vdi_on_linstor_sr
1 TEARDOWN M vdi_on_moosefs_sr
2 TEARDOWN M vdi_on_nfs_sr
- 7 TEARDOWN M vdi_on_xfs_sr
- 7 TEARDOWN M vdi_on_zfs_sr
+ 4 TEARDOWN M vdi_on_xfs_sr
+ 4 TEARDOWN M vdi_on_zfs_sr
1 TEARDOWN M vdi_on_zfsvol_sr
1 TEARDOWN M vdis
6 TEARDOWN M vm_on_cephfs_sr
- 12 TEARDOWN M vm_on_ext_sr
+ 8 TEARDOWN M vm_on_ext_sr
6 TEARDOWN M vm_on_glusterfs_sr
6 TEARDOWN M vm_on_largeblock_sr
- 5 TEARDOWN M vm_on_linstor_sr
- 12 TEARDOWN M vm_on_lvmohba_sr
- 12 TEARDOWN M vm_on_lvmoiscsi_sr
- 12 TEARDOWN M vm_on_lvm_sr
- 4 TEARDOWN M vm_on_moosefs_sr
+ 6 TEARDOWN M vm_on_linstor_sr
+ 8 TEARDOWN M vm_on_lvmohba_sr
+ 8 TEARDOWN M vm_on_lvmoiscsi_sr
+ 8 TEARDOWN M vm_on_lvm_sr
+ 6 TEARDOWN M vm_on_moosefs_sr
2 TEARDOWN M vm_on_nfs_sr
- 12 TEARDOWN M vm_on_xfs_sr
- 12 TEARDOWN M vm_on_zfs_sr
- 3 TEARDOWN M vm_on_zfsvol_sr
- 118 TEARDOWN M vm_ref[None]
+ 8 TEARDOWN M vm_on_xfs_sr
+ 8 TEARDOWN M vm_on_zfs_sr
+ 5 TEARDOWN M vm_on_zfsvol_sr
+ 126 TEARDOWN M vm_ref[None]
1 TEARDOWN M vm_with_vbds
4 TEARDOWN M windows_vm
- 3 TEARDOWN P cephfs_device_config
- 3 TEARDOWN P cephfs_sr
- 9 TEARDOWN P ext_sr
+ 2 TEARDOWN P cephfs_device_config
+ 2 TEARDOWN P cephfs_sr
+ 2 TEARDOWN P ext_sr
3 TEARDOWN P formatted_and_mounted_ext4_disk
1 TEARDOWN P fsp_config
1 TEARDOWN P fsp_sr
- 3 TEARDOWN P gluster_disk
- 3 TEARDOWN P _glusterfs_config
- 3 TEARDOWN P glusterfs_device_config
- 3 TEARDOWN P glusterfs_sr
- 3 TEARDOWN P gluster_volume_started
- 21 TEARDOWN P hostA2_with_xfsprogs
- 22 TEARDOWN P hostB1_with_xfsprogs
+ 2 TEARDOWN P gluster_disk
+ 2 TEARDOWN P _glusterfs_config
+ 2 TEARDOWN P glusterfs_device_config
+ 2 TEARDOWN P glusterfs_sr
+ 2 TEARDOWN P gluster_volume_started
+ 4 TEARDOWN P hostA2_with_xfsprogs
+ 4 TEARDOWN P hostB1_with_xfsprogs
1 TEARDOWN P host_no_sdn_controller
1 TEARDOWN P host_with_dom0_tests
1 TEARDOWN P host_with_dynamically_disabled_ept_sp
@@ -340,49 +340,49 @@
1 TEARDOWN P host_with_netdata
1 TEARDOWN P host_without_mlx_card
1 TEARDOWN P host_without_mlx_compat_loaded
- 7 TEARDOWN P host_without_zfs
+ 3 TEARDOWN P host_without_zfs
1 TEARDOWN P host_with_perf
1 TEARDOWN P host_with_ring0_tests
1 TEARDOWN P host_with_runx_repo
- 14 TEARDOWN P host_with_saved_yum_state
- 4 TEARDOWN P host_with_saved_yum_state_toolstack_restart
- 16 TEARDOWN P host_with_xfsprogs
- 17 TEARDOWN P host_with_zfs
- 3 TEARDOWN P host_with_zfsvol
+ 10 TEARDOWN P host_with_saved_yum_state
+ 3 TEARDOWN P host_with_saved_yum_state_toolstack_restart
+ 2 TEARDOWN P host_with_xfsprogs
+ 3 TEARDOWN P host_with_zfs
+ 2 TEARDOWN P host_with_zfsvol
2 TEARDOWN P largeblock_sr
- 1 TEARDOWN P _linstor_config
- 1 TEARDOWN P linstor_redundancy
- 1 TEARDOWN P linstor_sr
- 1 TEARDOWN P lvm_disks
- 6 TEARDOWN P lvmohba_device_config
- 16 TEARDOWN P lvmohba_sr
- 6 TEARDOWN P lvmoiscsi_device_config
- 16 TEARDOWN P lvmoiscsi_sr
- 16 TEARDOWN P lvm_sr
+ 2 TEARDOWN P _linstor_config
+ 2 TEARDOWN P linstor_redundancy
+ 2 TEARDOWN P linstor_sr
+ 2 TEARDOWN P lvm_disks
+ 2 TEARDOWN P lvmohba_device_config
+ 2 TEARDOWN P lvmohba_sr
+ 2 TEARDOWN P lvmoiscsi_device_config
+ 2 TEARDOWN P lvmoiscsi_sr
+ 2 TEARDOWN P lvm_sr
2 TEARDOWN P moosefs_device_config
2 TEARDOWN P moosefs_sr
3 TEARDOWN P nfs_device_config
- 10 TEARDOWN P nfs_sr
- 3 TEARDOWN P pool_with_ceph
- 3 TEARDOWN P pool_with_glusterfs
- 1 TEARDOWN P pool_with_linstor
+ 8 TEARDOWN P nfs_sr
+ 2 TEARDOWN P pool_with_ceph
+ 2 TEARDOWN P pool_with_glusterfs
+ 2 TEARDOWN P pool_with_linstor
2 TEARDOWN P pool_with_moosefs_enabled
2 TEARDOWN P pool_with_moosefs_installed
- 3 TEARDOWN P pool_without_ceph
- 3 TEARDOWN P pool_without_glusterfs
- 9 TEARDOWN P pool_with_saved_yum_state
- 10 TEARDOWN P sr_disk_wiped
- 1 TEARDOWN P storage_pool_name
- 6 TEARDOWN P _xfs_config
- 3 TEARDOWN P _xfs_config_on_hostA2
- 3 TEARDOWN P _xfs_config_on_hostB1
- 16 TEARDOWN P xfs_sr
- 21 TEARDOWN P xfs_sr_on_hostA2
- 22 TEARDOWN P xfs_sr_on_hostB1
+ 2 TEARDOWN P pool_without_ceph
+ 2 TEARDOWN P pool_without_glusterfs
+ 8 TEARDOWN P pool_with_saved_yum_state
+ 5 TEARDOWN P sr_disk_wiped
+ 2 TEARDOWN P storage_pool_name
+ 2 TEARDOWN P _xfs_config
+ 4 TEARDOWN P _xfs_config_on_hostA2
+ 4 TEARDOWN P _xfs_config_on_hostB1
+ 2 TEARDOWN P xfs_sr
+ 4 TEARDOWN P xfs_sr_on_hostA2
+ 4 TEARDOWN P xfs_sr_on_hostB1
1 TEARDOWN P xtf_runner
- 16 TEARDOWN P zfs_sr
- 3 TEARDOWN P zfsvol_sr
- 17 TEARDOWN P zpool_vol0
+ 2 TEARDOWN P zfs_sr
+ 2 TEARDOWN P zfsvol_sr
+ 3 TEARDOWN P zpool_vol0
1 TEARDOWN S disks
1 TEARDOWN S enabled_pci_uuid
1 TEARDOWN S enabled_pgpu_uuid
@@ -398,8 +398,8 @@
1 TEARDOWN S hosts
1 TEARDOWN S hosts_with_xo
1 TEARDOWN S host_with_hsts
- 44 TEARDOWN S image_format['qcow2']
- 44 TEARDOWN S image_format['vhd']
+ 1 TEARDOWN S image_format['qcow2']
+ 1 TEARDOWN S image_format['vhd']
1 TEARDOWN S local_sr_on_hostA1
1 TEARDOWN S local_sr_on_hostA2
1 TEARDOWN S pools_hosts_by_name_or_ipAre you sure you're comparing the right versions? |
Contributor
|
I generated a quick report from those diff, the difference is quite impressive: |
Member
|
I started a full run with vhd and qcow2. Wait & see |
stormi
force-pushed
the
stormi/ordering_constraints
branch
from
June 10, 2026 15:30
9568250 to
7ddd7a2
Compare
Member
Author
|
Rebased on master as it was starting to lag behind. I'll apply Gaëtan's suggestions now. |
stormi
force-pushed
the
stormi/ordering_constraints
branch
from
June 10, 2026 15:47
7ddd7a2 to
c6afd80
Compare
pytest's default ordering could interleave image_format=vhd/qcow2 tests in a way that repeatedly destroys and recreates SRs. It could also choose to cross package boundaries as an attempt to optimize test runs, but we'd prefer to see tests logically grouped, for easier log analysis and fixture management (which is perfectible and which we'll have to improve anyway). We start with pytest's computed ordering, then regroup tests by parametrized axis (currently only image_format) and by leaf python package, in order to keep related tests together and reduce needless costly context switching. Axis-less tests intentionally share the same ordering level as the first parametrized axis so they stay near their original position instead of being grouped at the beginning of the run. Interfering with pytest's ordering is not something to do lightly, so it's possible that we may discover issues caused by this. But meanwhile, this has shown a notable reduction (10%) of the number of setup/teardown operations when both vhd and qcow2 image formats are set, and test order closer to what we'd intuitively expect. Signed-off-by: Samuel Verschelde <stormi-xcp@ylix.fr>
stormi
force-pushed
the
stormi/ordering_constraints
branch
from
June 10, 2026 16:05
c6afd80 to
4fe1e21
Compare
Member
Author
|
This PR is ready. However I haven't tested the last pushes (rebase + apply Gaëtan's suggestions) thoroughly. I only ran a --setup-plan on tests/ in order to detect obvious failures. @vxgmichel could you run the test you had done previously in order to verify the output? |
Contributor
Just to be clear, it wasn't a test, just a script to summarize the differences of the two |
Member
Author
Thanks, that's the "test" I had in mind. It tends to show that I haven't broken the PR when I applied the review suggestions. Then I think we're good to go. Ping @rzr @ydirson. |
Member
Author
|
I think we'll give this PR one more day for additional reviews, then merge if none. |
Member
Author
As promised/warned, merging! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
pytest's default ordering could interleave image_format=vhd/qcow2 tests in a way that repeatedly destroys and recreates SRs. It could also choose to cross package boundaries as an attempt to optimize test runs, but we'd prefer to see tests logically grouped, for easier log analysis and fixture management (which is perfectible and which we'll have to improve anyway).
We start with pytest's computed ordering, then regroup tests by parametrized axis (currently only image_format) and by leaf python package, in order to keep related tests together and reduce needless costly context switching.
Axis-less tests intentionally share the same ordering level as the first parametrized axis so they stay near their original position instead of being grouped at the beginning of the run.
Interfering with pytest's ordering is not something to do lightly, so it's possible that we may discover issues caused by this. But meanwhile, this has shown a notable reduction (10%) of the number of setup/teardown operations when both vhd and qcow2 image formats are set, and test order closer to what we'd intuitively expect.