Introduce TOML-based configuration to replace data.py

The previous data.py approach had several shortcomings:
- No base configuration: users had to start from scratch or copy data.py-dist
- No environment management: working with multiple infra required to manually
  copy the data.py files
- No clear default and customization separation: the data.py, once copied
  from data.py-dist and customized, was mixing the defaults and the
  customizations

The new TOML config system addresses this by:
- Providing a base config.toml maintained by the project with sensible defaults,
  which can be updated without user action
- Supporting per-environment override files (config.<name>.toml) so users only
  specify what differs from the defaults, making their customizations explicit

Options have been grouped by sections for clarity.

Configuration sections and their contents:
- [host]: default XAPI credentials (user, password)
- [hosts]: per-host credential overrides
- [network]: management network name
- [pxe]: PXE server settings (arp_server)
- [vm]: VM settings (default_sr, images, equivalents, def_url)
- [install]: installer settings (answerfiles, iso_remaster, isos)
- [guest_tools]: guest tools packages and installer ISOs (win, other, installed, download_url)
- [ssh]: SSH client settings (pubkey, output_max_lines, ignore_banner)
- [storage]: storage backend configs (nfs, nfs4, nfs_iso, cifs_iso, cephfs, moosefs, lvmoiscsi)
- Root keys: objects_name_prefix, dns_server

Also includes config-schema.json, lib/config_loader.py (Pydantic models),
and scripts/migrate_data_py.py to help users migrate from data.py.

Signed-off-by: Gaëtan Lehmann <gaetan.lehmann@vates.tech>

Author: glehmann

.gitignore

@@ -4,3 +4,5 @@ data.py
 vm_data.py
 /scripts/guests/windows/id_rsa.pub
 .envrc
+# Config overrides (user environment-specific)
+config.*.toml

Makefile

@@ -11,10 +11,5 @@ check: ruff autopep8 flake8 mypy pyright
 # It runs on all files managed by git (untracked files are not modified)
 fix: ruff-fix autopep8-fix
 
-data.py:
-	@test -r data.py || echo "File 'data.py' does not exist. Refer to https://github.com/xcp-ng/xcp-ng-tests#configuration." && exit 1
-
-mypy pyright ruff: data.py
-
 ruff ruff-fix autopep8 autopep8-fix flake8 mypy pyright:
 	uv run prek -a $@

README.md

@@ -147,10 +147,68 @@ For Guest UEFI Secure Boot tests, the requirements are:
     * `efitools` for uefistored (in 8.2) or varstored (in 8.3+) auth var tests
     * `util-linux` for uefistored (in 8.2) or varstored (in 8.3+) auth var tests in Alpine VMs
 
-Many tests have specific requirements, detailed in a comment at the top of the test file: minimal number of hosts in a pool, number of pools, VMs with specific characteristics (OS, BIOS vs UEFI, additional tools installed in the VM, additional networks in the pool, presence of an unused disk on one host or every host...). Markers, jobs defined in `jobs.py` (`./jobs.py show JOBNAME` will display the requirements and the reference to a VM or VM group), VMs and VM groups defined in `vm-data.py-dist` may all help understanding what tests can run with what VMs.
+Many tests have specific requirements, detailed in a comment at the top of the test file: minimal number of hosts in a pool, number of pools, VMs with specific characteristics (OS, BIOS vs UEFI, additional tools installed in the VM, additional networks in the pool, presence of an unused disk on one host or every host...). Markers, jobs defined in `jobs.py` (`./jobs.py show JOBNAME` will display the requirements and the reference to a VM or VM group), VMs and VM groups defined in `vm_data.py` may all help understanding what tests can run with what VMs.
 
 ## Configuration
-The main configuration file is data.py. Copy data.py-dist to data.py and modify it if needed.
+
+### Using config.toml
+
+Test configuration is managed via TOML files. The project includes a `config.toml` file with all default settings.
+
+#### Default configuration
+
+The `config.toml` file in the repository contains all default settings for:
+- Host SSH credentials and per-host overrides
+- Network definitions
+- PXE server configuration
+- VM images and ISO definitions
+- Storage device configurations (NFS, CIFS, CephFS, MooseFS, LVM iSCSI)
+- Guest tools (Windows, other)
+- Test utilities and SSH keys
+
+#### Custom configuration
+
+To customize settings for your environment:
+
+1. **Create a custom config file** (e.g., `config.local.toml`):
+   ```toml
+   [host]
+   default_user = "root"
+   default_password = "your-password"
+
+   [host.per_host]
+   "192.168.1.10" = { user = "custom_user", password = "custom_pass" }
+
+   [pxe]
+   config_server = "pxe.example.com"
+   arp_server = "pxe.example.com"
+
+   [storage.nfs]
+   server = "10.0.0.2"
+   serverpath = "/mnt/shared"
+   ```
+
+2. **Run tests with custom config**:
+   ```bash
+   pytest --config=local --hosts=10.0.0.1
+   ```
+
+   This loads `config.toml` first, then merges `config.local.toml` on top.
+
+#### Configuration file paths
+
+- `config.toml` — default config (always loaded)
+- `config.default.toml` — local defaults (auto-loaded if exists and no `--config` specified)
+- `config.NAME.toml` — environment-specific overrides (loaded with `--config=NAME`)
+
+The `--config` flag is optional. If not specified:
+1. `config.toml` is loaded first
+2. If `config.default.toml` exists, it is merged on top (auto-detected)
+
+This allows you to:
+- Commit `config.toml` with project defaults to version control
+- Create `config.default.toml` locally (ignored by git) for your standard environment
+- Create `config.prod.toml`, `config.ci.toml`, etc. for other environments and select with `--config=prod`
 
 ## Running tests
 
@@ -169,8 +227,8 @@ The `--hosts` parameter can be specified several times. Then `pytest` will run t
 When a test requires a single pool of several hosts, only mention the master host in the `--hosts` option.
 
 Some tests accept an optional `--vm=OVA_URL|VM_key|IP_address` parameter. Those are tests that will import a VM before testing stuff on it:
-* `OVA_URL` is a URL to download an OVA. It can also be simply a filename, if your `data.py`'s `DEF_VM_URL` is correctly defined.
-* `VM_key` refers to a key in `data.py`'s `VM_IMAGES` dict. Example: `mini-linux-x86_64-uefi`.
+* `OVA_URL` is a URL to download an OVA. It can also be simply a filename, if your `config.toml`'s `vm.def_url` is correctly defined.
+* `VM_key` refers to a key in `config.toml`'s `vm.images` section. Example: `mini-linux-x86_64-uefi`.
 * `IP_address` allows you to reuse an existing running VM, skipping the whole import, start, wait for VM to be up setup. Can be useful as a development tool. Some tests that accept `--vm` do not support it.
 If `--vm` is not specified, defaults defined by the tests will be used.
 The `--vm` parameter can be specified several times. Then pytest will run several instances of the tests sequentially, one for each VM.
@@ -237,8 +295,6 @@ We wanted the job definitions to be in this git repository, that's why the job d
 
 To use `./jobs.py`, you also need to populate `vm_data.py` to define the VM groups that are necessary to run jobs (unless `--vm` is provided on the command line to override the defaults).
 
-The output of commands below is given as example and may not reflect the current state of the jobs definitions.
-
 #### List jobs
 ```
 $ ./jobs.py list
@@ -602,10 +658,10 @@ python scripts/test_install_xcpng.py 10.0.0.2 f0f5f010-80c6-25ae-44a2-1fb154e32d
 ```
 Note: in case of restore, the version must be that of the installer (here 8.2.1), not the version of XCP-ng that will be restored.
 
-The script requires the addressable name or IP of the PXE config server to be defined in `data.py`:
-```
-# PXE config server for automated XCP-ng installation
-PXE_CONFIG_SERVER = 'pxe'
+The script requires the addressable name or IP of the PXE config server to be defined in `config.toml`:
+```toml
+[pxe]
+config_server = "pxe"
 ```
 
 The `installer` parameter is optional. If you leave it empty it will be automatically defined as `http://<PXE_CONFIG_SERVER>/installers/xcp-ng/<version>/`.

config.toml

@@ -0,0 +1,260 @@
+# Configuration file for XCP-ng tests
+# To be customized for your environment
+
+# The following prefix will be added to the `name-label` parameter of XAPI objects
+# that the tests will create or import, such as VMs and SRs.
+# Default value: [your login/user]
+# Prefix for test-created objects in XAPI (empty = use username)
+objects_name_prefix = ""
+
+# This should be a working DNS server that's not used by any VM images.
+dns_server = "1.1.1.1"
+
+# Default size of VDIs created for storage tests.
+volume_size = "1 GiB"
+
+# Maximum amount of data written to a volume during storage tests.
+write_volume_cap = "2 GiB"
+
+# Block size to align span positions to when writing in volumes.
+write_volume_align = 1
+
+[host]
+# Default user and password to connect to a host through XAPI
+# Note: this won't be used for SSH.
+# You need to have an SSH key into the hosts' /root/.ssh/authorized_keys.
+default_user = "root"
+default_password = ""
+
+[hosts]
+# Override settings for specific hosts
+# skip_xo_config allows to not touch XO's configuration regarding the host
+# Else the default behaviour is to add the host to XO servers at the beginning
+# of the testing session and remove it at the end.
+# "10.0.0.1" = { user = "root", password = "" }
+# "testhost1" = { user = "root", password = "", skip_xo_config = true }
+
+[network]
+# Network names and descriptions
+mgmt = "Pool-wide network associated with eth0"
+
+# List of NICs available on host for network tests.
+# example: free_nics = ["eth1", "eth2"]
+free_nics = []
+
+[pxe]
+# PXE configuration server for automated XCP-ng installation
+config_server = "pxe"
+# Server on MGMT network, where ARP tables can reveal the MACs
+arp_server = "pxe"
+
+[vm]
+# Default VM images location
+# Values can be either full URLs or only partial URLs that will be automatically appended to def_url
+def_url = "http://pxe/images/"
+
+# Whether to cache VMs on the test host, that is import them only if not already
+# present in the target SR. This also causes the VM to be cloned at the beginning
+# of each test module, so that the original VM remains untouched.
+# /!\ The VM identifier in cache is simply the URL where it was imported from.
+# No checksum or date is checked.
+# A cached VM is just a VM which has a special description.
+# Example description: "[Cache for http://example.com/images/filename.xva]"
+# Delete the VM to remove it from cache.
+# This setting affects VMs managed by the `imported_vm` fixture.
+cache_imported = false
+
+# In some cases, we may prefer to favour a local SR to store test VM disks,
+# to avoid latency or unstabilities related to network or shared file servers.
+# However it's not good practice to make a local SR the default SR for a pool of several hosts.
+# Hence this configuration value that you can set to `local` so that our tests use this SR by default.
+# This setting affects VMs managed by the `imported_vm` fixture.
+# Possible values:
+# - 'default': keep using the pool's default SR
+# - 'local': use the first local SR found instead
+# - A UUID of the SR to be used
+default_sr = "default"
+
+[vm.images]
+# VM image definitions: name -> filename or URL
+# Values can be either full URLs or only partial URLs appended to vm.def_url
+"mini-linux-x86_64-bios" = "alpine-minimal-3.12.0.xva"
+"mini-linux-x86_64-uefi" = "alpine-uefi-minimal-3.12.0.xva"
+
+[vm.equivalents]
+# Image equivalences for caching/deduplication
+# Maps test image IDs to other IDs they can be substituted with
+# "test_id_1" = "test_id_2"
+
+[install]
+# Installation configuration
+# Path to the iso-remaster utility script
+# iso_remaster = "/home/user/src/xcpng/xcp/scripts/iso-remaster/iso-remaster.sh"
+
+[xo]
+# Path to the xo-cli utility
+# Default value: xo-cli found in PATH
+cli = "xo-cli"
+
+[install.answerfiles]
+# Base answer files for installation
+# Password hash is computed from host.default_password at config load time
+# When used, <PASSWORD_HASH> placeholder will be replaced with the actual hash
+INSTALL = { TAG = "installation", CONTENTS = [
+    { TAG = "root-password", type = "hash", CONTENTS = "<PASSWORD_HASH>" },
+    { TAG = "timezone", CONTENTS = "Europe/Paris" },
+    { TAG = "keymap", CONTENTS = "us" }
+] }
+UPGRADE = { TAG = "installation", mode = "upgrade" }
+RESTORE = { TAG = "restore" }
+
+[install.isos]
+# Base URL for XCP-ng installer ISOs
+base_url = "https://updates.xcp-ng.org/isos/"
+# Local cache directory for downloaded ISOs
+cache_dir = "/home/user/iso"
+
+[install.isos.definitions]
+# XCP-ng installer ISO definitions
+# path can be:
+# - absolute filename
+# - absolute URL
+# - path relative to base_url
+# Note the dirname part is ignored when looking in cache_dir, abuse this
+# for local-only ISO with things like "locally-built/my.iso" or "xs/8.3.iso".
+# If 'net-only' is set to 'True' only source of type URL will be possible.
+# By default the parameter is set to False.
+"83nightly" = { path = "http://unconfigured.iso", unsigned = true }
+"830" = { path = "8.3/xcp-ng-8.3.0.iso" }
+"82nightly" = { path = "http://unconfigured.iso", unsigned = true }
+"821.1" = { path = "8.2/xcp-ng-8.2.1-20231130.iso" }
+"821" = { path = "8.2/xcp-ng-8.2.1.iso" }
+"820" = { path = "8.2/xcp-ng-8.2.0.iso" }
+"81" = { path = "8.1/xcp-ng-8.1.0-2.iso" }
+"80" = { path = "8.0/xcp-ng-8.0.0.iso" }
+"76" = { path = "7.6/xcp-ng-7.6.0.iso" }
+"75" = { path = "7.5/xcp-ng-7.5.0-2.iso" }
+"xs8" = { path = "XenServer8_2024-03-18.iso" }
+"ch821.1" = { path = "CitrixHypervisor-8.2.1-2306-install-cd.iso" }
+"ch821" = { path = "CitrixHypervisor-8.2.1-install-cd.iso" }
+
+[guest_tools]
+# Guest tools ISO download location
+download_url = "http://pxe/isos/"
+
+[guest_tools.win]
+# Definitions of Windows guest tool ISOs to be tested
+
+[guest_tools.win.stable]
+# ISO name on SR or subpath of download_url
+name = "guest-tools-win.iso"
+# Whether ISO should be downloaded from download_url
+download = true
+# ISO-relative path of MSI file to be installed
+package = "package\\XenDrivers-x64.msi"
+# ISO-relative path of XenClean script
+xenclean_path = "package\\XenClean\\x64\\Invoke-XenClean.ps1"
+# ISO-relative path of root cert file to be installed before guest tools (optional)
+testsign_cert = "testsign\\XCP-ng_Test_Signer.crt"
+# What's the onboard family of our tools? This is equal to the WinPV VENDOR_NAME value
+onboard_family = "XCP-ng"
+
+[guest_tools.other]
+# Definition of ISO containing other guest tools to be tested
+# ISO name on SR or subpath of download_url
+name = "other-guest-tools-win.iso"
+# Whether ISO should be downloaded from download_url
+download = false
+
+[guest_tools.installed]
+# Definitions of other guest tools contained in guest_tools.other ISO
+
+[guest_tools.installed."xcp-ng-9.0.9000"]
+# Whether we are installing MSI files ("msi"), bare .inf drivers ("inf")
+# or nothing in case of Windows Update (absent or null)
+type = "msi"
+# ISO-relative path of this guest tool
+path = "xcp-ng-9.0.9000"
+# "path"-relative path of MSI or driver files to be installed
+package = "package\\XenDrivers-x64.msi"
+# Relative path of root cert file (optional)
+testsign_cert = "testsign\\XCP-ng_Test_Signer.crt"
+# Whether this guest tool version wants vendor device to be activated (optional, defaults to False)
+# Note: other guest tools may not install correctly with this setting enabled
+vendor_device = false
+# Can we upgrade automatically from this guest tool to our tools?
+upgradable = true
+# What is the expected onboarding phase after running XenClean when this tool is installed? (optional)
+# See test_xenclean.py ONBOARDING_PHASES for details
+onboarding_phase = "see test_xenclean.py ONBOARDING_PHASES"
+
+[guest_tools.installed.vendor]
+# Vendor device-specific guest tools
+vendor_device = true
+upgradable = false
+
+[ssh]
+# Public keys for a private key available to the test runner
+pubkey = """ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMnN/wVdQqHA8KsndfrLS7fktH/IEgxoa533efuXR6rw XCP-ng CI
+ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKz9uQOoxq6Q0SQ0XTzQHhDolvuo/7EyrDZsYQbRELhcPJG8MT/o5u3HyJFhIP2+HqBSXXgmqRPJUkwz9wUwb2sUwf44qZm/pyPUWOoxyVtrDXzokU/uiaNKUMhbnfaXMz6Ogovtjua63qld2+ZRXnIgrVtYKtYBeu/qKGVSnf4FTOUKl1w3uKkr59IUwwAO8ay3wVnxXIHI/iJgq6JBgQNHbn3C/SpYU++nqL9G7dMyqGD36QPFuqH/cayL8TjNZ67TgAzsPX8OvmRSqjrv3KFbeSlpS/R4enHkSemhgfc8Z2f49tE7qxWZ6x4Uyp5E6ur37FsRf/tEtKIUJGMRXN XCP-ng CI"""
+# Maximum number of SSH output lines to log before truncating
+output_max_lines = 20
+# If true, strips SSH banners from output
+ignore_banner = false
+
+[storage]
+# Storage device configurations (all optional, leave empty if not used)
+
+[storage.nfs]
+# Default NFS device config:
+# URL/Hostname of NFS server
+# server = "10.0.0.2"
+# Path to shared mountpoint
+# serverpath = "/path/to/shared/mount"
+
+[storage.nfs4]
+# Default NFS4+ only device config:
+# URL/Hostname of NFS server
+# server = "10.0.0.2"
+# Path to shared mountpoint
+# serverpath = "/path_to_shared_mount"
+# nfsversion = "4.1"
+
+[storage.nfs_iso]
+# Default NFS ISO device config:
+# URL/Hostname of NFS server and path to shared mountpoint
+# location = "10.0.0.2:/path/to/shared/mount"
+
+[storage.cifs_iso]
+# Default CIFS ISO device config:
+# location = r"\\10.0.0.2\<shared folder name>"
+# username = "<user>"
+# cifspassword = "<password>"
+# type = "cifs"
+# vers = "<1.0> or <3.0>"
+
+[storage.cephfs]
+# CephFS storage configuration
+# server = "10.0.0.2"
+# serverpath = "/vms"
+
+[storage.moosefs]
+# MooseFS storage configuration
+# masterhost = "mfsmaster"
+# masterport = "9421"
+# rootpath = "/vms"
+
+[storage.lvmohba]
+# LVM over HBA storage configuration
+# SCSIid = "id"
+
+[storage.lvmoiscsi]
+# LVM over iSCSI storage configuration
+# target = "192.168.1.1"
+# port = "3260"
+# targetIQN = "target.example"
+# SCSIid = "id"
+
+[storage.linstor]
+# LINSTOR storage configuration
+redundancy = 2