WIP Move to CentOS Stream 10 directly by lyarwood · Pull Request #2 · kubevirt/kubevirt-aie

lyarwood · 2026-02-12T11:59:21Z

What this PR does

Before this PR:

After this PR:

References

Why we need it and why it was done in this way

The following tradeoffs were made:

The following alternatives were considered:

Links to places where the discussion took place:

Special notes for your reviewer

Checklist

This checklist is not enforcing, but it's a reminder of items that could be relevant to every PR.
Approvers are expected to review this list.

Design: A design document was considered and is present (link) or not required
PR: The PR description is expressive enough and will help future contributors
Code: Write code that humans can understand and Keep it simple
Refactor: You have left the code cleaner than you found it (Boy Scout Rule)
Upgrade: Impact of this change on upgrade flows was considered and addressed if required
Testing: New code requires new unit tests. New features and bug fixes require at least one e2e test
Documentation: A user-guide update was considered and is present (link) or not required. You want a user-guide update if it's a user facing feature / API change.
Community: Announcement to kubevirt-dev was considered

Release note

NONE

Replace CentOS Stream 9 repository URLs with CentOS Stream 10 equivalents across all architectures (x86_64, aarch64, s390x). The repository structure remains the same (BaseOS, AppStream, CRB). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Lee Yarwood <lyarwood@redhat.com>

Update the builder Dockerfile to use CentOS Stream 10: - Base image: quay.io/centos/centos:stream10 - Java: java-21-openjdk-devel (CS10 only has Java 21+) - Python symlink: use -sf flag (CS10 already has /usr/bin/python) - JAVA_HOME: java-21 Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Lee Yarwood <lyarwood@redhat.com>

Update all cross-compilation toolchain configurations from CentOS Stream 9 to CentOS Stream 10: - x86_64: GCC 11 -> GCC 14, sysroot centos-stream-9 -> centos-stream-10 - aarch64: GCC 12 -> GCC 14, sysroot centos-stream-9 -> centos-stream-10 - s390x: GCC 12 -> GCC 14, sysroot centos-stream-9 -> centos-stream-10 Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Lee Yarwood <lyarwood@redhat.com>

Update all RPM dependency infrastructure for CentOS Stream 10: - hack/rpm-deps.sh: Use unversioned package defaults for CS10, replace curl-minimal with curl, make sevctl x86_64-only (not available on other architectures in CS10), skip bootstrap during regeneration to avoid circular dependency issues - rpm/BUILD.bazel: Regenerated with CS10 RPM packages (.el10) - WORKSPACE: Regenerated with CS10 RPM download entries All target names remain unsuffixed (e.g., testimage_x86_64, launcherbase_x86_64) matching the upstream convention. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Lee Yarwood <lyarwood@redhat.com>

In CentOS Stream 10, /var/run is no longer a symlink to ../run by default. This causes issues with libvirt socket paths since libvirt creates sockets in /run/libvirt/ but KubeVirt expects them at /var/run/libvirt/. Add a pkg_tar that creates the /var/run -> ../run symlink in the virt-launcher container image. The symlink must be relative (../run) rather than absolute (/run) because virt-handler accesses the container filesystem via /proc/<pid>/root/var/run - an absolute symlink would resolve /run relative to virt-handler's root namespace. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Lee Yarwood <lyarwood@redhat.com>

Update the default builder image name from "builder" to "builder-aie-nv" to distinguish the CentOS Stream 10 based builder from upstream. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Lee Yarwood <lyarwood@redhat.com>

kubevirt-bot · 2026-02-12T12:00:11Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: lyarwood
Once this PR has been reviewed and has the lgtm label, please assign iholder101 for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

The test is currently flaking. It uses ConsitsOf instead of Contains, to analyze the status which is currently showing 2 interfaces instead of one. The reason why the status includes two interfaces instead of one is as follows: The virt-handler's logic responsible for reporting interface status of SR-IOV devices is comprised of two steps: 1. Info is collected from the domain spec, while the MAC address is taken from the vmi.spec.domain.interfaces[].mac field (if exists) [1]. 2. Info from guest agent enriches the existing status entries in case the MAC address reported by GA can be correlated with the MAC address reported in the status [2]. Since the MAC address is unspecified in the spec, it is also unspecified in the status and is therefore not correlated. Since GA reporting is unrecognized, it is considered a guest only interface and is reported as an additional interface. This is a known issue kubevirt/kubevirt#16422. In order to fix correlation, set explicit MAC address in the spec in order to mitigate #1 above, and enable correlation in #2. The failure below [3] demonstrates how 2 interfaces were reported and how ConsitsOf was not satified with the results despite including the all the data that the gstruct matcher was asserting. [1] https://github.com/kubevirt/kubevirt/blob/395bd0da05f8b8d39326eb1b90da90b574a16e2b/pkg/network/setup/netstat.go#L308 [2] https://github.com/kubevirt/kubevirt/blob/395bd0da05f8b8d39326eb1b90da90b574a16e2b/pkg/network/setup/netstat.go#L333 [3] https://prow.ci.kubevirt.io/view/gs/kubevirt-prow/pr-logs/pull/kubevirt_kubevirt/16391/pull-kubevirt-e2e-kind-sriov/2004191256049094656 Expected <[]v1.VirtualMachineInstanceNetworkInterface | len:2, cap:2>: [ { IP: "", MAC: "", Name: "sriov-network", IPs: nil, PodInterfaceName: "podafa4cdb1f64", InterfaceName: "", InfoSource: "domain, multus-status", QueueCount: 0, LinkState: "", }, { IP: "", MAC: "86:4b:dc:31:e5:c8", Name: "", IPs: nil, PodInterfaceName: "", InterfaceName: "eth0", InfoSource: "guest-agent", QueueCount: 0, LinkState: "", }, ] to consist of <[]*gstruct.FieldsMatcher | len:1, cap:1>: [ { Fields: { "Name": <*matchers.EqualMatcher | 0xc0073e61a0>{ Expected: <string>"sriov-network", }, "InfoSource": <*matchers.ContainSubstringMatcher | 0xc004b126c0>{Substr: "domain", Args: nil}, }, IgnoreExtras: true, IgnoreMissing: false, failures: [ <*errors.NestedError | 0xc007a8d080>{ Path: ".Name", Err: <*errors.errorString | 0xc007108820>{ s: "Expected\n <string>: \nto equal\n <string>: sriov-network", }, }, <*errors.NestedError | 0xc007a8d0a0>{ Path: ".InfoSource", Err: <*errors.errorString | 0xc007108920>{ s: "Expected\n <string>: guest-agent\nto contain substring\n <string>: domain", }, }, ], }, ] the extra elements were <[]v1.VirtualMachineInstanceNetworkInterface | len:1, cap:1>: [ { IP: "", MAC: "86:4b:dc:31:e5:c8", Name: "", IPs: nil, PodInterfaceName: "", InterfaceName: "eth0", InfoSource: "guest-agent", QueueCount: 0, LinkState: "", Signed-off-by: Nir Dothan <ndothan@redhat.com>

On NVIDIA GB200 Grace Blackwell systems, GPUs are on separate NUMA nodes from all CPUs by design. A 2-superchip GB200 exposes: NUMA 0: Grace CPU kubevirt#1 (CPUs 0-71, ~490 GB LPDDR5X) NUMA 1: Grace CPU kubevirt#2 (CPUs 72-143, ~490 GB LPDDR5X) NUMA 2: Blackwell GPU kubevirt#1 (no CPUs, ~188 GB HBM) NUMA 10: Blackwell GPU kubevirt#2 (no CPUs, ~188 GB HBM) The existing PCIe NUMA-aware topology code discovers the host NUMA node for each passthrough device but discards it when no vCPUs are pinned to that node, causing the device to fall back to the default pci.0 bus without NUMA affinity information. Extend LookupDevicesNumaNodes to return both aligned devices (host NUMA node has pinned vCPUs) and unaligned devices (host NUMA node has no pinned vCPUs but is a valid NUMA node). When the expander bus assigner encounters an unaligned device, it creates a CPU-less guest NUMA cell for the device's host NUMA node and places the device under a pcie-expander-bus targeting that cell. On a GB200 each GPU naturally gets its own expander bus and SMMUv3 IOMMU device through the standard per-NUMA topology path, since each GPU is on its own NUMA node. Also adds an explicit check for NUMA node -1 in GetDeviceNumaNode to avoid relying on uint32 overflow for devices without NUMA affinity. Ref: https://docs.nvidia.com/dccpu/grace-perf-tuning-guide/system.html Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Lee Yarwood <lyarwood@redhat.com>

…vices On a 2-superchip GB200, each Grace CPU has 2 Blackwell GPUs connected via NVLink-C2C that share the same sysfs NUMA node as the CPU socket: GPU 1 (0008:01:00.0): NUMA 0 (Grace CPU kubevirt#1) GPU 2 (0009:01:00.0): NUMA 0 (Grace CPU kubevirt#1) GPU 3 (0018:01:00.0): NUMA 1 (Grace CPU kubevirt#2) GPU 4 (0019:01:00.0): NUMA 1 (Grace CPU kubevirt#2) VEP 115's per-NUMA expander bus placement puts both GPUs on the same pcie-expander-bus, sharing a single SMMUv3 IOMMU device. This causes CMD_SYNC timeout and tegra241_cmdqv errors in the guest, preventing multi-GPU initialization. When multiple IOMMU devices (marked with ACPI NodeSet "tofill") share a NUMA node, each now gets its own pcie-expander-bus controller and smmuv3 IOMMU device. A single IOMMU device on a NUMA node continues to use the shared per-NUMA expander bus. Non-IOMMU devices always share a pcie-expander-bus per NUMA node. Ref: VOYAGER-707 Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Lee Yarwood <lyarwood@redhat.com>

…g a NUMA node On a 2-superchip GB200, each Grace CPU has 2 Blackwell GPUs connected via NVLink-C2C that share the same sysfs NUMA node as the CPU socket: GPU 1 (0008:01:00.0): NUMA 0 (Grace CPU kubevirt#1) GPU 2 (0009:01:00.0): NUMA 0 (Grace CPU kubevirt#1) GPU 3 (0018:01:00.0): NUMA 1 (Grace CPU kubevirt#2) GPU 4 (0019:01:00.0): NUMA 1 (Grace CPU kubevirt#2) VEP 115's per-NUMA expander bus placement puts both GPUs on the same pcie-expander-bus, which means they share a single IOMMU device created by placeDevice. On ARM64 Grace systems this shared SMMUv3 causes CMD_SYNC timeout and tegra241_cmdqv errors in the guest, preventing multi-GPU initialization. When multiple IOMMU devices (marked with ACPI NodeSet "tofill") share a NUMA node, each now gets its own pcie-expander-bus controller. A single IOMMU device on a NUMA node continues to use the shared per-NUMA expander bus. Non-IOMMU devices always share a pcie-expander-bus per NUMA node. Ref: VOYAGER-707 Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Lee Yarwood <lyarwood@redhat.com>

lyarwood added 6 commits February 12, 2026 11:14

hack/builder: Rename builder image to builder-aie-nv

52626ee

Update the default builder image name from "builder" to "builder-aie-nv" to distinguish the CentOS Stream 10 based builder from upstream. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Lee Yarwood <lyarwood@redhat.com>

kubevirt-bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note-none Denotes a PR that doesn't merit a release note. dco-signoff: yes Indicates the PR's author has DCO signed all their commits. size/XXL labels Feb 12, 2026

kubevirt-bot requested a review from Barakmor1 February 12, 2026 12:00

kubevirt-bot added the sig/buildsystem Denotes an issue or PR that relates to changes in the build system. label Feb 12, 2026

kubevirt-bot requested review from brianmcarey, enp0s3 and xpivarc February 12, 2026 12:00

kubevirt-bot added the sig/compute label Feb 12, 2026

lyarwood closed this Feb 12, 2026

lyarwood mentioned this pull request May 5, 2026

PCIe NUMA topology: support cross-NUMA device placement #36

Closed

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

WIP Move to CentOS Stream 10 directly#2

WIP Move to CentOS Stream 10 directly#2
lyarwood wants to merge 6 commits into
kubevirt:release-1.8-aie-nvfrom
lyarwood:centos-stream-10-direct

lyarwood commented Feb 12, 2026

Uh oh!

kubevirt-bot commented Feb 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

lyarwood commented Feb 12, 2026

What this PR does

Before this PR:

After this PR:

References

Why we need it and why it was done in this way

Special notes for your reviewer

Checklist

Release note

Uh oh!

kubevirt-bot commented Feb 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants