Skip to content

WIP/DNM - Add CentOS Stream 10 build support#1

Closed
lyarwood wants to merge 7 commits into
kubevirt:release-1.8-aie-nvfrom
lyarwood:centos-stream-10
Closed

WIP/DNM - Add CentOS Stream 10 build support#1
lyarwood wants to merge 7 commits into
kubevirt:release-1.8-aie-nvfrom
lyarwood:centos-stream-10

Conversation

@lyarwood

@lyarwood lyarwood commented Feb 12, 2026

Copy link
Copy Markdown
Member

What this PR does

Before this PR:

KubeVirt can only be built using CentOS Stream 9 base images and RPM dependencies.

After this PR:

KubeVirt can be built with either CentOS Stream 9 (default) or CentOS Stream 10 using the KUBEVIRT_CENTOS_STREAM_VERSION environment variable.

Key changes:

  • RPM infrastructure: Dual CS9/CS10 build support with version-specific repo configs and Bazel macros
  • Builder images: New CS10 builder Dockerfile and build scripts (uses GCC 14, Java 21)
  • Bazel toolchain: Separate GCC 14 toolchain configuration for CS10
  • virt-launcher: Fix /var/run symlink handling for CS10 compatibility
  • RPM dependencies: Generated CS10 dependencies for x86_64, aarch64, and s390x

References

Why we need it and why it was done in this way

The following tradeoffs were made:

  • Separate builder images are required because CS10's glibc uses the .relr.dyn ELF section format which requires newer binutils than available in CS9
  • A separate Bazel C/C++ toolchain is needed because GCC 14 in CS10 has different include paths than GCC 11 in CS9

The following alternatives were considered:

  • Upgrading the CS9 builder to use newer binutils - rejected because it would diverge from the base OS toolchain

Links to places where the discussion took place:

Special notes for your reviewer

Example workflow to build and test with CS10:

$ env | grep KUBEVIRT
KUBEVIRT_PROVIDER=kind-1.34
KUBEVIRT_CENTOS_STREAM_VERSION=10
KUBEVIRT_MEMORY_SIZE=16384
$ ARCHITECTURES=amd64 make builder-build-cs10
[..]
2601281350-7f0e1d9b01
$ export KUBEVIRT_CS10_BUILDER_VERSION=2601281350-7f0e1d9b01-amd64
$ make cluster-up && make cluster-sync
[..]
$ virtctl create vm --instancetype u1.medium --preference fedora --volume-containerdisk src:quay.io/containerdisks/fedora:latest,name:fedora | kubectl apply -f -
[..]
$ export KUBECONFIG=$(./kubevirtci/cluster-up/kubeconfig.sh)
$ kubectl get pods
NAME                           READY   STATUS    RESTARTS   AGE
virt-launcher-vm-v9cnd-7gvdv   3/3     Running   0          5m55s
$ kubectl exec -ti virt-launcher-vm-v9cnd-7gvdv -- /bin/sh
sh-5.2$ cat /etc/redhat-release
CentOS Stream release 10 (Coughlan)
sh-5.2$ /usr/libexec/qemu-kvm --version
QEMU emulator version 10.1.0 (qemu-kvm-10.1.0-9.el10)
Copyright (c) 2003-2025 Fabrice Bellard and the QEMU Project developers

Checklist

This checklist is not enforcing, but it's a reminder of items that could be relevant to every PR.
Approvers are expected to review this list.

Release note

Add CentOS Stream 10 build support with KUBEVIRT_CENTOS_STREAM_VERSION environment variable

@lyarwood
rpm: Add dual CentOS Stream 9/10 build support infrastructure
e7804d9 
Introduce KUBEVIRT_CENTOS_STREAM_VERSION environment variable to control
which CentOS Stream version is used for container image RPMs. The default
is 9 (current version).

Key changes:
- Create rpm/repo-cs9.yaml and rpm/repo-cs10.yaml with version-specific
  CentOS mirror URLs
- Create rpm/centos_stream.bzl with Bazel macros for version selection
  using select() and config_setting
- Update hack/rpm-deps.sh to generate versioned targets with _cs9/_cs10
  suffix based on KUBEVIRT_CENTOS_STREAM_VERSION
- Add hack/rpm-deps-aliases.sh to generate version-selecting aliases
- Add BAZEL_CS_CONFIG to common.sh and all build scripts for automatic
  config selection based on the environment variable
- Add Makefile targets: rpm-deps-cs9, rpm-deps-cs10, rpm-deps-all
- Update .bazelrc with cs9/cs10 config settings

Note: WORKSPACE and rpm/BUILD.bazel changes will be regenerated.

Usage:
  # Build with CS9 (default)
  make bazel-build

  # Build with CS10
  KUBEVIRT_CENTOS_STREAM_VERSION=10 make bazel-build

  # Regenerate RPM dependencies
  make rpm-deps-cs9
  make rpm-deps-cs10
  make rpm-deps-all

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Lee Yarwood <lyarwood@redhat.com>
@lyarwood
rpm: Add version-selecting aliases and fix libvirt-libs paths
10a6101 
- Add path-based centos_stream_alias targets for libvirt-libs tar2files
  outputs (e.g., libvirt-libs_x86_64/usr/lib64)
- Add compound config_settings for platform + CentOS Stream version
  combinations (e.g., x86_64_cs9, x86_64_cs10)
- Update BUILD.bazel cc_library strip_include_prefix to be version-aware
- Remove invalid top-level libvirt-libs aliases (tar2files creates
  path-based targets, not single targets)
- Update sandbox_hash after CS10 sandbox regeneration

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Lee Yarwood <lyarwood@redhat.com>
@lyarwood
hack/builder: Add CentOS Stream 10 builder support
318cbd5 
Add infrastructure for building CentOS Stream 10 based builder images:

- Dockerfile.cs10: CS10 variant of the builder Dockerfile using
  quay.io/centos/centos:stream10 as base image
  - Uses Java 21 instead of Java 11 (CS10 only has Java 21+)
  - Uses -sf for python symlink (CS10 already has /usr/bin/python)
- build-cs10.sh: Build script for CS10 builder images
- common.sh: Add DOCKER_CS10_IMAGE and DOCKER_CS10_CROSS_IMAGE variables
- dockerized: Select CS10 builder when KUBEVIRT_CENTOS_STREAM_VERSION=10
  - Allows KUBEVIRT_BUILDER_IMAGE to override the version check
- Makefile: Add builder-build-cs10 target

Usage:
1. Build CS10 builder: make builder-build-cs10
2. Use local image: KUBEVIRT_BUILDER_IMAGE=quay.io/kubevirt/builder-cs10:<version>-amd64
3. Build with CS10: KUBEVIRT_CENTOS_STREAM_VERSION=10 make bazel-build

Or with published image:
1. Set version: export KUBEVIRT_CS10_BUILDER_VERSION=<version>
2. Build: KUBEVIRT_CENTOS_STREAM_VERSION=10 make bazel-build

The CS10 builder is required for building KubeVirt with CS10 RPMs
because CS10's glibc uses the .relr.dyn ELF section format which
requires a newer binutils than what's available in the CS9 builder.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Lee Yarwood <lyarwood@redhat.com>
@lyarwood
rpm: Generate CentOS Stream 10 RPM dependencies
73c28c9 
Add CentOS Stream 10 RPM dependencies for all architectures:
- x86_64
- aarch64
- s390x

CentOS Stream 9 packages remain at their original versions from main.

Key changes:
- Generate rpmtree and tar2files targets for CS10
- Add path-based aliases for libvirt-libs tar2files outputs
- Add libvirt-libs to the aliases list in hack/rpm-deps-aliases.sh
- Update sandbox_hash for CS10

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Lee Yarwood <lyarwood@redhat.com>
@lyarwood
virt-launcher: Fix /var/run symlink for CentOS Stream 10 compatibility
9262c14 
In CentOS Stream 10, libvirt creates sockets in /run/libvirt/ but
KubeVirt expects them at /var/run/libvirt/. Unlike CS9, the CS10
filesystem package creates /var/run as a directory instead of a
symlink to /run.

The BUILD.bazel changes:
- Rename virt-launcher binary to virt-launcher-bin
- Add entrypoint.sh wrapper script
- Add var-run-symlink-tar for static symlink (backup)
- Change container entrypoint to use the wrapper script

Signed-off-by: Lee Yarwood <lyarwood@redhat.com>
Assisted-By: Claude <noreply@anthropic.com>
@lyarwood
bazel/toolchain: Add CentOS Stream 10 GCC 14 toolchain
7008534 
Add a separate C/C++ toolchain configuration for CentOS Stream 10 which
uses GCC 14 (paths under /usr/lib/gcc/x86_64-redhat-linux/14/).

Key changes:
- Create cc_toolchain_config_cs10.bzl with GCC 14 include paths
- Register x86_64_linux_toolchain_cs10 with target_settings for is_cs10
- CS10 toolchain registered first for priority during resolution

The toolchain is selected automatically when building with
KUBEVIRT_CENTOS_STREAM_VERSION=10 based on the config_setting.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Lee Yarwood <lyarwood@redhat.com>
@lyarwood
bazel/toolchain: Add ARM64 and s390x CS10 GCC 14 toolchains
ea14659 
Extend CentOS Stream 10 toolchain support to ARM64 and s390x architectures,
complementing the existing x86_64 CS10 toolchain.

Key changes:
- Create cc_toolchain_config_cs10.bzl for aarch64 with GCC 14 paths
- Create cc_toolchain_config_cs10.bzl for s390x with GCC 14 paths
- Register aarch64_linux_toolchain_cs10 and s390x_linux_toolchain_cs10
- CS10 toolchains selected automatically via is_cs10 target_settings

Tested with successful builds of C/C++ and Go binaries for both architectures
using the crossbuild-aarch64 and crossbuild-s390x configs.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Lee Yarwood <lyarwood@redhat.com>
@kubevirt-bot kubevirt-bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. dco-signoff: yes Indicates the PR's author has DCO signed all their commits. size/XXL labels Feb 12, 2026
@kubevirt-bot kubevirt-bot added the sig/buildsystem Denotes an issue or PR that relates to changes in the build system. label Feb 12, 2026
@kubevirt-bot kubevirt-bot requested a review from EdDev February 12, 2026 09:35
@kubevirt-bot

kubevirt-bot commented Feb 12, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: lyarwood
Once this PR has been reviewed and has the lgtm label, please assign vladikr for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@lyarwood lyarwood closed this Feb 12, 2026
rthallisey pushed a commit that referenced this pull request Apr 10, 2026
@nirdothan
e2e, net, sriov: Fix mem hotplug flake
47c1b1b 
The test is currently flaking.
It uses ConsitsOf instead of Contains, to analyze the status which is
currently showing 2 interfaces instead of one.

The reason why the status includes two interfaces instead of one is
as follows:

The virt-handler's logic responsible for reporting interface status of
SR-IOV devices is comprised of two steps:
1. Info is collected from the domain spec, while the MAC address is
   taken from the vmi.spec.domain.interfaces[].mac field
   (if exists) [1].
2. Info from guest agent enriches the existing status entries in case
   the MAC address reported by GA can be correlated with the MAC address
   reported in the status [2].

Since the MAC address is unspecified in the spec, it is also unspecified
in the status and is therefore not correlated.
Since GA reporting is unrecognized, it is considered a guest only
interface and is reported as an additional interface.
This is a known issue kubevirt/kubevirt#16422.

In order to fix correlation, set explicit MAC address in the spec in
order to mitigate #1 above, and enable correlation in #2.

The failure below [3] demonstrates how 2 interfaces were reported and
how ConsitsOf was not satified with the results despite including the
all the data that the gstruct matcher was asserting.

[1] https://github.com/kubevirt/kubevirt/blob/395bd0da05f8b8d39326eb1b90da90b574a16e2b/pkg/network/setup/netstat.go#L308

[2] https://github.com/kubevirt/kubevirt/blob/395bd0da05f8b8d39326eb1b90da90b574a16e2b/pkg/network/setup/netstat.go#L333

[3] https://prow.ci.kubevirt.io/view/gs/kubevirt-prow/pr-logs/pull/kubevirt_kubevirt/16391/pull-kubevirt-e2e-kind-sriov/2004191256049094656

 Expected
     <[]v1.VirtualMachineInstanceNetworkInterface | len:2, cap:2>: [
         {
             IP: "",
             MAC: "",
             Name: "sriov-network",
             IPs: nil,
             PodInterfaceName: "podafa4cdb1f64",
             InterfaceName: "",
             InfoSource: "domain, multus-status",
             QueueCount: 0,
             LinkState: "",
         },
         {
             IP: "",
             MAC: "86:4b:dc:31:e5:c8",
             Name: "",
             IPs: nil,
             PodInterfaceName: "",
             InterfaceName: "eth0",
             InfoSource: "guest-agent",
             QueueCount: 0,
             LinkState: "",
         },
     ]
 to consist of
     <[]*gstruct.FieldsMatcher | len:1, cap:1>: [
         {
             Fields: {
                 "Name": <*matchers.EqualMatcher | 0xc0073e61a0>{
                     Expected: <string>"sriov-network",
                 },
                 "InfoSource": <*matchers.ContainSubstringMatcher |
                             0xc004b126c0>{Substr: "domain", Args: nil},
             },
             IgnoreExtras: true,
             IgnoreMissing: false,
             failures: [
                 <*errors.NestedError | 0xc007a8d080>{
                     Path: ".Name",
                     Err: <*errors.errorString | 0xc007108820>{
                         s: "Expected\n    <string>: \nto equal\n
                                 <string>: sriov-network",
                     },
                 },
                 <*errors.NestedError | 0xc007a8d0a0>{
                     Path: ".InfoSource",
                     Err: <*errors.errorString | 0xc007108920>{
                         s: "Expected\n    <string>: guest-agent\nto
                              contain substring\n    <string>: domain",
                     },
                 },
             ],
         },
     ]
 the extra elements were
     <[]v1.VirtualMachineInstanceNetworkInterface | len:1, cap:1>: [
         {
             IP: "",
             MAC: "86:4b:dc:31:e5:c8",
             Name: "",
             IPs: nil,
             PodInterfaceName: "",
             InterfaceName: "eth0",
             InfoSource: "guest-agent",
             QueueCount: 0,
             LinkState: "",

Signed-off-by: Nir Dothan <ndothan@redhat.com>
lyarwood added a commit to lyarwood/kubevirt-aie that referenced this pull request May 5, 2026
@lyarwood
PCIe NUMA topology: support cross-NUMA device placement
e222fd7 
On NVIDIA GB200 Grace Blackwell systems, GPUs are on separate NUMA
nodes from all CPUs by design. A 2-superchip GB200 exposes:

  NUMA 0:  Grace CPU kubevirt#1 (CPUs 0-71, ~490 GB LPDDR5X)
  NUMA 1:  Grace CPU kubevirt#2 (CPUs 72-143, ~490 GB LPDDR5X)
  NUMA 2:  Blackwell GPU kubevirt#1 (no CPUs, ~188 GB HBM)
  NUMA 10: Blackwell GPU kubevirt#2 (no CPUs, ~188 GB HBM)

The existing PCIe NUMA-aware topology code discovers the host NUMA node
for each passthrough device but discards it when no vCPUs are pinned to
that node, causing the device to fall back to the default pci.0 bus
without NUMA affinity information.

Extend LookupDevicesNumaNodes to return both aligned devices (host NUMA
node has pinned vCPUs) and unaligned devices (host NUMA node has no
pinned vCPUs but is a valid NUMA node). When the expander bus assigner
encounters an unaligned device, it creates a CPU-less guest NUMA cell
for the device's host NUMA node and places the device under a
pcie-expander-bus targeting that cell.

On a GB200 each GPU naturally gets its own expander bus and SMMUv3 IOMMU
device through the standard per-NUMA topology path, since each GPU is
on its own NUMA node.

Also adds an explicit check for NUMA node -1 in GetDeviceNumaNode to
avoid relying on uint32 overflow for devices without NUMA affinity.

Ref: https://docs.nvidia.com/dccpu/grace-perf-tuning-guide/system.html

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Lee Yarwood <lyarwood@redhat.com>
@lyarwood lyarwood mentioned this pull request May 5, 2026
Closed
9 tasks
lyarwood added a commit to lyarwood/kubevirt-aie that referenced this pull request May 5, 2026
@lyarwood
pci-placement: Create per-device expander bus and smmuv3 for IOMMU de…
8147a3b 
…vices

On a 2-superchip GB200, each Grace CPU has 2 Blackwell GPUs connected
via NVLink-C2C that share the same sysfs NUMA node as the CPU socket:

  GPU 1 (0008:01:00.0): NUMA 0 (Grace CPU kubevirt#1)
  GPU 2 (0009:01:00.0): NUMA 0 (Grace CPU kubevirt#1)
  GPU 3 (0018:01:00.0): NUMA 1 (Grace CPU kubevirt#2)
  GPU 4 (0019:01:00.0): NUMA 1 (Grace CPU kubevirt#2)

VEP 115's per-NUMA expander bus placement puts both GPUs on the same
pcie-expander-bus, sharing a single SMMUv3 IOMMU device. This causes
CMD_SYNC timeout and tegra241_cmdqv errors in the guest, preventing
multi-GPU initialization.

When multiple IOMMU devices (marked with ACPI NodeSet "tofill") share
a NUMA node, each now gets its own pcie-expander-bus controller and
smmuv3 IOMMU device. A single IOMMU device on a NUMA node continues
to use the shared per-NUMA expander bus. Non-IOMMU devices always
share a pcie-expander-bus per NUMA node.

Ref: VOYAGER-707

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Lee Yarwood <lyarwood@redhat.com>
lyarwood added a commit to lyarwood/kubevirt-aie that referenced this pull request May 5, 2026
@lyarwood
pci-placement: Create per-device expander bus and smmuv3 for IOMMU de…
f5b1fc3 
…vices

On a 2-superchip GB200, each Grace CPU has 2 Blackwell GPUs connected
via NVLink-C2C that share the same sysfs NUMA node as the CPU socket:

  GPU 1 (0008:01:00.0): NUMA 0 (Grace CPU kubevirt#1)
  GPU 2 (0009:01:00.0): NUMA 0 (Grace CPU kubevirt#1)
  GPU 3 (0018:01:00.0): NUMA 1 (Grace CPU kubevirt#2)
  GPU 4 (0019:01:00.0): NUMA 1 (Grace CPU kubevirt#2)

VEP 115's per-NUMA expander bus placement puts both GPUs on the same
pcie-expander-bus, sharing a single SMMUv3 IOMMU device. This causes
CMD_SYNC timeout and tegra241_cmdqv errors in the guest, preventing
multi-GPU initialization.

When multiple IOMMU devices (marked with ACPI NodeSet "tofill") share
a NUMA node, each now gets its own pcie-expander-bus controller and
smmuv3 IOMMU device. A single IOMMU device on a NUMA node continues
to use the shared per-NUMA expander bus. Non-IOMMU devices always
share a pcie-expander-bus per NUMA node.

Ref: VOYAGER-707

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Lee Yarwood <lyarwood@redhat.com>
lyarwood added a commit to lyarwood/kubevirt-aie that referenced this pull request May 5, 2026
@lyarwood
pci-placement: Create per-device expander bus and smmuv3 for IOMMU de…
776f0c7 
…vices

On a 2-superchip GB200, each Grace CPU has 2 Blackwell GPUs connected
via NVLink-C2C that share the same sysfs NUMA node as the CPU socket:

  GPU 1 (0008:01:00.0): NUMA 0 (Grace CPU kubevirt#1)
  GPU 2 (0009:01:00.0): NUMA 0 (Grace CPU kubevirt#1)
  GPU 3 (0018:01:00.0): NUMA 1 (Grace CPU kubevirt#2)
  GPU 4 (0019:01:00.0): NUMA 1 (Grace CPU kubevirt#2)

VEP 115's per-NUMA expander bus placement puts both GPUs on the same
pcie-expander-bus, sharing a single SMMUv3 IOMMU device. This causes
CMD_SYNC timeout and tegra241_cmdqv errors in the guest, preventing
multi-GPU initialization.

When multiple IOMMU devices (marked with ACPI NodeSet "tofill") share
a NUMA node, each now gets its own pcie-expander-bus controller and
smmuv3 IOMMU device. A single IOMMU device on a NUMA node continues
to use the shared per-NUMA expander bus. Non-IOMMU devices always
share a pcie-expander-bus per NUMA node.

Ref: VOYAGER-707

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Lee Yarwood <lyarwood@redhat.com>
lyarwood added a commit to lyarwood/kubevirt-aie that referenced this pull request May 5, 2026
@lyarwood
pci-placement: Create per-device expander bus and smmuv3 for IOMMU de…
a50a762 
…vices

On a 2-superchip GB200, each Grace CPU has 2 Blackwell GPUs connected
via NVLink-C2C that share the same sysfs NUMA node as the CPU socket:

  GPU 1 (0008:01:00.0): NUMA 0 (Grace CPU kubevirt#1)
  GPU 2 (0009:01:00.0): NUMA 0 (Grace CPU kubevirt#1)
  GPU 3 (0018:01:00.0): NUMA 1 (Grace CPU kubevirt#2)
  GPU 4 (0019:01:00.0): NUMA 1 (Grace CPU kubevirt#2)

VEP 115's per-NUMA expander bus placement puts both GPUs on the same
pcie-expander-bus, sharing a single SMMUv3 IOMMU device. This causes
CMD_SYNC timeout and tegra241_cmdqv errors in the guest, preventing
multi-GPU initialization.

When multiple IOMMU devices (marked with ACPI NodeSet "tofill") share
a NUMA node, each now gets its own pcie-expander-bus controller and
smmuv3 IOMMU device. A single IOMMU device on a NUMA node continues
to use the shared per-NUMA expander bus. Non-IOMMU devices always
share a pcie-expander-bus per NUMA node.

Ref: VOYAGER-707

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Lee Yarwood <lyarwood@redhat.com>
lyarwood added a commit to lyarwood/kubevirt-aie that referenced this pull request May 5, 2026
@lyarwood
pci-placement: Isolate pcie-expander-bus per IOMMU device when sharin…
cb22f80 
…g a NUMA node

On a 2-superchip GB200, each Grace CPU has 2 Blackwell GPUs connected
via NVLink-C2C that share the same sysfs NUMA node as the CPU socket:

  GPU 1 (0008:01:00.0): NUMA 0 (Grace CPU kubevirt#1)
  GPU 2 (0009:01:00.0): NUMA 0 (Grace CPU kubevirt#1)
  GPU 3 (0018:01:00.0): NUMA 1 (Grace CPU kubevirt#2)
  GPU 4 (0019:01:00.0): NUMA 1 (Grace CPU kubevirt#2)

VEP 115's per-NUMA expander bus placement puts both GPUs on the same
pcie-expander-bus, which means they share a single IOMMU device
created by placeDevice. On ARM64 Grace systems this shared SMMUv3
causes CMD_SYNC timeout and tegra241_cmdqv errors in the guest,
preventing multi-GPU initialization.

When multiple IOMMU devices (marked with ACPI NodeSet "tofill") share
a NUMA node, each now gets its own pcie-expander-bus controller. A
single IOMMU device on a NUMA node continues to use the shared
per-NUMA expander bus. Non-IOMMU devices always share a
pcie-expander-bus per NUMA node.

Ref: VOYAGER-707

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Lee Yarwood <lyarwood@redhat.com>
lyarwood added a commit to lyarwood/kubevirt-aie that referenced this pull request May 5, 2026
@lyarwood
pci-placement: Isolate pcie-expander-bus per IOMMU device when sharin…
dfeeba9 
…g a NUMA node

On a 2-superchip GB200, each Grace CPU has 2 Blackwell GPUs connected
via NVLink-C2C that share the same sysfs NUMA node as the CPU socket:

  GPU 1 (0008:01:00.0): NUMA 0 (Grace CPU kubevirt#1)
  GPU 2 (0009:01:00.0): NUMA 0 (Grace CPU kubevirt#1)
  GPU 3 (0018:01:00.0): NUMA 1 (Grace CPU kubevirt#2)
  GPU 4 (0019:01:00.0): NUMA 1 (Grace CPU kubevirt#2)

VEP 115's per-NUMA expander bus placement puts both GPUs on the same
pcie-expander-bus, which means they share a single IOMMU device
created by placeDevice. On ARM64 Grace systems this shared SMMUv3
causes CMD_SYNC timeout and tegra241_cmdqv errors in the guest,
preventing multi-GPU initialization.

When multiple IOMMU devices (marked with ACPI NodeSet "tofill") share
a NUMA node, each now gets its own pcie-expander-bus controller. A
single IOMMU device on a NUMA node continues to use the shared
per-NUMA expander bus. Non-IOMMU devices always share a
pcie-expander-bus per NUMA node.

Ref: VOYAGER-707

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Lee Yarwood <lyarwood@redhat.com>
lyarwood added a commit to lyarwood/kubevirt-aie that referenced this pull request May 6, 2026
@lyarwood
pci-placement: Isolate pcie-expander-bus per IOMMU device when sharin…
cfb9432 
…g a NUMA node

On a 2-superchip GB200, each Grace CPU has 2 Blackwell GPUs connected
via NVLink-C2C that share the same sysfs NUMA node as the CPU socket:

  GPU 1 (0008:01:00.0): NUMA 0 (Grace CPU kubevirt#1)
  GPU 2 (0009:01:00.0): NUMA 0 (Grace CPU kubevirt#1)
  GPU 3 (0018:01:00.0): NUMA 1 (Grace CPU kubevirt#2)
  GPU 4 (0019:01:00.0): NUMA 1 (Grace CPU kubevirt#2)

VEP 115's per-NUMA expander bus placement puts both GPUs on the same
pcie-expander-bus, which means they share a single IOMMU device
created by placeDevice. On ARM64 Grace systems this shared SMMUv3
causes CMD_SYNC timeout and tegra241_cmdqv errors in the guest,
preventing multi-GPU initialization.

When multiple IOMMU devices (marked with ACPI NodeSet "tofill") share
a NUMA node, each now gets its own pcie-expander-bus controller. A
single IOMMU device on a NUMA node continues to use the shared
per-NUMA expander bus. Non-IOMMU devices always share a
pcie-expander-bus per NUMA node.

Ref: VOYAGER-707

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Lee Yarwood <lyarwood@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Barakmor1 Barakmor1 Awaiting requested review from Barakmor1
@brianmcarey brianmcarey Awaiting requested review from brianmcarey
@EdDev EdDev Awaiting requested review from EdDev
@enp0s3 enp0s3 Awaiting requested review from enp0s3
@xpivarc xpivarc Awaiting requested review from xpivarc

Assignees

No one assigned

Labels

dco-signoff: yes Indicates the PR's author has DCO signed all their commits. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/buildsystem Denotes an issue or PR that relates to changes in the build system. sig/compute size/XXL

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@lyarwood @kubevirt-bot