Introduce and use `hack/sync-nv-rpms.sh` by lyarwood · Pull Request #3 · kubevirt/kubevirt-aie

lyarwood · 2026-02-26T14:58:26Z

What this PR does

Before this PR:

The kubevirt-aie project uses standard el10 RPMs from CentOS Stream 10 mirrors for libvirt and qemu-kvm packages across all architectures.

After this PR:

A new hack/sync-nv-rpms.sh script automates syncing NV-variant RPMs (el10nv dist tag) for libvirt and qemu-kvm from the CentOS Stream 10 koji build system. The el10nv packages replace the standard el10 packages in x86_64 and aarch64 cs10 rpmtree targets. s390x targets continue to use standard el10 packages as el10nv builds are not available for that architecture.

The script:

Auto-discovers the latest el10nv version and release from koji
Downloads each sub-package RPM, computes SHA-256 checksums and extracts epochs
Adds el10nv rpm() entries to WORKSPACE alongside existing el10 entries
Updates rpm/BUILD.bazel cs10 rpmtree targets for x86_64/aarch64 to reference el10nv packages

Synced packages:

libvirt 11.10.0-10.1.el10nv (8 sub-packages)
qemu-kvm 10.1.0-12.el10nv (9 sub-packages)

References

Why we need it and why it was done in this way

The NV-variant RPMs for libvirt and qemu-kvm include patches and configuration specific to the NVIDIA GPU virtualization use case. Consuming these from the koji build system ensures the kubevirt-aie images are built with the correct driver and device support for NV hardware passthrough on x86_64 and aarch64.

The following tradeoffs were made:

The sync script hardcodes the sub-package/arch matrix rather than discovering it dynamically, since the set of packages is stable and this avoids fragile HTML parsing of koji directory listings for each sub-package.
The script directly replaces el10 references with el10nv in BUILD.bazel rather than using a build-time toggle, keeping the build configuration simple.

The following alternatives were considered:

Making el10nv usage configurable at build time behind a flag — this was considered but dropped in favour of a simpler direct replacement approach. The sync script is idempotent and can be re-run to update versions.

Links to places where the discussion took place:

Special notes for your reviewer

The first commit introduces the script, the second commit contains the output of running it. To verify: reset to the first commit, run hack/sync-nv-rpms.sh, and diff against the second commit — the WORKSPACE checksums should match (given the same koji builds are still available).

Version/release overrides are available via environment variables (LIBVIRT_NV_VERSION, LIBVIRT_NV_RELEASE, QEMU_NV_VERSION, QEMU_NV_RELEASE) for pinning specific builds.

Checklist

Design: A design document was considered and is present (link) or not required
PR: The PR description is expressive enough and will help future contributors
Code: Write code that humans can understand and Keep it simple
Refactor: You have left the code cleaner than you found it (Boy Scout Rule)
Upgrade: Impact of this change on upgrade flows was considered and addressed if required
Testing: New code requires new unit tests. New features and bug fixes require at least one e2e test
Documentation: A user-guide update was considered and is present (link) or not required. You want a user-guide update if it's a user facing feature / API change.
Community: Announcement to kubevirt-dev was considered
AI Contributions: The PR abides by the KubeVirt AI Contribution Policy.

Release note

NONE

lyarwood · 2026-02-26T15:01:54Z

/hold

Growing list of deps that need to land first... Any help would be appreciated.

prow: Add CS10 build presubmit jobs for kubevirt project-infra#4745 - Introduces CentOS Stream 10 build lanes in kubevirt/kubevirt
centos-stream-10: Fix arm64 and s390x builds kubevirt#16938 - Fixes CentOS Stream 10 builds on kubevirt/kubevirt, tested by the above lanes. This PR needs to be rebased on this.
kubevirt-aie: Switch to KUBEVIRT_CENTOS_STREAM_VERSION=10 project-infra#4738 - Switches the kubevirt-aie jobs to CentOS Stream 10

lyarwood · 2026-02-27T17:09:12Z

/close

Introduce a script to sync NV-variant RPMs (el10nv) for libvirt and qemu-kvm from the CentOS Stream 10 internal koji build system. The script discovers the latest el10nv builds, downloads RPMs to compute SHA-256 checksums and epochs, then updates WORKSPACE with new rpm() entries and rpm/BUILD.bazel with el10nv package references for x86_64 and aarch64 cs10 rpmtree targets. s390x targets are left unchanged. Signed-off-by: Lee Yarwood <lyarwood@redhat.com> Assisted-By: Claude <noreply@anthropic.com>

Generated by hack/sync-nv-rpms.sh. Adds 31 el10nv rpm() entries to WORKSPACE and updates 44 references in rpm/BUILD.bazel cs10 rpmtree targets for x86_64 and aarch64. s390x targets continue to use standard el10 packages. libvirt 11.10.0-10.1.el10nv (x86_64, aarch64) qemu-kvm 10.1.0-12.el10nv (x86_64, aarch64) Signed-off-by: Lee Yarwood <lyarwood@redhat.com> Assisted-By: Claude <noreply@anthropic.com>

lyarwood · 2026-02-27T20:13:41Z

Latest changes are just shfmt fixes I missed in the original submission, the actual builds and e2e tests were both successful thankfully.

lyarwood · 2026-02-27T21:59:23Z

/test pull-kubevirt-aie-kind-1.35-sig-compute-arm64-1.8-aie-nv

lyarwood · 2026-02-28T09:48:09Z

/hold

Growing list of deps that need to land first... Any help would be appreciated.

prow: Add CS10 build presubmit jobs for kubevirt project-infra#4745 - Introduces CentOS Stream 10 build lanes in kubevirt/kubevirt

centos-stream-10: Fix arm64 and s390x builds kubevirt#16938 - Fixes CentOS Stream 10 builds on kubevirt/kubevirt, tested by the above lanes. This PR needs to be rebased on this.

FWIW this was backported early to this branch once approved on main through #4.

kubevirt-aie: Switch to KUBEVIRT_CENTOS_STREAM_VERSION=10 project-infra#4738 - Switches the kubevirt-aie jobs to CentOS Stream 10

With this all in place I think this is ready to merge early next week now.

/hold cancel

jean-edouard

This script is intense... Also highly experimental and dependent on the layout of external webpages.
I love awk as a parser, but the awk code needs a lot of comments, nobody is fluent in that language (anymore?). I'd also suggest moving it to its own .awk file.

At a higher level, what this script does sounds very similar to hack/rpm-deps.sh, except that hack/rpm-deps.sh is super simple. Is there maybe a way to use more similar logic?

Thanks

jean-edouard · 2026-03-02T16:08:14Z

+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Copyright 2025 Red Hat, Inc.


Don't we want to use usual Kubevirt copyright wording? Either way, year needs +1

lyarwood · 2026-03-02T19:07:25Z

This script is intense... Also highly experimental and dependent on the layout of external webpages. I love awk as a parser, but the awk code needs a lot of comments, nobody is fluent in that language (anymore?). I'd also suggest moving it to its own .awk file.

Thanks @jean-edouard, yeah appreciate it's awful but in the context of a one off effort having to pull in builds from koji without underlying repos I thought it was acceptable.

I can regenerate the script in bash if you think it would be easier to maintain on this branch?

At a higher level, what this script does sounds very similar to hack/rpm-deps.sh, except that hack/rpm-deps.sh is super simple. Is there maybe a way to use more similar logic?

It's unfortunately very different without access to actual repos and we ultimately don't want to modify the hack/rpm-deps.sh scripts at all on this branch as doing so might cause later rebases to conflict if they change on release-1.8.

Fix the copyright header year (2025 -> 2026) and extract the inline awk script into hack/sync-nv-rpms-update-workspace.awk with comments to help maintainers understand the WORKSPACE update logic. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Lee Yarwood <lyarwood@redhat.com>

jean-edouard · 2026-03-02T20:04:23Z

This script is intense... Also highly experimental and dependent on the layout of external webpages. I love awk as a parser, but the awk code needs a lot of comments, nobody is fluent in that language (anymore?). I'd also suggest moving it to its own .awk file.

Thanks @jean-edouard, yeah appreciate it's awful but in the context of a one off effort having to pull in builds from koji without underlying repos I thought it was acceptable.

I can regenerate the script in bash if you think it would be easier to maintain on this branch?

If doing things the rpm-deps way isn't feasible, at least we should:

mark the script as experimental somehow
extract the awk to its own file
add a healthy amount of comment to the awk code

Ultimately, this is just a maintenance script, not something that users, or even regular devs, are expected to use.

At a higher level, what this script does sounds very similar to hack/rpm-deps.sh, except that hack/rpm-deps.sh is super simple. Is there maybe a way to use more similar logic?

It's unfortunately very different without access to actual repos and we ultimately don't want to modify the hack/rpm-deps.sh scripts at all on this branch as doing so might cause later rebases to conflict if they change on release-1.8.

I'm not saying we should overload rpm-deps.sh, I just don't understand why we can't use similar mechanisms (basically bazeldnf) to update the RPMs instead of all that sed/awk.

lyarwood · 2026-03-02T20:16:46Z

This script is intense... Also highly experimental and dependent on the layout of external webpages. I love awk as a parser, but the awk code needs a lot of comments, nobody is fluent in that language (anymore?). I'd also suggest moving it to its own .awk file.

Thanks @jean-edouard, yeah appreciate it's awful but in the context of a one off effort having to pull in builds from koji without underlying repos I thought it was acceptable.
I can regenerate the script in bash if you think it would be easier to maintain on this branch?

If doing things the rpm-deps way isn't feasible, at least we should:

mark the script as experimental somehow

I'm not sure why, this isn't kubevirt/kubevirt and in the context of this branch this is a required script.

extract the awk to its own file

add a healthy amount of comment to the awk code

ACK done.

Ultimately, this is just a maintenance script, not something that users, or even regular devs, are expected to use.

At a higher level, what this script does sounds very similar to hack/rpm-deps.sh, except that hack/rpm-deps.sh is super simple. Is there maybe a way to use more similar logic?

It's unfortunately very different without access to actual repos and we ultimately don't want to modify the hack/rpm-deps.sh scripts at all on this branch as doing so might cause later rebases to conflict if they change on release-1.8.

I'm not saying we should overload rpm-deps.sh, I just don't understand why we can't use similar mechanisms (basically bazeldnf) to update the RPMs instead of all that sed/awk.

Ultimately bazeldnf needs actual dnf repos to query, unfortunately the AIE SIG has decided to not provide this for the NV effort thus forcing us to pull directly from koji.

There are a few command line options, the original koji and koji-tool, but both would require changing the builder image etc. Maybe something for a follow up.

jean-edouard · 2026-03-02T20:19:26Z

This script is intense... Also highly experimental and dependent on the layout of external webpages. I love awk as a parser, but the awk code needs a lot of comments, nobody is fluent in that language (anymore?). I'd also suggest moving it to its own .awk file.

Thanks @jean-edouard, yeah appreciate it's awful but in the context of a one off effort having to pull in builds from koji without underlying repos I thought it was acceptable.
I can regenerate the script in bash if you think it would be easier to maintain on this branch?

If doing things the rpm-deps way isn't feasible, at least we should:

mark the script as experimental somehow

I'm not sure why, this isn't kubevirt/kubevirt and in the context of this branch this is a required script.

extract the awk to its own file

add a healthy amount of comment to the awk code

ACK done.

Ultimately, this is just a maintenance script, not something that users, or even regular devs, are expected to use.

At a higher level, what this script does sounds very similar to hack/rpm-deps.sh, except that hack/rpm-deps.sh is super simple. Is there maybe a way to use more similar logic?

It's unfortunately very different without access to actual repos and we ultimately don't want to modify the hack/rpm-deps.sh scripts at all on this branch as doing so might cause later rebases to conflict if they change on release-1.8.

I'm not saying we should overload rpm-deps.sh, I just don't understand why we can't use similar mechanisms (basically bazeldnf) to update the RPMs instead of all that sed/awk.

Ultimately bazeldnf needs actual dnf repos to query, unfortunately the AIE SIG has decided to not provide this for the NV effort thus forcing us to pull directly from koji.

There are a few command line options, the original koji and koji-tool, but both would require changing the builder image etc. Maybe something for a follow up.

That sounds great! I'll have another look at the scripts. Thanks

jean-edouard

I can't find anything particularly wrong in the scripts. I did my best and reviewed this line by line, but honestly the cognitive complexity here is beyond what my small human brain can handle, so I'm approving without a clear big-picture view.
It's ok because:

This is a maintenance script
In a separate repo
The second commit proves it works, and nothing will immediately break if it stops working

I just really hope all PRs don't become this convoluted in the near future.....

/approve

kubevirt-bot · 2026-03-02T21:23:14Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jean-edouard, lyarwood

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [jean-edouard]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

fossedihelm · 2026-03-03T13:20:04Z

+            local tmpfile="${TMPDIR}/${rpm_filename}"
+
+            echo "  ${rpm_filename}"
+            curl -sSL -o "${tmpfile}" "${rpm_url}"


Should we verify GPG signatures?

These packages aren't signed so we can't unfortunately:

$ rpm -K qemu-kvm-10.1.0-12.el10nv.x86_64.rpm qemu-kvm-10.1.0-12.el10nv.x86_64.rpm: digests OK $ rpm -qip qemu-kvm-10.1.0-12.el10nv.x86_64.rpm Name : qemu-kvm Epoch : 19 Version : 10.1.0 Release : 12.el10nv Architecture: x86_64 Install Date: (not installed) Group : Unspecified Size : 0 License : GPL-2.0-only AND GPL-2.0-or-later AND CC-BY-3.0 Signature : (none) Source RPM : qemu-kvm-10.1.0-12.el10nv.src.rpm Build Date : Tue 17 Feb 2026 17:59:38 GMT Build Host : x86-04.stream.rdu2.redhat.com Packager : builder@centos.org Vendor : CentOS URL : http://www.qemu.org/ Summary : QEMU is a machine emulator and virtualizer Description : qemu-kvm is an open source virtualizer that provides hardware emulation for the KVM hypervisor. qemu-kvm acts as a virtual machine monitor together with the KVM kernel modules, and emulates the hardware for a full system such as a PC and its associated peripherals.

fossedihelm · 2026-03-03T14:05:48Z

/lgtm

lyarwood requested review from fossedihelm, vladikr and xpivarc February 26, 2026 14:58

kubevirt-bot added release-note-none Denotes a PR that doesn't merit a release note. dco-signoff: yes Indicates the PR's author has DCO signed all their commits. size/XL labels Feb 26, 2026

kubevirt-bot requested a review from brianmcarey February 26, 2026 14:59

kubevirt-bot added the sig/buildsystem Denotes an issue or PR that relates to changes in the build system. label Feb 26, 2026

kubevirt-bot requested a review from enp0s3 February 26, 2026 14:59

kubevirt-bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 26, 2026

lyarwood marked this pull request as draft February 26, 2026 15:11

kubevirt-bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Feb 26, 2026

lyarwood force-pushed the el10nv branch from 5a75ec2 to dcea8fe Compare February 27, 2026 16:33

kubevirt-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 27, 2026

lyarwood marked this pull request as ready for review February 27, 2026 17:05

kubevirt-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 27, 2026

lyarwood closed this Feb 27, 2026

lyarwood reopened this Feb 27, 2026

lyarwood marked this pull request as draft February 27, 2026 17:13

kubevirt-bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 27, 2026

lyarwood force-pushed the el10nv branch from dcea8fe to 8d5510f Compare February 27, 2026 17:13

lyarwood marked this pull request as ready for review February 27, 2026 17:14

kubevirt-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 27, 2026

kubevirt-bot requested a review from jean-edouard February 27, 2026 17:14

lyarwood force-pushed the el10nv branch from 8d5510f to f9b6655 Compare February 27, 2026 20:10

lyarwood added 2 commits February 27, 2026 20:11

lyarwood force-pushed the el10nv branch from f9b6655 to b83632f Compare February 27, 2026 20:11

kubevirt-bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 28, 2026

jean-edouard reviewed Mar 2, 2026

View reviewed changes

kubevirt-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 2, 2026

fossedihelm reviewed Mar 3, 2026

View reviewed changes

kubevirt-bot assigned fossedihelm Mar 3, 2026

kubevirt-bot added the lgtm Indicates that a PR is ready to be merged. label Mar 3, 2026

kubevirt-bot merged commit 28c016b into kubevirt:release-1.8-aie-nv Mar 3, 2026
8 checks passed

Uh oh!

Conversation

lyarwood commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does

Before this PR:

After this PR:

References

Why we need it and why it was done in this way

Special notes for your reviewer

Checklist

Release note

Uh oh!

lyarwood commented Feb 26, 2026

Uh oh!

lyarwood commented Feb 27, 2026

Uh oh!

lyarwood commented Feb 27, 2026

Uh oh!

lyarwood commented Feb 27, 2026

Uh oh!

lyarwood commented Feb 28, 2026

Uh oh!

jean-edouard left a comment

Choose a reason for hiding this comment

Uh oh!

jean-edouard Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

lyarwood commented Mar 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jean-edouard commented Mar 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lyarwood commented Mar 2, 2026

Uh oh!

jean-edouard commented Mar 2, 2026

Uh oh!

jean-edouard left a comment

Choose a reason for hiding this comment

Uh oh!

kubevirt-bot commented Mar 2, 2026

Uh oh!

fossedihelm Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

lyarwood Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

fossedihelm commented Mar 3, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

lyarwood commented Feb 26, 2026 •

edited

Loading

lyarwood commented Mar 2, 2026 •

edited

Loading

jean-edouard commented Mar 2, 2026 •

edited

Loading