Fedora 44 + PCI passthrough

Note

(more about this metric and what it means at securityscorecards.dev) - track progress on anything surfaced by it in this repo's issues

Prerequisites

• Fedora 44 (fresh install off the live USB image)
• Computer with
  • Two graphics cards
  • Motherboard with the Intel 200 series chipset (Union Point) or newer
• A fresh backup of anything you don't care to lose

🆕 Just want the RPMs? There's a dnf repo, so you can skip every (ACS only) step below and install the patched kernel directly. It's rebuilt daily against the newest Fedora kernel, so dnf update keeps you current.

sudo dnf config-manager addrepo --from-repofile=https://some-natalie.github.io/fedora-acs-override/acs-override.repo
sudo dnf install kernel-acs kernel-acs-core kernel-acs-devel kernel-acs-modules kernel-acs-modules-extra

These are named kernel-acs* rather than kernel* on purpose — they're a separate package family from Fedora's own kernel, so installing them never replaces or conflicts with the stock one. Both stay installed and bootable, dnf update tracks each independently, and you pick which one runs at the boot menu or with grubby --set-default. uname -r tells you which you're on, since the ACS builds carry .acs in the release.

The five most recent kernel versions stay in the repo, so if a new one breaks your passthrough you can boot the previous entry, or make it the default:

dnf list --showduplicates kernel-acs-core    # what's available
sudo grubby --info=ALL | grep -E 'index|title'
sudo grubby --set-default=/boot/vmlinuz-7.1.9-200.acs.fc44.x86_64

Kernels are installonly packages, so dnf downgrade refuses to touch them. To pull an older build back down, install it by exact version — it lands alongside what you already have rather than replacing it:

sudo dnf install kernel-acs-core-7.1.8-200.acs.fc44 kernel-acs-modules-7.1.8-200.acs.fc44

Those five stick around after a Fedora release goes EOL and stops being built, so upgrading Fedora is your call to make rather than something a build schedule forces.

The packages and the repo metadata are both signed, so dnf asks you to accept the signing key the first time. Check it before you accept it — gpg --show-keys on what the site serves should print FC3F2A6C5D05CE26434442BBD9500E334C48DD8B, matching pages/RPM-GPG-KEY-acs-override here in the repo. Every build is attested too, so gh attestation verify <rpm> --repo some-natalie/fedora-acs-override will tell you which workflow run produced it.

Tip

The two graphics cards can be different models. If your cards are identical, you'll need to do some extra steps that are prefaced with (ACS only). These steps include compiling your own kernel to include Alex Williamson's patch to allow any PCIe device to use Access Control Services. More information on this patch, why it's necessary, and what it does is available on the LKML thread that proposed it. If your cards aren't identical, skip these steps.

Setup and configuration of the host machine

1. Add RPM Fusion

   sudo dnf install https://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm https://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm

2. (ACS only) - Install the dependencies to start building your own kernel.

   sudo dnf install fedpkg fedora-packager rpmdevtools ncurses-devel pesign bpftool

3. (ACS only) - Set up your home build directory (if you haven't ever built any RPMs before)

   rpmdev-setuptree

4. (ACS only) - Install the kernel source and finish installing dependencies.

   koji download-build --arch=src kernel-6.2.15-300.fc40.src.rpm
   rpm -Uvh kernel-6.2.15-300.fc40.src.rpm
   cd rpmbuild/SPECS/
   sudo dnf builddep kernel.spec

5. (ACS only) - Add the ACS patch from this repo as ~/rpmbuild/SOURCES/add-acs-override.patch.

   curl -o ~/rpmbuild/SOURCES/add-acs-override.patch https://raw.githubusercontent.com/some-natalie/fedora-acs-override/main/acs/add-acs-override.patch    

6. (ACS only) - Edit ~/rpmbuild/SPECS/kernel.spec to rename the package, set the build ID, and add the patch. Since each release of the spec file could change, it's not much help giving line numbers, but the first two of these should be near the top of the file. Change the package name so your build is a separate package family from Fedora's kernel instead of competing with it — every subpackage is named off this one line, so kernel-acs-core, kernel-acs-devel and the rest follow.

   %global package_name kernel-acs

   Then set the build id, near the top with the other release information, so uname -r says which kernel you're on.

   # Set buildid
   %define buildid .acs

   To add the patch, add the two lines below to the spec file in the section for patches (usually right below the sources).

   # ACS override patch
   Patch1000: add-acs-override.patch

   Then tell it to apply the patch in the prep section. It will be below the ApplyOptionalPatch() function definition, normally right above the # END OF PATCH APPLICATIONS comment.

   ApplyOptionalPatch add-acs-override.patch

7. (ACS only) - Compile! This takes a long time.

   rpmbuild -bb kernel.spec

8. (ACS only) - Install the new packages!

   cd ~/rpmbuild/RPMS/x86_64
   sudo dnf localinstall *.rpm

   ℹ️ You should now have at least the following packages installed: kernel-acs, kernel-acs-core, kernel-acs-devel, kernel-acs-modules, and kernel-acs-modules-extra. Your stock kernel packages are untouched and still bootable.

9. Update and reboot

   sudo dnf clean all
   sudo dnf update -y
   sudo reboot

10. Install virtualization software and add yourself to the user group

    sudo dnf install @virtualization
    sudo usermod -G libvirt -a $(whoami)
    sudo usermod -G kvm -a $(whoami)

11. Install (proprietary) nVidia drivers and remove/blacklist (open source) nouveau drivers.

    sudo su -
    dnf install xorg-x11-drv-nvidia akmod-nvidia kernel-devel xorg-x11-drv-nvidia-cuda vulkan vdpauinfo libva-utils
    dnf remove *nouveau*
    echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf

    ℹ️ On the ACS kernel that's kernel-acs-devel instead of kernel-devel. It provides kernel-devel-uname-r, which is what akmod matches against, so the nVidia module builds against it the same way.

12. Reboot again!

    sudo reboot

13. Verify that the nVidia drivers are installed correctly.

    lsmod | grep nouveau   # This should display nothing
    lsmod | grep nvidia    # This should display at least a couple things

14. Edit /etc/default/grub to enable IOMMU, blacklist nouveau, and load vfio-pci first.

    If your video cards are different:

    GRUB_CMDLINE_LINUX="rd.driver.pre=vfio-pci rd.driver.blacklist=nouveau modprobe.blacklist=nouveau rd.lvm.lv=fedora/root rd.lvm.lv=fedora/swap rhgb quiet intel_iommu=on iommu=pt"

    If your video cards are identical, use this instead:

    GRUB_CMDLINE_LINUX="rd.driver.pre=vfio-pci rd.driver.blacklist=nouveau,nova_core modprobe.blacklist=nouveau,nova_core nvidia-drm.modeset=1 rhgb quiet intel_iommu=on iommu=pt pcie_acs_override=downstream"

15. Rebuild GRUB's configuration

    sudo grub2-mkconfig -o /boot/grub2/grub.cfg

16. Edit /etc/modprobe.d/kvm.conf. These two lines are edited out due to stability concerns. YMMV.

    #options kvm_intel nested=1
    #options kvm_amd nested=1

17. Create or edit /etc/modprobe.d/local.conf, adding the line below:

    install vfio-pci /sbin/vfio-pci-override.sh

18. Create or edit /etc/dracut.conf.d/local.conf, adding the line below:

    add_drivers+= " vfio vfio_iommu_type1 vfio_pci vfio_virqfd "
    install_items+=" /sbin/vfio-pci-override.sh /usr/bin/find /usr/bin/dirname "

19. Create a file /sbin/vfio-pci-override.sh with permissions 755 (file in this directory of the repo).

20. Rebuild using dracut

    sudo dracut -f --kver `uname -r`

21. Reboot again!

22. Verify that your target hardware is using vfio-pci as the driver. Omit the -s 00:02:00 on another machine to get the entire output, as this argument narrows the output down to the device specified.

    nataliepc /h/n/k/kernel $ lspci -vv -n -s 00:02:00
    02:00.0 0300: 10de:1c82 (rev a1) (prog-if 00 [VGA controller])
      Subsystem: 3842:6251
      Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
      Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
      Interrupt: pin A routed to IRQ 10
      Region 0: Memory at dc000000 (32-bit, non-prefetchable) [disabled] [size=16M]
      Region 1: Memory at a0000000 (64-bit, prefetchable) [disabled] [size=256M]
      Region 3: Memory at b0000000 (64-bit, prefetchable) [disabled] [size=32M]
      Region 5: I/O ports at d000 [disabled] [size=128]
      Expansion ROM at dd000000 [disabled] [size=512K]
      Capabilities: <access denied>
      Kernel driver in use: vfio-pci
      Kernel modules: nouveau, nvidia_drm, nvidia
    
    02:00.1 0403: 10de:0fb9 (rev a1)
      Subsystem: 3842:6251
      Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
      Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
      Interrupt: pin B routed to IRQ 11
      Region 0: Memory at dd080000 (32-bit, non-prefetchable) [disabled] [size=16K]
      Capabilities: <access denied>
      Kernel driver in use: vfio-pci
      Kernel modules: snd_hda_intel

23. Proceed to set up your virtual machine.

Resources

• Alex Williamson's blog on VFIO tips and tricks
• Arch Linux wiki post on PCI passthrough

Disclaimer

I put this together based on my own machine at home because I knew I'd forget this process if I ever had to do it again. There was a lot of reading of Bugzilla, StackOverflow, and a bunch of blogs/forums/mailing lists all over the internet. Thanks to everyone who did something similar so I could cobble together something from all of them that Works On My Machine. This is by no means the only way to solve the problem. 😃