Skip to content

Support including additional GPU functions on different IOMMU groups - #3478

Draft
masnax wants to merge 5 commits into
lxc:mainfrom
masnax:gpu-functions
Draft

Support including additional GPU functions on different IOMMU groups#3478
masnax wants to merge 5 commits into
lxc:mainfrom
masnax:gpu-functions

Conversation

@masnax

@masnax masnax commented Jun 11, 2026

Copy link
Copy Markdown
Collaborator

Incus' current behaviour with the gpu device type is to auto-load all additional GPU functions in the same IOMMU group.

If GPU functions are on different IOMMU groups, then they are not imported, and adding them as explicit pci devices causes Incus to import them as discrete PCI devices on separate root ports, which then has to be amended with raw.qemu.conf or raw.qemu.scriptlet.

Here, I've added a functions config key that accepts comma delimited additional function numbers (1-7) to enforce inclusion as long as the following path exists on the main device:

ls /sys/bus/pci/devices/0000:40:00.0/ -l | grep consumer
lrwxrwxrwx 1 root root         0 Jun 10 16:57 consumer:pci:0000:40:00.1 -> ../../../../../virtual/devlink/pci:0000:40:00.0--pci:0000:40:00.1
lrwxrwxrwx 1 root root         0 Jun 10 16:57 consumer:pci:0000:40:00.2 -> ../../../../../virtual/devlink/pci:0000:40:00.0--pci:0000:40:00.2
lrwxrwxrwx 1 root root         0 Jun 10 16:57 consumer:pci:0000:40:00.3 -> ../../../../../virtual/devlink/pci:0000:40:00.0--pci:0000:40:00.3

# But not present in the same IOMMU group:
ls /sys/bus/pci/devices/0000:40:00.0/iommu_group/devices  -l
total 0
lrwxrwxrwx 1 root root 0 Jun 10 17:52 0000:40:00.0 -> ../../../../devices/pci0000:00/0000:00:03.2/0000:3e:00.0/0000:3f:00.0/0000:40:00.0

So to include all of these, the devices section would look like this:

devices:
  gpu:
    functions: "1,2,3"
    pci: 40:00.0
    type: gpu

With the changes here, additional functions are enforced as a fallback to the existing IOMMU group inclusions.


Some additional steps are required before this can be considered ready:

  • Properly handle "transactional" unplug of the entire slot. This is related to Allow (un)plugging whole IOMMU groups at once #3422 but with the caveat that IOMMU groups may not be the whole source of truth.
  • Potentially use a more generic solution rather than the functions key, so it can apply to non-GPU devices too. This could look like explicit pci and gpu devices, where Incus internally determines if they ought to be treated as the same multifunction device.

masnax added 5 commits June 10, 2026 12:30
…mu_group loop

Signed-off-by: Max Asnaashari <max.asna@futurfusion.io>
…her iommu groups

Signed-off-by: Max Asnaashari <max.asna@futurfusion.io>
Signed-off-by: Max Asnaashari <max.asna@futurfusion.io>
Signed-off-by: Max Asnaashari <max.asna@futurfusion.io>
Signed-off-by: Max Asnaashari <max.asna@futurfusion.io>
@github-actions github-actions Bot added the Documentation Documentation needs updating label Jun 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Documentation Documentation needs updating

Development

Successfully merging this pull request may close these issues.

1 participant