Build AMD MxGPU and package them

[olivierlambert]

https://github.com/GPUOpen-LibrariesAndSDKs/MxGPU-Virtualization

AMD already build some RPMs for XS, and the package is free, but it seems it's only built for 4.4 kernel (see mxgpu-4.4.0+10-modules-1.0.5.amd-1.x86_64.rpm)

So we should be able to package that ourselves.

[stormi]

I can't find the source RPM so I'll have to try to build things manually from that github repo and see if it matches the contents of the RPM.

[olivierlambert]

The RPM is not in Citrix ISO but on AMD website, in the driver page, you'll have a "XenServer driver download". You can find the RPMs in there.

[stormi]

I had found it but there is no source RPM, only the binary one.

[olivierlambert]

Indeed, no SRPM. Maybe there's one for KVM.

[armouredking]

Was able to manually build the Github repo on XCP-ng with current kernel in v8 and load the gim module. I get the proper lspci output with the card showing all the virtual interfaces on PCI. But the control interfaces don't seem to know how to use it? I can't attach the GPUs to the VM in XOA or in Center, it just tells me I can passthrough the entire card but not the virtual adapters (Virtual GPU type in Center is just "Passthrough whole GPU").

[stormi]

@armouredking sorry for late answer. I haven't had the opportunity to test myself yet so I don't know what to expect.

[stormi]

Bookmark for myself: latest officially built MxGPU GIM (for XS >= 7.4 and < 8 ): https://www.amd.com/en/support/kb/release-notes/rn-pro-mxgpu-gim-1-05

[stormi]

@armouredking did you follow the deployment guide from the page linked above?

[stormi]

The mxgpu RPM that AMD distributed for XS 7.4+ contains more than what the github repository has:

.
|-- etc
|   |-- gim_config
|   `-- mxgpu-whitelist
|-- lib
|   `-- modules
|       `-- 4.4.0+10
|           `-- updates
|               |-- gim-api.ko
|               `-- gim.ko
`-- opt
    `-- xensource
        `-- bin
            `-- gimtool

Two configuration files, a tool named gimtool (with no headers saying what the license is, though the RPM License tag is GPLv2 so we can consider it's free).

I don't know what gim-api.ko is.

In https://github.com/GPUOpen-LibrariesAndSDKs/MxGPU-Virtualization master builds fine with our kernel in XCP-ng 8.0, but this pull request may improve the compatibility. I'm not able to evaluate it and haven't the hardware to test. Edit: actually the pull request seems not necessary anymore. The code since this commit takes kernel >= 4.14 into account. There's an open pull request for kernel >= 5 though, but we aren't there yet.

[Arvee42]

Confirmed. Can compile the github drivers on XCP-NG 8.0. Modprobe seems happy but can only pass through the GPU on an S7150x2. Had to yum install gcc git and kernel-devel to allow the gim.sh script to run.

[Arvee42]

Compared lspci output from 8.0 and 7.6. 7.6 shows a lot more GPU devices versus only 2 "Tonga XT GL" GPUs on 8.0.

Dunno if that helps.