Run Nvidia Vulkan headless without Xorg!

[boberfly]

Hi all,

I was looking around and discovered something kind of amazing and can potentially resolve the need for the nv_vulkan_wrapper entirely. This is what I did:

1. Make a file for testing in $HOME/.local/share/vulkan/icd.d/nvidiaegl_icd.json
2. Fill it with this, which is a copy of nvidia's standard icd only it is pointing to another library (I am using 415.27 of nvidia's driver):
   { "file_format_version" : "1.0.0", "ICD": { "library_path": "libEGL_nvidia.so.0", "api_version" : "1.1.84" } }
3. Copy primus_vk.json to $HOME/.local/share/vulkan/implicit_layer.d/ like normal that is pointing to the actual primus_vk.so and NOT the nv_vulkan_wrapper at all.
4. Run a Vulkan app with just enabling the ENABLE_PRIMUS_LAYER=1 your_vulkan_app

I'm doing this with The-Forge and the log file now tells me it is using my Nvidia card instead of my AMD card. I am not using an optimus laptop here also, so this seems to work on a dual-gpu setup in a regular PC. My display is going through a Vega Frontier Edition but it is being rendered on a Quadro K2000 (I will later test on an RTX 2070 and see if it works).

I have noticed some crazy stutter though, so there might be something else which needs to be fixed here.

[leonmaxx]

Quite interesting, I'll surely will test this approach on this weekend. Probably will need to modify bumblebee to not start Xorg, or create some script to enable discrete video card and load nvidia modules to test.

Edit: optirun already have option --no-xorg :).

[leonmaxx]

I have noticed some crazy stutter though

Do your test app have V-Sync enabled?

[boberfly]

@leonmaxx possibly, V-Sync might be the issue here you're right. Maybe I need to apply that modesetting flag too I'll try that when I get the chance, cheers for the tip!

As for the bumblebee question, this is probably right but I can't use it here as I get some error, but I am not worried about power usage in my workstation so no big deal... :)

[boberfly]

One thing I noticed is that vulkaninfo will display that the device does exist, but it segfaults when looking for available outputs, which makes sense as we are loading a vulkan instance from EGL in an environment which it is not expecting (perhaps that modesetting flag nvidia-drm.modeset=1 is the key to enabling this?)

[leonmaxx]

@boberfly If you use kernel 4.17+, you can use in-kernel pci-e power management. All you need is just to unload nvidia kernel modules and set /sys/bus/pci/*device*/power/control to auto, and kernel will put your card to suspend state. It'll wake-up automatically when nvidia kernel modules is loaded.

[leonmaxx]

If interested I use this udev rule to apply pm automatically on boot nvrtpm.rules:

ACTION=="add|change", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x030200", ATTR{power/control}="auto"
ACTION=="add|change", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x030000", ATTR{power/control}="auto"

[boberfly]

Sadly anything past kernel 4.15 absolutely does not work on my machine, the amdgpu driver doesn't seem to work anymore in 4.17 (it did before) and past 4.18 onwards there is something weird with my hardware topology which does a kernel panic on startup, something I need to bug up about but haven't had the chance or patience yet...

[leonmaxx]

I can confirm it's working! I just launched WoT with suggested changes to .json and ENABLE_PRIMUS_LAYER=1 optirun --no-xorg wine WorldOfTanks.exe and it works!

[leonmaxx]

If anyone wants to test I added primus-vk-headless package to my copr repository which works without Xorg.

[boberfly]

Good to hear @leonmaxx is the performance decent, and do you have nvidia-drm.modeset=1 set with vsync on/off?

[boberfly]

I have a patch set which will come soon which allows 2 environment variables to set which GPU does what via vendorID:deviceID hex numbers, at least in my case I need to do this with 2 discrete GPUs.

[leonmaxx]

Performance is good, I have stable 60 FPS with V-Sync on. Mouse latency seems to be better.
I do not have nvidia-drm.modeset=1 set. I use only nvidia kernel module, other modules nvidia-drm and nvidia-modeset is disabled using alias ... off.
In my notebook I have GeForce 1050 without hardware outputs (PCI device class 0x302 - 3D accelerator), possibly that is why it works without modeset.

[leonmaxx]

And I didn't noticed any stuttering.

[boberfly]

@leonmaxx hey I got vsync working, it has fixed the hitching but it is very very slow on The-Forge unfortunately

Also I just made a PR which allows to set env vars on what GPU to use for display and rendering if you wanted to test it, but I guess you don't need it on optimus laptops...