RFC: CMake/VCPKG: Require SDL3 to have either X11 or Wayland backends

[SharkWipf]

Probably not for merging. And untested. I'm drafting this, because I'm not happy with this solution, but I want to illustrate the problem.

Currently, SDL3 will happily build without wayland and/or x11 if the required dev libraries aren't available, without raising any errors.
The build will complete without any indication of something being wrong, and a LFS binary is produced that cannot launch a GUI under any circumstances, because it lacks X11/Wayland support:

$ ./build/LichtFeld-Studio
[...]
Failed to initialize SDL: No available video device

$ SDL_VIDEODRIVER=x11 ./build/LichtFeld-Studio
[...]
Failed to initialize SDL: x11 not available

$ SDL_VIDEODRIVER=wayland ./build/LichtFeld-Studio
[...]
Failed to initialize SDL: wayland not available

To make matters worse, installing these libraries later does not invalidate the build cache, so the user has to figure out what's wrong from no errors, figure out the missing packages, then do a full --fresh build, to solve the problem.

So, the solution would be simple you'd think. Make SDL3 require either X11 or Wayland or fail the build.
Except it doesn't seem SDL3's vcpkg exposes any way to control this.

It is possible to override SDL3's vcpkg portfile as done in this PR. But this gets messy fast. Because in order to do so, we also have to override rmgui's portfile, because it needs to pull in our new custom SDL3 version. And any other package pulling in SDL3 will also need a custom portfile.
Disregarding that this specific implementation is a mess of course, the whole portfile override solution is nasty. But I don't think there's any other solution, at least not pre-compile.

So in short:

• SDL3 will happily build without either X11 or Wayland if the required libraries aren't available at build time
• This is a problem because it doesn't throw errors and can't be enforced
• The solutions available are messy and nasty

What is the desired way forward here, if any?

The required (and currently undocumented) packages seem to be (Debian packages, Wayland and X11 mixed):

libx11-dev libxext-dev libxcursor-dev libxi-dev libxfixes-dev libxrandr-dev libxrender-dev libxss-dev libxtst-dev libwayland-dev libwayland-bin wayland-protocols libxkbcommon-dev libegl-dev libdecor-0-dev libdrm-dev libgbm-dev

[SharkWipf]

Lol. The CI proves it works at least I guess 😅

[MrNeRF]

ok, thanks. I will look into it tomorrow

[SharkWipf]

Nothing too urgent, only got one report so far. Maybe just updating the docs is enough for now

[SharkWipf]

Alright, update, I have just verified this behavior, as well as the fix.

I am thinking the cleanest fix here will be not to force checks on SDL, but to do pre-flight checks in our own CMake that'll trigger an error when known-required packages are missing.
This has as additional huge side benefit that we can do ahead-of-time build environment sanity checking, and in doing so avoid hitting users with confusing misplaced VCPKG errors out of our control.
That alone would be a huge win IMO, because it's a massive source of confusion.

If you're not opposed to that, I can work on that. Just let me know.

From discord, direct copy-paste:

---

Okay, I just confirmed: SDL3 does not rebuild after you install the required packages even if you delete your entire build folder and rerun cmake with --fresh, thanks to a messed up vcpkg build file and vcpkg lacking any and all mechanisms to manually invalidate a cached package and force a rebuild.

Solution:

1. Make sure VCPKG_ROOT is still set (echo "$VCPKG_ROOT")

2. cd to your LichtFeld-Studio folder (not the build subdir.

3. Wipe SDL3 (and its dependents) from the build cache only:

lfs="$(pwd)"; cd "$VCPKG_ROOT" && ./vcpkg remove sdl3:x64-linux --recurse --x-install-root="$LFS/build/vcpkg_installed: --x-packages-root="$VCPKG_ROOT/packages" --x-buildtrees-root="$VCPKG_ROOT/buildtrees"; cd "$lfs"

4. Make sure you have all required packages for sure:

sudo apt install libegl1-mesa-dev libegl-dev libibus-1.0-dev libwayland-dev libx11-dev libxcursor-dev libxext-dev libxfixes-dev libxft-dev libxi-dev libxkbcommon-dev libxrandr-dev libxss-dev libxtst-dev

5. Rebuild LFS, with your usual command, but add the VCPKG_BINARY_SOURCES='clear;default,write' (crucial, if you forget this you have to redo all of this again)

VCPKG_BINARY_SOURCES='clear;default,write'; cmake -B build -DCMAKE_BUILD_TYPE=Release -G Ninja --fresh && cmake --build build -- -j$(nproc);

(Alternatively you can wipe your entire build folder and rerun with that VCPKG_BINARY_SOURCES='clear;default,write', but that will rebuild everything and take forever.)

I also confirmed this is indeed the fix

I also confirmed that aside from cuda/cuda-toolkit itself, these are the minimal (for Wayland and X11 SDL support at least) packages you need on debian (and thus presumably popos, ubuntu, mint, etc) you need to have apt installed to build LFS GUI properly:

autoconf autoconf-archive automake build-essential cmake curl git libcuda1 libegl-dev libegl1-mesa-dev libgl1-mesa-dev libglu1-mesa-dev libglx-dev libibus-1.0-dev libmesa-dev libtool libwayland-dev libx11-dev libxcursor-dev libxext-dev libxfixes-dev libxft-dev libxi-dev libxkbcommon-dev libxrandr-dev libxss-dev libxtst-dev mesa-dev nasm ninja ninja-build npm opengl-dev pkg-config python3 tar unzip wget zip

I will wrap this up in a more proper documentation tomorrow or so
Validated on an extremely bare-bones debootstrap chroot to rule out any pre-installed packages 😄
I know Fedora/EL needs other packages manually installed though so I'm not done yet

[MrNeRF]

Hmm, I added now a check so that the cmake configuration will fail in case the sdl version is not correct.
I think that's better then the custom overlays.

[SharkWipf]

Hmm, at a glance, I don't think this fixes it? That was what my codex also suggested at first and I rejected it, because it only checks the final built version.
The problem with that is, aside from wasting resources building the wrong sdl3, vcpkg sdl3 will not rebuild once you do install the correct packages. The stale build will not be rebuilt unless you manually clear both the vcpkg build cache and the site-installed sdl3, which vcpkg makes extremely difficult and unintuitive.
So preventing the "bad" SDL3 from being built would be the better outcome here.
I agree the overlay approach is bad, but a simple preflight cmake configuration/package checker of our own, before running vcpkg, would probably be the cleanest approach.

Edit: Actually it does provide instructions for the user how to unstale this I guess. Still, I think LFS could benefit from a general pre-flight check, as vcpkg output is a mess to figure out whenever errors happen, and a lot of stuff we could easily prevent by simply doing some pre-flight cmake checks of known-needed packages.

Edit 2: So I guess what I'm saying is, we'd ideally have both.