external_deps: revamp lib build, add cmake and configure wrapper, update lib feature disablement, update lib versions

[illwieckz]

This is WIP, current external_deps build status:

system	deps build	static build	test
linux-amd64-default	✅️	✅️	✅️
linux-arm64-default	✅️
linux-i686-default	✅️	✅️	✅️
linux-armhf-default	✅️
windows-amd64-mingw	✅️	N/A	✅️
windows-i686-mingw	✅️ Buster ❌️ GLEW Noble	N/A	✅️
windows-amd64-msvc	✅️
windows-i686-msvc	✅️
macos-amd64-default	✅️	N/A	✅️

I haven't tested if the engine builds and runs properly with those. WIP

What this PR does:

• Use CMake when packages provide a CMakeLists.txt file
• Factorize much code
• Make a nice list of package base url, so it's easy to click all of them and check for new versions
• Update package versions
• Update package build options

[illwieckz]

For windows-i686-mingw I get this with GLEW:

i686-w64-mingw32-ld -nostdlib -shared -soname libglew32.dll --out-implib lib/libglew32.dll.a
     -o lib/glew32.dll tmp/linux-mingw32/default/shared/glew.o
     -Lexternal_deps/build-windows-i686-mingw_10/prefix/lib -L/usr/i686-w64-mingw32/lib
     -lopengl32 -lgdi32 -luser32 -lkernel32 
i686-w64-mingw32-ld: tmp/linux-mingw32/default/shared/glew.o:glew.c:(.text+0x141ab):
 undefined reference to `strlen'

Everything else build.

[illwieckz]

For windows-amd64-msvc I get this with Vorbis:

[ 84%] Linking C shared library libvorbis.dll
/usr/bin/x86_64-w64-mingw32-ld:
 external_deps/build-windows-amd64-msvc_10/vorbis/libvorbis-1.3.7/win32/vorbis.def:3: syntax error
/usr/bin/x86_64-w64-mingw32-ld:
 external_deps/build-windows-amd64-msvc_10/vorbis/libvorbis-1.3.7/win32/vorbis.def: file format not recognized; treating as linker script
/usr/bin/x86_64-w64-mingw32-ld:
 external_deps/build-windows-amd64-msvc_10/vorbis/libvorbis-1.3.7/win32/vorbis.def:2: syntax error

Everything else build.

[illwieckz]

• OGG has a CMakeLists.txt file but no install target, so I returned back to configure for this one.
• Opus requires at least CMake 3.16.

[illwieckz]

When building newer Opus on Debian buster (the distro we use for our releases), I get this:

opus/opus-1.5.2/silk/x86/NSQ_del_dec_avx2.c:959:43: warning: implicit declaration of function '_mm_loadu_si64'; did you mean '_mm_loadu_si32'? [-Wimplicit-function-declaration]
         __m256i x = _mm256_cvtepi16_epi64(_mm_loadu_si64(&x16[i]));
                                           ^~~~~~~~~~~~~~
                                           _mm_loadu_si32

Debian Buster provides GCC 8, but GCC 11.3 or GCC 12 may be required:
https://stackoverflow.com/questions/72837929/mm-loadu-si32-not-recognized-by-gcc-on-ubuntu

[illwieckz]

When building newer Opus on Debian buster (the distro we use for our releases), I get this: […]

Telling Opus to not assume more than SSE2 fixed that.

[illwieckz]

So, the MinGW GLEW and MSVC Vorbis errors are the only ones.

[illwieckz]

So the Vorbis build error is actually a bug:

• [libvorbis] Build error microsoft/vcpkg#22990
• [libvorbis] Fix mingw-dynamic, modernize port microsoft/vcpkg#23761

I added a workaround.

[illwieckz]

I don't get why I get that GLEW build error with MinGW, the build function has not been modified, and the version has not been updated.

[illwieckz]

I don't get why I get that GLEW build error with MinGW, the build function has not been modified, and the version has not been updated.

Also the code is the same for both windows-amd64-msvc, windows-i686-msvc, windows-amd64-mingw and windows-i686-mingw, but it only fails with windows-i686-mingw… That doesn't make sense.

[illwieckz]

So, I don't know what happened, now I don't reproduce the MinGW GLEW error… Maybe I gorgot to prune the prefix folder and some stray files messed-up…

[illwieckz]

Ah, I now see something: I reproduce the bug with MinGW from Ubuntu 24.04 Noble, not with MinGW from Debian 10 Buster. So, since we produce release builds with Debian Buster, it's not a big problem, but it should be fixed for the future…

[slipher]

What's the purpose of migrating things to build with CMake?

[illwieckz]

So with this a static build for linux-amd64 completes and runs.

[slipher]

So with this a static build for linux-amd64 completes and runs.

Huh? It worked before, so I have a hard time believing that is suddenly necessary to change the build system of 8 packages.

[illwieckz]

I never said this is a response to “What's the purpose of migrating things to build with CMake?”, I'm just reporting the status of me testing that branch. I said in first post:

I haven't tested if the engine builds and runs properly with those.

So now I'm running those tests.

[DolceTriade]

What's the motivation behind this change?

[illwieckz]

Purposes:

1. Update packages to latest versions,
2. Update packages build options for latest versions (for example curl adds protocols, so we have to disable them),
3. Mutualize cmake/configure code as generic reused functions and avoid copy/paste boilerplate,
4. Rely on CMake as much as possible to make it easy to spot new CMake build options to disable them and other changes.

When using configure it is hard to compare the list of already used options with the new options, one has to read configure's whole output and compare with what's currently used. On the contrary with cmake, one just runs the existing command, then go to the build dir and run ccmake, and see what's enabled and should not, and report the difference to the build script.

[illwieckz]

windows-amd64-mingw and windows-i686-mingw engine both build and run.

[illwieckz]

For some reasons libpng now provides png.framework on macOS and the engine build looks for it instead of the static library (even with USE_STATIC_LIBS), so I made generic the copy of any *.framework from deps. I don't know if we prefer static linking or dynamic linking on macOS, static linking allows further LTO optimizations so we may prefer that.

[illwieckz]

linux-i686-default and macos-amd64-default engine both build and run.

[illwieckz]

I now squashed many commits and properly named them. I'm now entering the phase where it is ready to review.

I'm gonna produce test archives so the CI can test them.

[illwieckz]

It can also be the opportunity to change some other things. For example we use some dylib on macOS from some prebuilt packages, if we don't have the choice, it's fine. But among libs that are either provided prebuilt or that we compile some may provide both a dylib and a static archive, until now we preferred the dylib for some unknown reasons, but we may prefer to ship the static archive instead if it works: it would give us less files to package in the built engine archive, and maybe it allows more LTO to be done at engine build time.

[illwieckz]

Also I noticed that SDL2 has some audio backends (like pulseaudio), and OpenAL has an SDL2 backend (among others like pulseaudio), so we may check we only build them once. Either we use OpenAL on native audio backend directly and we make sure SDL2 doesn't build any native audio backend, either we use OpenAL on SDL2 and make sure that's the only OpenAL backend to be built and that SDL2 has the required native audio backends being built.

[illwieckz]

As a remind I prefer to use the cmake wrapper when CMake is provided for a library and that actually works, because it makes easier to manage the enablement/disablement of features that come when bumping the library versions.

[slipher]

As a remind I prefer to use the cmake wrapper when CMake is provided for a library and that actually works, because it makes easier to manage the enablement/disablement of features that come when bumping the library versions.

How do you figure it's easier? Traditional configure script workflow:

• ./configure --help to see options
• Add feature-disabling command line options to build.sh

CMake workflow:

• Create build dir
• Run CMake. Configuration must succeed, else not all options may be created.
• ccmake or cmake -L/cmake -LA to see options
• Add feature-disabling command line options to build.sh

ISTM that CMake comes out worse there. Is there a better way to do it?

[slipher]

Is there a better way to do it?

Oh, I see it was posted further up:

When using configure it is hard to compare the list of already used options with the new options, one has to read configure's whole output and compare with what's currently used. On the contrary with cmake, one just runs the existing command, then go to the build dir and run ccmake, and see what's enabled and should not, and report the difference to the build script.

I don't think that checking ON/OFF states is a reliable workflow though. Some new features may be off by default initially, but later changed to on by default. Or it may do the annoying thing where stuff is automatically turned on or of depending on what dependencies are installed.

[slipher]

Rebased following #1629. Resulting build.sh text is the same as the original modulo NASM_BASEURL.

[slipher]

I tried to test builds before/after to see if the changes broke anything. There was a lot of stuff broken but much of it was also broken without the changes. I opened #1642 with fixes for those issues which are present both before and after. That fixes the GLEW Windows error mentioned in the OP.

I got some issues with the macos bash not liking certain constructs:

• The ^^ in the log function. That was a pre-existing bug so I put a fix in More external_deps build fixes #1642.
• Evaluating empty arrays. Luckily we don't have any need for variables with spaces, so arrays don't provide any value and can be replaced with regular variables.
• The ${@} in the configure_build function seemed to cause an error whenever the function got 0 arguments. This one is really weird as $@ seems like a very standard, ancient shell functionality. I hacked around it for testing with ${@:-}, not sure if this is really write (would it add an empty string arg?) but it seemed to work.

[slipher]

I guess the problem is that the old Bash sometimes doesn't distinguished between "undefined" and "empty" variables. Maybe for the configure_build $@ thing we could wrap it in set +u/set -u?

To answer some very old posts from the thread above:

Also I noticed that SDL2 has some audio backends (like pulseaudio), and OpenAL has an SDL2 backend (among others like pulseaudio), so we may check we only build them once. Either we use OpenAL on native audio backend directly and we make sure SDL2 doesn't build any native audio backend, either we use OpenAL on SDL2 and make sure that's the only OpenAL backend to be built and that SDL2 has the required native audio backends being built.

We aren't using SDL audio.

Regarding static vs. dynamic libs, I guess static is better for releases since the linker has the opportunity to remove unused code, reducing the file size. Dynamic may be better for development since then you don't have to link anything so the build could be faster. MSVC libs have to be dynamic because we can't produce static libs in its format. Fortuitously, MSVC is the one platform which is never used for releases, only development. So my conclusion would be static libs are preferable for everything but MSVC.

[slipher]

After applying some fixes for the old Bash compatibility and the fix for pkgconfig from #1642, I successfully built and tested the Mac amd64 deps and client with the latest toolchain and OS on my arm64 Mac machine.