Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Linux 2025 #8255

Open
wants to merge 70 commits into
base: master
Choose a base branch
from
Open

Linux 2025 #8255

wants to merge 70 commits into from

Conversation

danoli3
Copy link
Member

@danoli3 danoli3 commented Jan 15, 2025

Linux

  • Updates for GCC14 / GCC13 / GCC12 / GCC11 / GCC10 (from GCC6 std=c++11)
  • C++ now variant to C++23 depending GCC version. 24.04 = GCC14
  • Download Script now updated to download correct GCC based on whats installed.
  • Performance Boost for Linux apps 400% for correct GCC
  • Split Linux64 to -TARGET = Linux - A = 64
  • Split allows for all versions of Linux to fall into correct locations (arm64 etc)
  • Updated Workflow for updated Linux Libraries
  • Fixed Nightly borked

@danoli3
Copy link
Member Author

danoli3 commented Jan 15, 2025

Removed Gold LD for the moment regarding:
#8239

@danoli3 danoli3 added the linux label Jan 15, 2025
@danoli3 danoli3 added this to the 0.12.1 milestone Jan 15, 2025
@danoli3
Copy link
Member Author

danoli3 commented Jan 15, 2025

Makefile linux:

fi
building with makefiles
**** Building OF core ****
  HOST_OS=Linux
  PLATFORM_ARCH=x86_64
  PLATFORM_OS=Linux
  HOST_ARCH=x86_64
  HOST_OS=Linux
  CROSS_COMPILING=0
  PLATFORM_VARIANT=default
  IS_RASPBIAN=
  =================== config.mk platform detection ================
  PLATFORM_ARCH=x86_64
  PLATFORM_OS=Linux
  PLATFORM_VARIANT=default
  PLATFORM_LIB_SUBPATH=linux/64
  =================== config.mk paths =============================
  OF_ADDONS_PATH=../../../addons
  OF_EXAMPLES_PATH=../../../examples
  OF_APPS_PATH=../../../apps
  OF_LIBS_PATH=../../../libs
  OF_LIBS_OPENFRAMEWORKS_PATH=../../../libs/openFrameworks
  OF_LIBS_OF_COMPILED_PATH=../../../libs/openFrameworksCompiled
  OF_LIBS_OF_COMPILED_PROJECT_PATH=../../../libs/openFrameworksCompiled/project
  OF_SHARED_MAKEFILES_PATH=./makefileCommon
  OF_PLATFORM_MAKEFILES=../../../libs/openFrameworksCompiled/project/linux/64
  OF_CORE_LIB_PATH=../../../libs/openFrameworksCompiled/lib/linux/64
  makefileCommon/config.shared.mk:259: *** recipe commences before first target.  Stop.
  Error: Process completed with exit code 2.

Really need to get the cmake stuff in soon.

I'd re-added gold ld again with command check.

  • todo Add to install libs setup

@danoli3
Copy link
Member Author

danoli3 commented Jan 15, 2025

Latest libraries now has Linux arm64:
https://github.com/openframeworks/apothecary/releases/tag/latest

Linux 64

@danoli3
Copy link
Member Author

danoli3 commented Jan 16, 2025

Getting action download info
Error: This request has been automatically failed because it uses a deprecated version of `actions/upload-artifact: v3`. Learn more: https://github.blog/changelog/2024-04-16-deprecation-notice-v3-of-the-artifact-actions/

image

Weird. Disabling caches / updating checkout to 4.2.2
Added workflow depend on linux for others (its fastest and saves cpu - once its working)

@danoli3
Copy link
Member Author

danoli3 commented Jan 16, 2025

Linux empty example is compiling just not addonsExample

@ofTheo
Copy link
Member

ofTheo commented Jan 16, 2025

Looks like PLATFORM_CORE_EXLUSIONS is not defined in config.shared.mk

image

@danoli3
Copy link
Member Author

danoli3 commented Jan 16, 2025

@ofTheo yeah the library subpath was being used for the targetarch name in for the makefile in the directory! So with linux/64 it was bugging out. I thought it was line spacing tab stuff.

Makefiles are so sensitive though wow. It looks like maybe the streamer issue just reported is a spaces instead of tabs issue if they had run the code.

  ccache g++ -o bin/linux64_debug  obj/linux/64/Debug/src/ofApp.o obj/linux/64/Debug/src/main.o   ../../../libs/openFrameworksCompiled/lib/linux/64/libopenFrameworksDebug.a  -Wl,-rpath=./libs:./bin/libs:./ -Wl,--as-needed -Wl,--gc-sections -fuse-ld=gold ../../../libs/brotli/lib/linux/64/libbrotlienc.a ../../../libs/brotli/lib/linux/64/libbrotlicommon.a ../../../libs/brotli/lib/linux/64/libbrotlidec.a ../../../libs/fmt/lib/linux/64/libfmt.a ../../../libs/glew/lib/linux/64/libGLEW.a ../../../libs/glfw/lib/linux/64/libglfw3.a ../../../libs/tess2/lib/linux/64/libtess2.a ../../../libs/zlib/lib/linux/64/zlib.a -lgstapp-1.0 -lgstvideo-1.0 -lgstbase-1.0 -lgstreamer-1.0 -ludev -lfontconfig -lfreetype -lsndfile -lopenal -lssl -lcrypto -lcurl -lrtaudio -lpulse-simple -lpulse -pthread -lasound -L/usr/lib64 -lGLEW -lGL -lX11 -lGLU -lgtk-3 -lgdk-3 -lz -lpangocairo-1.0 -lpango-1.0 -lharfbuzz -latk-1.0 -lcairo-gobject -lcairo -lgdk_pixbuf-2.0 -lgio-2.0 -lgobject-2.0 -lglib-2.0 -lmpg123  -lglut -lX11 -lXrandr -lXxf86vm -lXi -lXcursor -ldl -lpthread -lfreeimage -lpugixml -luriparser
  ../../../libs/openFrameworks/sound/ofOpenALSoundPlayer.cpp:1009: error: undefined reference to 'kiss_fftr_alloc'
  ../../../libs/openFrameworks/sound/ofOpenALSoundPlayer.cpp:1020: error: undefined reference to 'kiss_fftr_alloc'
  ../../../libs/openFrameworks/sound/ofOpenALSoundPlayer.cpp:1065: error: undefined reference to 'kiss_fftr'
  ../../../libs/openFrameworks/sound/ofOpenALSoundPlayer.cpp:1095: error: undefined reference to 'kiss_fftr'
  collect2: error: ld returned 1 exit status

Linking Kiss is missing! Sweet looking into

@ofTheo
Copy link
Member

ofTheo commented Jan 16, 2025 via email

@danoli3
Copy link
Member Author

danoli3 commented Jan 17, 2025

Complete

Ubuntu
24.04.1
LT
Build Success all GCC targets

Ubuntu
22.04.1
LT
Build Success GCC10 (max GCC for 22.04)

@danoli3
Copy link
Member Author

danoli3 commented Jan 17, 2025

Added a utility pipe to apothecary and just did the same for oF
image

this will make everything a lot easier to update before the gui stuff is done
./of or ./of.sh
usage following our function names: setup, update
of setup runs any setup scripts if applicable, apt gets etc
of update runs any download libs scripts

currently just targeting auto the platform that it is.
macOS will call default macos
Windows will call the default scripts for vs

This will drop clicks / folder clicks and bash running into 1 open repo in terminal: boom run command

@@ -80,7 +80,7 @@ linux:
ADDON_LIBS_EXCLUDE = libs/assimp
ADDON_INCLUDES_EXCLUDE = libs/assimp/%

linux64:
linux/64:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will the old linux64 and linuxarmv6l still work for PG etc?

@@ -68,21 +68,21 @@ ios:
ADDON_LIBS = libs/svgtiny/lib/ios/svgtiny.a
ADDON_LIBS += libs/libxml2/lib/ios/xml2.a

linux64:
ADDON_LIBS = libs/svgtiny/lib/linux64/libsvgtiny.a
ADDON_LIBS += libs/libxml2/lib/linux64/libxml2.a
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Likewise will libs still get picked up in older addons from lib/linux64

endif

# if not defined, construct the default PLATFORM_LIB_SUBPATH
ifndef PLATFORM_LIB_SUBPATH
# determine from the arch
ifeq ($(PLATFORM_OS),Linux)
ifeq ($(PLATFORM_ARCH),x86_64)
PLATFORM_LIB_SUBPATH=linux64
PLATFORM_LIB_SUBPATH=linux/64
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks like older projects with addons in lib/linux64/ might not work based on this?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes this does break older addons, something like this is required

# Separate target and arch with a slash in addon_config.mk (linux64 -> linux/64) 
find $OF_ROOT/addons -name addon_config.mk -exec sed -i 's/\(linux\)\([a-zA-Z0-9]\+\):/\1\/\2:/g' {} \;

@danoli3
Copy link
Member Author

danoli3 commented Jan 18, 2025

Good idea although make file location is a very minimal issue compared to all the std:: upgrades required for older projects... but I've added the beginning on an auto-upgrade script.

this script will automatically replace issue strings in addon_config.mk for Linux.

Called via:
./of upgrade addons
./of upgrade apps

In the future this can also add in std:: replacements. Just requires stop and prompt by user since they need to back up files.
Backup mk files are created and removed only upon user input

@ofTheo
Copy link
Member

ofTheo commented Jan 18, 2025

Upgrade script are always handy, but I think we have to be able to support addons that would be fully compliant up until this PR is merged.

I know its a pain to support two names, but I'd either suggest leaving the current system of linux64, linuxarmv6l or, making sure both linux/64 and linux64 libs get picked up by make files / PG etc.

This feels like the amount it will break, is not worth the conceptual tidiness of having the subfolder approach.

Curious what others think, but I am def nervous as it's the osx / macOS issue again, but with even more platforms.

@danoli3
Copy link
Member Author

danoli3 commented Jan 18, 2025

Actually all those issues were caused by explicit makefile definitions like this one, when we coded not to at all use them! thanks for reminder ~ these are redundant now. Even the exclude code should be removed and just use actual platform defines.

In fact lets just remove these completely as the projectGenerator figures this out automatically.

Here's a test fully removing these .a explicit:
image

Xcode:
image
Make:
image

Let me test this on native linux see if it just works without this

@dimitre
Copy link
Member

dimitre commented Jan 18, 2025

I think the optional subfolders will complicate the regex needed by makefiles.
The best is having only one folder depth. If any improvement / disambiguating is needed I suggest using a target triplet as a single folder

@ofTheo
Copy link
Member

ofTheo commented Feb 26, 2025

A couple of questions for this PR to get merged:

  • can we avoid subfolders ( just dealt with issues in emscripten due to this )
  • can we avoid naming the gcc version ie: ( and just have lib/linux64/myLib.a )
  • can we pick a minimum gcc version for libs? ie: gcc10? gcc6 libs have been stable across later platforms.
  • we don't need i386 - 0.12.1 is 64bit only

I saw you mentioned:

Performance Boost for Linux apps 400% for correct GCC

How was that benchmarked?

@danoli3
Copy link
Member Author

danoli3 commented Feb 27, 2025

A couple of questions for this PR to get merged:

  • can we avoid subfolders ( just dealt with issues in emscripten due to this )
  • can we avoid naming the gcc version ie: ( and just have lib/linux64/myLib.a )
  • can we pick a minimum gcc version for libs? ie: gcc10? gcc6 libs have been stable across later platforms.
  • we don't need i386 - 0.12.1 is 64bit only

I saw you mentioned:

Performance Boost for Linux apps 400% for correct GCC

How was that benchmarked?

subfolders are required - linux has two targets aarch64 and 64 by default the developer needs to build for these now. ARM64 is the standard going forward. i.e. all Macs running parallels/vm run arm64 linux

Not having the subfolders caused the issue as historically we always had subfolders for platforms

  • iOS/armv7/
  • android/armv7

For GCC version, if you are thinking for nightly you can set to GCC13 or a minimal one and set download for that.

We can restrict GCC deploys to the majors though:

  • Ubuntu 20.04 LTS: Default GCC 10.
  • Ubuntu 22.04 LTS: Default GCC 11.
  • Ubuntu 24.04 LTS: Default GCC 13 (e.g., gcc-13 13.2.0-23ubuntu4).
  • Ubuntu latest: GCC 14

Bench marks of 400% maybe inflated but based on analysis of some benchmarks of GCC6 vs GCC11 -> 40% increase with a 5-10% optimisation per version. a closer analysis suggests about a 100% increase in performance for x64. 400% for ARM64 linux vs running in EMU mode x64 binaries, however that is actually 800%

Evidence-Based Estimate
Phoronix (GCC 7 vs. GCC 11, April 2021):
Across ~40 benchmarks, GCC 11 was ~20-30% faster than GCC 7 on x86_64 (e.g., FLAC encoding, ray tracing). That’s ~5-7.5% per version over 4 steps.
SPEC CPU 2017 (GCC 6 vs. GCC 10, Intel Data, 2020):
GCC 10 improved SPECint by 25-35% over GCC 6 (4 steps, ~6-8% per version) on x86_64, higher with PGO (40%).
ARM64 Context (GCC 9 to 14):
From GCC 9 to 14 (5 steps), ARM64-specific gains (e.g., SVE, NEON) suggest 40-60% total improvement (8-12% per version), per my prior GCC 14 analysis.
Extrapolating:
GCC 6 to 14 (x86_64): 40-60% total (5-7% per version), based on SPEC and Phoronix trends extended from GCC 11.
GCC 6 to 14 (ARM64): 50-80% total (6-10% per version), due to stronger ARM64 focus in later releases.
Conclusion: GCC 14 vs. GCC 6
Not 10-20% Per Version: A 10-20% gain per release would imply 114-306% total improvement, which exceeds observed data. Real gains are more modest and workload-dependent.
Actual Improvement:
x86_64: GCC 14 is 40-60% faster than GCC 6 (~5-7% per version). Peaks at ~80% in vectorized code.
ARM64: GCC 14 is 50-80% faster than GCC 6 (~6-10% per version), up to 100% in optimized cases (e.g., SVE-enabled loops).
Why Lower Than 10-20%?: Gains diminish as compilers mature; GCC 6 was already decently optimized, and later versions refine rather than revolutionize codegen.

However for arm64 with SIMD / Neon this is indeed about a 400% increase with GCC

Emulation Overhead
QEMU Emulation:
Running aarch64 binaries on x86_64 via QEMU (user-mode emulation) incurs significant overhead. Studies (e.g., Linaro, 2020) show QEMU emulation slows execution by 2x to 10x (100-900% slower) compared to native, depending on workload:
Compute-heavy (e.g., math): ~2-3x slower.
I/O or syscall-heavy: ~5-10x slower.
For C++ with GCC 14, assume a 3x slowdown (200%) as a median estimate for mixed workloads.
Native ARM64 Baseline:
A modern ARM64 CPU (e.g., Graviton 3, Apple M1) running GCC 14 aarch64 binaries is already competitive with x86_64. Some benchmarks (e.g., Phoronix, Graviton 3 vs. Zen 4) show ARM64 within 10-20% of x86_64 performance, or better in power-efficient tasks.
GCC 14’s Role
GCC 14 on x86_64 (emulated aarch64) doesn’t benefit from aarch64-specific optimizations, as the binary targets x86_64. The emulation layer (QEMU) interprets aarch64 instructions, negating hardware advantages.
Native ARM64 GCC 14 binaries leverage SVE, NEON, and fused instructions, unavailable in emulation.
Estimated Speed Difference
Native ARM64 (GCC 14): Baseline performance.
x86_64 Emulated (GCC 14): ~3x slower (200% overhead) due to QEMU.
Relative Speed Increase: Native ARM64 is 200-300% faster (3-4x) than emulated x86_64 for typical C++ workloads. For compute-intensive code (e.g., matrix ops), it could be 150-200% faster (2.5-3x), as emulation overhead is lower

Basically newer GCC's are closer to Clang performance but we should look at a Clang compiler for Linux for 13 as well after CMake toolchains implemented. Not going to do a clang makefile system changes

@danoli3
Copy link
Member Author

danoli3 commented Feb 27, 2025

Or we can just force apt install of GCC14 for the user ? seems like its worth it and only ship GCC14

@ofTheo
Copy link
Member

ofTheo commented Feb 27, 2025

@danoli3 in terms of the GCC stuff, one of our goals is to reduce our overall project complexity so I think simplifying things down to one GCC or maybe two GCCs would be the way to go.

We could have the base GCC eg GCC 10 and then latest which could be 14 or whatever is latest.

Forcing latest wouldn't be a good approach as 14 is not easily available on 22.04 and installing newer GCC on older machines is non-trivial.

@danoli3
Copy link
Member Author

danoli3 commented Mar 3, 2025

Linux 64 also working. Addons probably as broken as the other platforms due to projectGenerator.
image

Rebaseing was a nightmare, shall merge this very soon.

@danoli3
Copy link
Member Author

danoli3 commented Mar 4, 2025

adding setupGCC.sh testing compiling it if no package manager - 140mb download takes like 15 mins to compile on default GCC if not higher installed. by default it will use package one so instant. more a backup.
installed after compile GCC is 1.7gb so not worth packaging ourselves
image

@danoli3
Copy link
Member Author

danoli3 commented Mar 4, 2025

GCC will automatically be at 10 (lowest) unless installed manually by developer where it detects higher GCC
image

For nightly this means it will download GCC10 as we will set it to. no seperate folders, ensuring compatibility

@danoli3
Copy link
Member Author

danoli3 commented Mar 4, 2025

this is the same idea as MSVC 2019 and MSVC 2022 as different GCC's link to different libstdc++ sub versions and compatibility across. It should be tested at later lengths

archlinux is at gcc14

danoli3 added 5 commits March 4, 2025 18:28
* commit '5630d2034d835e81225a225355e97f4144b21179':
  Restore target build dir behavior for Xcode. Closes openframeworks#8321 (openframeworks#8340)
  of script updates (openframeworks#8339)
  _OBJC_CLASS_$_AVFoundationVideoPlayer fix (openframeworks#8337)

# Conflicts:
#	scripts/of.sh
@ofTheo
Copy link
Member

ofTheo commented Mar 6, 2025

suggest not merging until after 0.12.1
lot of changes needed in PG front and backend to support the new platform format

as soon as we get 0.12.1 out we can merge and then make sure new PG tools all work well with this.

@ofTheo ofTheo modified the milestones: 0.12.1, 0.13.0 Mar 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

4 participants