[RFC] Re-add support for macOS #2019
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit introduces logic to ensure that CMAKE_OSX_ARCHITECTURES is always set on Apple targets. It can either be specified on the CLI (i.e., in case of cross-compilation) or left blank (in which case it is set to CMAKE_HOST_SYSTEM_PROCESSOR). This will be used throughout the project to determine for which architecture to compile on Apple targets. Signed-off-by: Philipp Remy <[email protected]>
The previous OptimizeForArchitecture only included proper support for x86(_64). The new implementation supports x86(_64), arm(64), ppc and adds newer CPU families. Furthermore, it adapts OFA to account for cross-compilation on Apple targets. When targeting arm64, it selects apple-m1 as the baseline architecture (making it compatible with all Mac Apple Silicon Chips). When targeting x86_64, it selects skylake as the baseline architecture (making it compatible with all Intel Macs not older than 2015). All of this can be manually overridden by sepcifying TARGET_ARCHITECTURE on the command line. The code is derived/copied from https://gitlab.inria.fr/gismo/gismo, which is licensed under MPL-2. Signed-off-by: Philipp Remy <[email protected]>
Instead of setting CFLAGS or CXXFLAGS manually, simply use the CMake targets OpenMP::OpenMP_C and OpenMP::OpenMP_CXX respectively. This applies to dependencies (MeshSD), nonfree (VLSift) and the AliceVision targets itself. Signed-off-by: Philipp Remy <[email protected]>
This commit adds support for compiling library targets into Apple Frameworks on macOS. Embedding resources into the bundle is supported as well. It is checked early if the user attempts to build a universal binary, which is currently unsuppprted due to missing support in the dependency building code. Signed-off-by: Philipp Remy <[email protected]>
Executable file names are suffixed with their respective versions in the format "MAJOR.MINOR". This is fine as long as the executable (as on Windows) has a proper file extension. On macOS (and other Unix OSes as well), the appended version suffix could be falsely interpreted as a file extension. Change "MAJOR.MINOR" to "MAJOR_MINOR" formatting. Signed-off-by: Philipp Remy <[email protected]>
CUDA has long been removed and even longer been deprecated on Apple platforms. Guard any CUDA functionality behind if(APPLE) statements by default so CUDA won't be used on Apple targets. Signed-off-by: Philipp Remy <[email protected]>
Setting CMP0135 to NEW resolves a bunch of warnings and allows for proper rebuilds if the URL of a dependencies changed. Signed-off-by: Philipp Remy <[email protected]>
This is mainly useful for Apple targets (and therefore only enabled for Apple targets by default) where the system compiler (aka AppleClang) supports OpenMP but does not ship the required library and headers. Signed-off-by: Philipp Remy <[email protected]>
PopSIFT has a hard dependency on CUDA being available. Remove it from the target list if CUDA is disabled by the project (by setting AV_USE_CUDA to OFF). Signed-off-by: Philipp Remy <[email protected]>
…e targets On Apple targets, optimized (=faster) implementations of BLAS/LAPACK and Sparse Solvers (since 2017) are available in Accelerate.framework (a system-provided Framework existent on every machine by default). BLAS and LAPACK aree used by multiple dependencies, each of them being capable of handling Apple's Accelerate.framework. SuiteSparse is *one* possible backend for Sparse Solvers in Ceres, but it can use 'AccelerateSparse' (= Accelerate.framework) as an alternative. During testing, no performance difference was noticable. This relieves Apple targets from needing to have a Fortran compiler installed and reduces final bundle size by ~160MB (GraphBLAS in SuiteSparse is *huge*). Signed-off-by: Philipp Remy <[email protected]>
…res on Apple targets Ceres can utilize the system Accelerate.framework instead of using SuiteSparse to enable fast Sparse Solvers. Therefore, don't require SuiteSparse on Apple platforms. Signed-off-by: Philipp Remy <[email protected]>
…edded Geogram As we can now safely rely on CMAKE_OSX_ARCHITECTURES when determining which architecture to build on Apple targets, use CMAKE_OSX_ARCHITECTURES instead of CMAKE_SYSTEM_PROCESSOR when setting the Vorpaline Platforms. This enables cross-compiling Geogram on Apple targets from x86_64 to arm64 and vice-versa. Signed-off-by: Philipp Remy <[email protected]>
This commit introduces two changes:
(1) It bumps libtiff from v4.5.0 to v4.7.1 (appears to be ABI and API compatible)
(2) It switches to the CMake build system in the libtiff tree.
This change was required as the configure script caused the install_name on Apple targets to be a hardcoded absolute path instead of the recommended @rpath approach, making this library completely unportable. With the new CMake build system @rpath is properly used.
Signed-off-by: Philipp Remy <[email protected]>
This commit introduces two changes: (1) Bumps the version of libPNG from 1.6.39 to 1.6.50 (appears to be ABI and API compatible), as 1.6.39 requires a non-existing fp16.h header on macOS. (2) Sets DPNG_ARM_NEON to ON, if an ARM processor is used. NEON is broadly supported on reasonably modern ARM CPUs. Signed-off-by: Philipp Remy <[email protected]>
Bump version of libVPX from v1.13.0 to v1.15.2 (appears to be ABI and API compatible), because v1.13.0 uses Linux specific linker flags on Apple targets. Signed-off-by: Philipp Remy <[email protected]>
Bump ffmpeg version from v5.1.2 to v7.1.1. This should be fine, as OpenCV and OpenImageIO appear to be the only dependants of ffmpeg, with both having support for ffmpeg 7.X. This fixes an issue on x86_64 Linux where inline assembly failed to compile due to wrong syntax (at least with GCC 13). Signed-off-by: Philipp Remy <[email protected]>
…SX Architecture This patch automatically selects the correct ONNXRuntime binary, based on CMAKE_OSX_ARCHITECTURES (which is used throughout the whole CMake infrastructure) and allows for cross-compiling macOS for x86_64 and vice-versa. Signed-off-by: Philipp Remy <[email protected]>
Currently, pkgconfig only looks in */lib64/* directories, causing FLANN to not find lz4 on Apple targets. This might work on Linux, but on Apple platforms the differentiation between lib64 and lib does not existby default. This commit allows pkgconfig to look in */lib/* as well, as this is the default installation directory on macOS. The path is prepended, so on Linux targets */lib64/* should still take precedence. Signed-off-by: Philipp Remy <[email protected]>
XercesC is a required hard dependency of libE57Format and it cannot build as an embedded project. Allow the user to rely on building a XercesC library as a dependency. Because it is *only* required by libE57Format, make it a cmake_dependent_option() based on whether libE57Format should be build. Signed-off-by: Philipp Remy <[email protected]>
This commit introduces two changes: (1) Bump version from v3.1.1 to v3.2.0 (appears to be ABI and API compatible). v3.1.1 caused compilation errors on Apple targets because of missing documentation paragraphs. The project internally sets -Wall and -Werror, so these become compilation errors. (2) Explicitly tell libE57Format to install its CMake configuration files to <INSTALL_DIR>/share/E57Format. These files are not installed to the standard location by using GNUInstallDirs. Signed-off-by: Philipp Remy <[email protected]>
Bump version of pcl from v1.13.0 to v1.15.1 (appears to be API and ABI compatible). This fixes compilation errors on Apple targets. Signed-off-by: Philipp Remy <[email protected]>
…cutable
This commit introduces two changes:
(1) Bump version of pxr from v23.05 to v25.08. This fixes compilation errors with newer Boost versions on macOS.
(2) Explicitly set the Python executable to "python3" (as system-installed on macOS), and keep "python" on other targets.
Signed-off-by: Philipp Remy <[email protected]>
This commit introduces the possibility to build an embedded PCRE2 library. This is required by SWIG, so only build it if SWIG should be build itself. Signed-off-by: Philipp Remy <[email protected]>
As we currently set -DUSE_PYTHON=ON when building OpenImageIO, it has a hard dependency on pybind11. Building this currently defaults to OFF. If no external pybind11 is found, this will cause a CMake error. As the project should be able to be build without any external dependencies in its standard configuration, build an embedded pybind11 by default. Signed-off-by: Philipp Remy <[email protected]>
…X to be @rpath If ALICEVISION_USE_RPATH is set, tell libVPX to set its install name directory to @rpath using environment LDFLAGS, as the autoconf script does not have a native switch for that. Signed-off-by: Philipp Remy <[email protected]>
This commit fixes a build logic issue in libVPX, where the configure script is only able to detect Darwin up to version 24.X.X (macOS Sonoma). macOS 26 Tahoe is Darwin 25.X.X, causing the configure script to fall back to generic-gnu, which assumes a Linux host OS. As a workaround in CMake, we detect (using CMAKE_SYSTEM_VERSION) if the host Darwin is > 2. In that case, we explicitly set the newest available toolchain for libVPX, which logically is [x86_64/arm64]-darwin24-gcc. To be removed when libVPX either changes its detection logic or a newer ABI- and API-compatible version with proper Darwin > 24 support is released. Signed-off-by: Philipp Remy <[email protected]>
…rpath If ALICEVISION_USE_RPATH is set, explicitly set the install name directory for the ffmpeg libraries to @rpath using the available b2 CLI argument. Signed-off-by: Philipp Remy <[email protected]>
As for other dependencies, do not build the tests for Ceres (as they are not needed and not tested). Besides from saving compilation time, this allows for cross-compiling Ceres on macOS. Otherwise, Ceres will build an embedded glog but does not pass through CMAKE_OSX_ARCHITECTURES, causing linking errors because of mismatched architectures. Signed-off-by: Philipp Remy <[email protected]>
If the user explicitly disables rpath handling when building AliceVision by setting ALICEVISION_USE_RPATH=OFF, respect this in the embedded dependencies. This mainly fixes some older dependencies, where this is not set to ON by default and therefore results in a library with absolute install names (which drastically reduces portability of the resulting binary). Signed-off-by: Philipp Remy <[email protected]>
…CORE_BUILD_FLAGS Some dependencies were missing the said flags. This ensures a predictable build with matching options for all dependencies. FIXME: LEMON currently does not support C++20. Therefore, CMAKE_CORE_BUILD_FLAGS is not applied to LEMON. Signed-off-by: Philipp Remy <[email protected]>
Explicitly pass through top-level CMake variables when building AliceVision as an external project in Dependencies.cmake. This change allows for the following configure step to respect any choices the user made at the top level. FIXME: We should ideally only have one place to set the options - the very top-level CMakeLists.txt. But that would be a breaking change, as we would need to remove all "AV_" prefixed variables and replace them with the src/CMakeLists.txt "ALICEVISION_" prefix style. Signed-off-by: Philipp Remy <[email protected]>
aliceVision_selectConnectedViews linked to aliceVision_depthMap although it does not use the library. This caused a linker issue on macOS, as the target was not guarded behind a ALICEVISION_HAVE_CUDA check. Remove the unnecessary link completely. Signed-off-by: Philipp Remy <[email protected]>
Adapt the install rpaths on Apple targets to accommodate for the typical library paths. This includes the standard Unix paths (lib, next-to-loading-binary) and the Apple bundle structures (Libraries, Frameworks). Add every possibility for @loader_path (for dynamic libraries) and @executable_path (for executables) to ensure proper lookup based on the context. Signed-off-by: Philipp Remy <[email protected]>
This commit introduces several changes to the general build infrastructure. Given CMAKE_OSX_ARCHITECTURES, it is now possible to cross-compile the whole project and its dependencies from x86_64 to aarch64 and vice versa on macOS. This is easily achievable because Apple can compile for both architectures with the same sysroot. Choosing the correct flags depends on the underlying build system:
(1) CMake: In CMake, one can just set CMAKE_OSX_ARCHITECTURES to either arm64 or x86_86.
(2) Autoconf/Automake: Usually involves setting the --host flag and passing additional CFLAGS, CXXFLAGS and LDFLAGS (with "-arch x86_64/arm64").
(3) libVPX: A target must be specified using the --target flag (includes the major Darwin version number).
(4) Boost: The architecture for b2 must be set using the "architecture=X" flag - additionally pass CFLAGS, CXXFLAGS and LDFLAGS (with "-arch x86_64/arm64").
(5) ffmpeg: Must pass the additional CFLAGS, CXXFLAGS and LDFLAGS (with "-arch x86_64/arm64") and the --arch (with x86_64 or aarch64(!)), --enable-cross-compile and --sysroot (xcrun --sdk macosx --show-sdk-path) flags.
Signed-off-by: Philipp Remy <[email protected]>
…pple targets where necessary As it might be desirable to later change the embedded rpaths of a binary, "install_name_tool" expects enough padding between the header and the first data section of a Mach-O file. This caused issues when changing the rpaths of the following dependencies, which added little to no padding to the Mach-O header: - libtiff - ffmpeg Pass "-headerpad_max_install_names" explicitly as linker flags to these dependencies when building them on Apple platforms. This adds the maximum amount of padding to the header (per ld man page: "Automatically adds space for future expansion of load commands such that all paths could expand to MAXPATHLEN."). Signed-off-by: Philipp Remy <[email protected]>
As we do not build Ceres with SuiteSparse support (Ceres can utilize Accelerate.framework), do not ask for SuiteSparse on Apple platforms. As an equivalent, make sure to find a Ceres with AccelerateSparse (can be disabled in the same manner as Ceres with SuiteSparse). Signed-off-by: Philipp Remy <[email protected]>
…ries for AliceVision Currently, the name is set by CMAKE_SYSTEM_PROCESSOR. However, this does not always match the target architecture, as users can cross-compile on Apple targets. On Apple, use CMAKE_OSX_ARCHITECTURES instead. Signed-off-by: Philipp Remy <[email protected]>
This commit adds a Python script (darwin_bundle.py) which takes several Mach-O files (or Framework folders) as input and attempts to resolve all required dependencies to create a standalone (i.e, self-contained) bundle. A CMake target for use after the AliceVision build is added to automatically invoke the script with all targets of the project (can be invoked with make darwin-bundle). This works somewhat similar to what the regular bundle target does, but the CMake implementation did not appear to work for Apple targets (out of the box). Signed-off-by: Philipp Remy <[email protected]>
Due to the additional requirements and the unique build process of this platform (cross-compiling, environment, ...) move the build instructions to a custom file. This commit also introduces a list of available target architectures in the OFA system, available at src/cmake/OFA/SupportedArchitectures.md. Signed-off-by: Philipp Remy <[email protected]>
This commit introduces two changes:
(1) It correctly sets the SWIG executable for Apple targets.
(2) It adapts to the Apple linker (ld), which does not allow undefined symbols in shared objects (.so) by default.
Regarding (2), see SWIG issue 2469.
Signed-off-by: Philipp Remy <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
[WIP] Project support for macOS
Well, I decided that this might be ready for an RFC now. It’s still a draft because I don’t yet have a solid solution for compiling the Boost libraries on macOS reliably — currently, only AppleClang is supported due to issues with the Boost CMake build system and an LLVM upstream issue. Once that is solved, there are no outstanding no-gos from my side.
I know that these look like some heavy changes, but I believe that I did not break any Windows/Linux infrastructure. I tried my best to do CI on both Windows and Linux, and at least with the official GitHub CI, they appeared to work. Compiling embedded dependencies on Linux works as expected, if you don't attempt to build an embedded Assimp - but that issue was already present before (on recent GCCs, one header is missing a
#include <cstdint>, which is a one-line fix).Overview of changes
Each commit has its own description on what changed and why that change is required in my opinion. But as a quick overview:
x86_64was properly supported, now there is very solidarm/arm64, expandedx86_64and rudimentaryppcsupport./src). Now the user can overwrite all variables properly by passing them on the CMake CLI (backwards-comparible).Accelerate.framework, which Ceres can use instead of Suitesparse. This lifts the requirement for a Fortran compiler on macOS and drastically reduces compilation times and bundle size. When running the unit tests I did not notice any performance differences.-arch <ARCH>to the compiler, the project fully supports cross-compiling fromarm64tox86_64on macOS - including all required dependencies. The target architecture can be set by specifyingCMAKE_OSX_ARCHITECTURES=<arm64|x86_64>on the CMake CLI. Note that compiling universal binaries is not supported at this point, because some dependencies do not have the required logic to do this in one step.INSTALL_macOS.mdprovides specific instructions on how to compile this project successfully on macOS.AppleClangis now supported - this requires that OpenMP is installed as an external dependency (enabled by default on macOS), as the Apple SDK does not ship it. Furthermore, the project now uses the unified CMake targetsOpenMP::OpenMP_CandOpenMP::OpenMP_CXXrespectively instead of setting the required flags manually.[TBA] Open questions
Drawbacks
Because AliceVision has so many dependencies - some with very specific configurations - I think we cannot officially support building the project with package-manager provided dependencies. I just stumbled from roadblock to roadblock when attempting to standardize the required dependencies for the different package managers. So currently the only option is to build the project with
ALICEVISION_BUILD_DEPENDENCIES=ON. However, this matches the recommendation for Linux, so I think this should be fine. If someone wants to attempt to build it with Homebrew/MacPorts/Nix anyway, that might or might not work. But at least we don't have the obligation to ensure compatibility for these cases.There is no CI yet. I decided against CI, because it currently requires to rebuild all dependencies over and over again - the GitHub runners usually only have 3 threads available, so this is quite slow. I tested pre-built dependencies, but that currently does not work because there are some dependencies hard-code absolute paths into their CMake config modules - making them not portable.
There are no unit tests yet. I ran the unit tests on my Mac, and two are currently failing:
acRansac_test.cpp: I think there is some undefined behavior here, which caused a segfault on macOS. See this gist for a detailed description and proposed solution.fundamental10PSolver_test.cpp: The check on line 184 fails. I have no idea why and I don't know anywhere near enough to solve this. Help is much appreciated. For the results see this gist.Future plans
This is the first step to make macOS a fully supported platform for this project again. The next steps would be to add the Apple Metal DepthMap backend and maybe generally abstract the DepthMap library to allow for multiple backends with a unified interface.
Also: Add a proper CI solution and ensure that all unit tests pass.
Feedback is appreciated!
These patches are the result of weeks of debugging, investigating and rebasing - but I am sure that there are still issues and edge cases that I completely missed. So fresh viewpoints and remarks are much appreciated. If anyone has the chance to test this out, please do so and let me know if there are any issues.