Incorrect linking of arm64 library with x86_64 compilation

[cimes-isi]

Hi Pieter. We ran into an interesting corner case running on an Apple M2 CPU running Anaconda installed for x86_64 (which I suppose macOS then emulates for automatically). The most correct solution to the problem is simply to install Anaconda for arm64, then everything works as expected. I thought we should report the experience anyway, in case you think there is something that can/should be done in py-build-cmake for it, though I'm not requesting that you do so.

In short, py-build-cmake appears to have cmake compile for x86-64 but cmake still tries to link against a dependent library (libyaml-cpp) that is built for arm64. My layman's guess is that py-build-cmake is picking up the x86_64 arch from the conda execution environment (e.g., via distlib?), but the cross-compilation configuration is somehow incomplete.

I'm not entirely sure what the correct thing to do should be, but I don't think it should get into a situation where it's trying to link libraries of the wrong architecture. Perhaps at least one of:

1. The arm64 library path should not be searched.
2. The arm64 library itself should not "found" by cmake even if the path is searched.
3. The native code should be compiled as arm64 rather than x86-64.

Obtaining file:///Users/haonanwa/Projects/PipeEdge
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Preparing editable metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [200 lines of output]
      CMake Warning (dev) at /opt/homebrew/Cellar/cmake/3.27.0/share/cmake/Modules/GNUInstallDirs.cmake:243 (message):
        Unable to determine default CMAKE_INSTALL_LIBDIR directory because no
        target architecture is known.  Please enable at least one language before
        including GNUInstallDirs.
      Call Stack (most recent call first):
        CMakeLists.txt:3 (include)
      This warning is for project developers.  Use -Wno-dev to suppress it.
      
      -- Checking for py-build-cmake environment - found
      --   Using PEP 427-compatible install paths
      -- The CXX compiler identification is AppleClang 14.0.3.14030022
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      -- Configuring done (0.4s)
      -- Generating done (0.0s)
      CMake Warning:
        Manually-specified variables were not used by the project:
      
          PY_BUILD_CMAKE_PACKAGE_NAME
          Python3_EXECUTABLE
          Python3_FIND_REGISTRY
          Python3_FIND_STRATEGY
          Python3_ROOT_DIR
      
      
      -- Build files have been written to: /Users/haonanwa/Projects/PipeEdge/.py-build-cmake_cache/cp37-cp37m-macosx_10_9_x86_64
      [ 66%] Building CXX object CMakeFiles/sched-pipeline.dir/sched-pipeline.cpp.o
      [ 66%] Building CXX object CMakeFiles/sched-pipeline.dir/schedule.cpp.o
      /Users/haonanwa/Projects/PipeEdge/src-native/schedule.cpp:200:10: warning: variable 'stage_num' set but not used [-Wunused-but-set-variable]
        size_t stage_num = 0;
               ^
      1 warning generated.
      [100%] Linking CXX executable sched-pipeline
      ld: warning: ignoring file /opt/homebrew/lib/libyaml-cpp.0.7.0.dylib, building for macOS-x86_64 but attempting to link with file built for macOS-arm64
      Undefined symbols for architecture x86_64:
        "YAML::InvalidNode::~InvalidNode()", referenced from:
            _main in sched-pipeline.cpp.o
        NOTE: a missing vtable usually means the first non-inline virtual member function has no definition.
        "vtable for YAML::Exception", referenced from:
            YAML::Exception::Exception(YAML::Mark const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) in sched-pipeline.cpp.o
        NOTE: a missing vtable usually means the first non-inline virtual member function has no definition.
      ld: symbol(s) not found for architecture x86_64
      clang: error: linker command failed with exit code 1 (use -v to see invocation)
      make[2]: *** [sched-pipeline] Error 1
      make[1]: *** [CMakeFiles/sched-pipeline.dir/all] Error 2
      make: *** [all] Error 2
      Traceback (most recent call last):
        File "/Users/haonanwa/opt/anaconda3/envs/pipeedge/lib/python3.7/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 167, in prepare_metadata_for_build_editable
          hook = backend.prepare_metadata_for_build_editable
      AttributeError: module 'py_build_cmake.build' has no attribute 'prepare_metadata_for_build_editable'
      
      During handling of the above exception, another exception occurred:
      
      Traceback (most recent call last):
        File "/Users/haonanwa/opt/anaconda3/envs/pipeedge/lib/python3.7/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/Users/haonanwa/opt/anaconda3/envs/pipeedge/lib/python3.7/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/Users/haonanwa/opt/anaconda3/envs/pipeedge/lib/python3.7/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 176, in prepare_metadata_for_build_editable
          whl_basename = build_hook(metadata_directory, config_settings)
        File "/private/var/folders/th/npccwh1j0hj106lcytk5m_4c0000gn/T/pip-build-env-zw52vokl/overlay/lib/python3.7/site-packages/py_build_cmake/build.py", line 86, in build_editable
          editable=True)
        File "/private/var/folders/th/npccwh1j0hj106lcytk5m_4c0000gn/T/pip-build-env-zw52vokl/overlay/lib/python3.7/site-packages/py_build_cmake/build.py", line 248, in build_wheel_in_dir
          self.do_native_cross_cmake_build(paths, cfg, pkg_info)
        File "/private/var/folders/th/npccwh1j0hj106lcytk5m_4c0000gn/T/pip-build-env-zw52vokl/overlay/lib/python3.7/site-packages/py_build_cmake/build.py", line 334, in do_native_cross_cmake_build
          cfg.cross, package_info, native_install_dir)
        File "/private/var/folders/th/npccwh1j0hj106lcytk5m_4c0000gn/T/pip-build-env-zw52vokl/overlay/lib/python3.7/site-packages/py_build_cmake/build.py", line 511, in run_cmake
          cmaker.build()
        File "/private/var/folders/th/npccwh1j0hj106lcytk5m_4c0000gn/T/pip-build-env-zw52vokl/overlay/lib/python3.7/site-packages/py_build_cmake/cmake.py", line 190, in build
          self.run(cmd, cwd=cwd, check=True, env=env)
        File "/private/var/folders/th/npccwh1j0hj106lcytk5m_4c0000gn/T/pip-build-env-zw52vokl/overlay/lib/python3.7/site-packages/py_build_cmake/cmake.py", line 68, in run
          return self.runner.run(*args, **kwargs)
        File "/private/var/folders/th/npccwh1j0hj106lcytk5m_4c0000gn/T/pip-build-env-zw52vokl/overlay/lib/python3.7/site-packages/py_build_cmake/cmd_runner.py", line 25, in run
          return sp_run(*args, **kwargs)
        File "/Users/haonanwa/opt/anaconda3/envs/pipeedge/lib/python3.7/subprocess.py", line 512, in run
          output=stdout, stderr=stderr)
      subprocess.CalledProcessError: Command '['cmake', '--build', '/Users/haonanwa/Projects/PipeEdge/.py-build-cmake_cache/cp37-cp37m-macosx_10_9_x86_64', '--config', 'RelWithDebInfo', '-j']' returned non-zero exit status 2.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Cheers.

[cimes-isi]

CC @Mystery-Golden-Retriever (problem originated on his laptop)

[tttapa]

Thanks for the report!

Could you set the environment variable PY_BUILD_CMAKE_VERBOSE=1 and post the full output again, to see the environment and command line options used for all subprocesses?

Determining the architecture using CMake (regardless of what py-build-cmake does) is a bit tricky on Apple Silicon: https://cmake.org/cmake/help/latest/variable/CMAKE_HOST_SYSTEM_PROCESSOR.html#macos-platforms

Note that py-build-cmake does not enable cross-compilation on its own, except for the following:

1. It sets the CMAKE_OSX_ARCHITECTURES variable based on the value of the ARCHFLAGS environment variable on macOS, and
2. it sets the CMAKE_GENERATOR_PLATFORM variable and some others on Windows when the platform specified by the config file pointed to by the DIST_EXTRA_CONFIG variable is different from the platform of the running interpreter.

This is done in order to support cibuildwheel, which uses these environment variables to drive cross-compilation.
The difference between Windows and macOS is that for Windows, py-build-cmake sets CMAKE_SYSTEM_NAME, causing CMake to enter cross-compilation mode. On macOS, only CMAKE_OSX_ARCHITECTURES is set. This may cause the compiler to cross-compile, but does not enable the cross-compilation mode of CMake.

---

Just thinking out loud: are you able to build your project stand-alone (without py-build-cmake, just invoking CMake directly) when using an x86_64 version of CMake on an M2 machine?

Personally, I would treat building for x86 on an M2 as a cross-compilation situation. In other words, it's best to take control of the cross-compilation entirely, and write an appropriate CMake toolchain file, as well as a py-build-cmake cross-configuration as explained in https://tttapa.github.io/py-build-cmake/Cross-compilation.html. With the right CMake toolchain file, you can prevent CMake from picking up packages for the build system that were not intended for the target system.

For the further improvement of py-build-cmake, we would have to determine when to switch CMake into cross-compilation mode on macOS:

• Should we always enter cross-compilation mode if ARCHFLAGS is set to any value?
• Or only if the values doesn't match the true architecture of the machine?
• Or only if the value doesn't match the architecture of the CMake binary? (Which can be different from the machine architecture.)
• What about Universal2 binaries, containing both ARM and x86 versions of the binaries?

[cimes-isi]

Could you set the environment variable PY_BUILD_CMAKE_VERBOSE=1 and post the full output again, to see the environment and command line options used for all subprocesses?

I have an M1 mac, but don't have an Anaconda installation. I have a miniforge conda installed and would rather not try overloading it with an Anaconda unless really needed. I can try the following though.

Just thinking out loud: are you able to build your project stand-alone (without py-build-cmake, just invoking CMake directly) when using an x86_64 version of CMake on an M2 machine?

It doesn't look like CMake releases an x86_64-only build, just a universal one, so I'll try that. When forcing x64_64, the result is essentially the same:

$ file ~/Downloads/cmake-3.27.1-macos-universal/CMake.app/Contents/bin/cmake
/Users/cimes/Downloads/cmake-3.27.1-macos-universal/CMake.app/Contents/bin/cmake: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64] [arm64]
/Users/cimes/Downloads/cmake-3.27.1-macos-universal/CMake.app/Contents/bin/cmake (for architecture x86_64):	Mach-O 64-bit executable x86_64
/Users/cimes/Downloads/cmake-3.27.1-macos-universal/CMake.app/Contents/bin/cmake (for architecture arm64):	Mach-O 64-bit executable arm64

$ arch -x86_64 ~/Downloads/cmake-3.27.1-macos-universal/CMake.app/Contents/bin/cmake ..
CMake Warning (dev) at /Users/cimes/Downloads/cmake-3.27.1-macos-universal/CMake.app/Contents/share/cmake-3.27/Modules/GNUInstallDirs.cmake:243 (message):
  Unable to determine default CMAKE_INSTALL_LIBDIR directory because no
  target architecture is known.  Please enable at least one language before
  including GNUInstallDirs.
Call Stack (most recent call first):
  CMakeLists.txt:3 (include)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Checking for py-build-cmake environment - not found
--   Using default install paths
-- The CXX compiler identification is AppleClang 14.0.3.14030022
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done (0.6s)
-- Generating done (0.0s)
-- Build files have been written to: /Users/cimes/local/fogsys/PipeEdge-Test/src-native/build

$ arch -x86_64 ~/Downloads/cmake-3.27.1-macos-universal/CMake.app/Contents/bin/cmake --build .
[ 33%] Building CXX object CMakeFiles/sched-pipeline.dir/sched-pipeline.cpp.o
[ 66%] Building CXX object CMakeFiles/sched-pipeline.dir/schedule.cpp.o
/Users/cimes/local/fogsys/PipeEdge-Test/src-native/schedule.cpp:200:10: warning: variable 'stage_num' set but not used [-Wunused-but-set-variable]
  size_t stage_num = 0;
         ^
1 warning generated.
[100%] Linking CXX executable sched-pipeline
ld: warning: ignoring file /opt/homebrew/lib/libyaml-cpp.0.7.0.dylib, building for macOS-x86_64 but attempting to link with file built for macOS-arm64
Undefined symbols for architecture x86_64:
  "YAML::InvalidNode::~InvalidNode()", referenced from:
      yaml_model YAML::Node::as<yaml_model>() const in sched-pipeline.cpp.o
      YAML::Node::EnsureNodeExists() const in sched-pipeline.cpp.o
      YAML::Node::Scalar() const in sched-pipeline.cpp.o
      YAML::Node::Type() const in sched-pipeline.cpp.o
      YAML::Node::Mark() const in sched-pipeline.cpp.o
      unsigned long YAML::Node::as<unsigned long>() const in sched-pipeline.cpp.o
      std::__1::vector<unsigned long, std::__1::allocator<unsigned long>> YAML::Node::as<std::__1::vector<unsigned long, std::__1::allocator<unsigned long>>>() const in sched-pipeline.cpp.o
      ...
  "YAML::BadSubscript::~BadSubscript()", referenced from:
      YAML::detail::node* YAML::detail::node_data::get<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<YAML::detail::memory_holder>) const in sched-pipeline.cpp.o
      YAML::detail::node* YAML::detail::node_data::get<char [7]>(char const (&) [7], std::__1::shared_ptr<YAML::detail::memory_holder>) const in sched-pipeline.cpp.o
      YAML::detail::node* YAML::detail::node_data::get<char [14]>(char const (&) [14], std::__1::shared_ptr<YAML::detail::memory_holder>) const in sched-pipeline.cpp.o
      YAML::detail::node* YAML::detail::node_data::get<char [15]>(char const (&) [15], std::__1::shared_ptr<YAML::detail::memory_holder>) const in sched-pipeline.cpp.o
      YAML::detail::node* YAML::detail::node_data::get<char [8]>(char const (&) [8], std::__1::shared_ptr<YAML::detail::memory_holder>) const in sched-pipeline.cpp.o
      YAML::detail::node* YAML::detail::node_data::get<char [6]>(char const (&) [6], std::__1::shared_ptr<YAML::detail::memory_holder>) const in sched-pipeline.cpp.o
      YAML::detail::node* YAML::detail::node_data::get<char [11]>(char const (&) [11], std::__1::shared_ptr<YAML::detail::memory_holder>) const in sched-pipeline.cpp.o
      ...
  "YAML::BadConversion::~BadConversion()", referenced from:
      YAML::TypedBadConversion<yaml_model>::~TypedBadConversion() in sched-pipeline.cpp.o
      YAML::TypedBadConversion<unsigned long>::~TypedBadConversion() in sched-pipeline.cpp.o
      YAML::TypedBadConversion<std::__1::vector<unsigned long, std::__1::allocator<unsigned long>>>::~TypedBadConversion() in sched-pipeline.cpp.o
      YAML::TypedBadConversion<std::__1::vector<double, std::__1::allocator<double>>>::~TypedBadConversion() in sched-pipeline.cpp.o
      YAML::TypedBadConversion<double>::~TypedBadConversion() in sched-pipeline.cpp.o
      YAML::TypedBadConversion<std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, yaml_device_type, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const, yaml_device_type>>>>::~TypedBadConversion() in sched-pipeline.cpp.o
      YAML::TypedBadConversion<yaml_device_type>::~TypedBadConversion() in sched-pipeline.cpp.o
      ...
  "YAML::ostream_wrapper::write(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&)", referenced from:
      YAML::operator<<(YAML::ostream_wrapper&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) in sched-pipeline.cpp.o
  "YAML::detail::memory::create_node()", referenced from:
      YAML::detail::memory_holder::create_node() in sched-pipeline.cpp.o
  "YAML::detail::node_data::empty_scalar()", referenced from:
      YAML::Node::Scalar() const in sched-pipeline.cpp.o
  "YAML::detail::node_data::mark_defined()", referenced from:
      YAML::detail::node_ref::mark_defined() in sched-pipeline.cpp.o
  "YAML::detail::node_data::end()", referenced from:
      YAML::detail::node_ref::end() in sched-pipeline.cpp.o
  "YAML::detail::node_data::begin()", referenced from:
      YAML::detail::node_ref::begin() in sched-pipeline.cpp.o
  "YAML::detail::node_data::set_null()", referenced from:
      YAML::detail::node_ref::set_null() in sched-pipeline.cpp.o
  "YAML::Emitter::PrepareNode(YAML::EmitterNodeType::value)", referenced from:
      YAML::Emitter& YAML::Emitter::WriteIntegralType<unsigned long>(unsigned long) in sched-pipeline.cpp.o
  "YAML::Emitter::SetLocalValue(YAML::EMITTER_MANIP)", referenced from:
      YAML::operator<<(YAML::Emitter&, YAML::EMITTER_MANIP) in sched-pipeline.cpp.o
  "YAML::Emitter::StartedScalar()", referenced from:
      YAML::Emitter& YAML::Emitter::WriteIntegralType<unsigned long>(unsigned long) in sched-pipeline.cpp.o
  "YAML::Emitter::Write(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&)", referenced from:
      YAML::operator<<(YAML::Emitter&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) in sched-pipeline.cpp.o
  "YAML::Emitter::Emitter()", referenced from:
      print_schedule(std::__1::vector<host_sched_stage, std::__1::allocator<host_sched_stage>> const&) in sched-pipeline.cpp.o
  "YAML::Emitter::~Emitter()", referenced from:
      print_schedule(std::__1::vector<host_sched_stage, std::__1::allocator<host_sched_stage>> const&) in sched-pipeline.cpp.o
  "YAML::LoadFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&)", referenced from:
      load_files(schedule_ctx&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>) in sched-pipeline.cpp.o
  "YAML::Emitter::PrepareIntegralStream(std::__1::basic_stringstream<char, std::__1::char_traits<char>, std::__1::allocator<char>>&) const", referenced from:
      YAML::Emitter& YAML::Emitter::WriteIntegralType<unsigned long>(unsigned long) in sched-pipeline.cpp.o
  "YAML::Emitter::good() const", referenced from:
      YAML::Emitter& YAML::Emitter::WriteIntegralType<unsigned long>(unsigned long) in sched-pipeline.cpp.o
  "YAML::Emitter::c_str() const", referenced from:
      print_schedule(std::__1::vector<host_sched_stage, std::__1::allocator<host_sched_stage>> const&) in sched-pipeline.cpp.o
  "typeinfo for YAML::InvalidNode", referenced from:
      yaml_model YAML::Node::as<yaml_model>() const in sched-pipeline.cpp.o
      YAML::Node::EnsureNodeExists() const in sched-pipeline.cpp.o
      YAML::Node::Scalar() const in sched-pipeline.cpp.o
      YAML::Node::Type() const in sched-pipeline.cpp.o
      YAML::Node::Mark() const in sched-pipeline.cpp.o
      unsigned long YAML::Node::as<unsigned long>() const in sched-pipeline.cpp.o
      std::__1::vector<unsigned long, std::__1::allocator<unsigned long>> YAML::Node::as<std::__1::vector<unsigned long, std::__1::allocator<unsigned long>>>() const in sched-pipeline.cpp.o
      ...
  "typeinfo for YAML::BadSubscript", referenced from:
      YAML::detail::node* YAML::detail::node_data::get<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<YAML::detail::memory_holder>) const in sched-pipeline.cpp.o
      YAML::detail::node* YAML::detail::node_data::get<char [7]>(char const (&) [7], std::__1::shared_ptr<YAML::detail::memory_holder>) const in sched-pipeline.cpp.o
      YAML::detail::node* YAML::detail::node_data::get<char [14]>(char const (&) [14], std::__1::shared_ptr<YAML::detail::memory_holder>) const in sched-pipeline.cpp.o
      YAML::detail::node* YAML::detail::node_data::get<char [15]>(char const (&) [15], std::__1::shared_ptr<YAML::detail::memory_holder>) const in sched-pipeline.cpp.o
      YAML::detail::node* YAML::detail::node_data::get<char [8]>(char const (&) [8], std::__1::shared_ptr<YAML::detail::memory_holder>) const in sched-pipeline.cpp.o
      YAML::detail::node* YAML::detail::node_data::get<char [6]>(char const (&) [6], std::__1::shared_ptr<YAML::detail::memory_holder>) const in sched-pipeline.cpp.o
      YAML::detail::node* YAML::detail::node_data::get<char [11]>(char const (&) [11], std::__1::shared_ptr<YAML::detail::memory_holder>) const in sched-pipeline.cpp.o
      ...
  "typeinfo for YAML::BadConversion", referenced from:
      typeinfo for YAML::TypedBadConversion<yaml_model> in sched-pipeline.cpp.o
      typeinfo for YAML::TypedBadConversion<unsigned long> in sched-pipeline.cpp.o
      typeinfo for YAML::TypedBadConversion<std::__1::vector<unsigned long, std::__1::allocator<unsigned long>>> in sched-pipeline.cpp.o
      typeinfo for YAML::TypedBadConversion<std::__1::vector<double, std::__1::allocator<double>>> in sched-pipeline.cpp.o
      typeinfo for YAML::TypedBadConversion<double> in sched-pipeline.cpp.o
      typeinfo for YAML::TypedBadConversion<std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, yaml_device_type, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const, yaml_device_type>>>> in sched-pipeline.cpp.o
      typeinfo for YAML::TypedBadConversion<yaml_device_type> in sched-pipeline.cpp.o
      ...
  "vtable for YAML::InvalidNode", referenced from:
      YAML::InvalidNode::InvalidNode(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) in sched-pipeline.cpp.o
  NOTE: a missing vtable usually means the first non-inline virtual member function has no definition.
  "vtable for YAML::BadSubscript", referenced from:
      YAML::BadSubscript::BadSubscript<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>(YAML::Mark const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) in sched-pipeline.cpp.o
      YAML::BadSubscript::BadSubscript<char [7]>(YAML::Mark const&, char const (&) [7]) in sched-pipeline.cpp.o
      YAML::BadSubscript::BadSubscript<char [14]>(YAML::Mark const&, char const (&) [14]) in sched-pipeline.cpp.o
      YAML::BadSubscript::BadSubscript<char [15]>(YAML::Mark const&, char const (&) [15]) in sched-pipeline.cpp.o
      YAML::BadSubscript::BadSubscript<char [8]>(YAML::Mark const&, char const (&) [8]) in sched-pipeline.cpp.o
      YAML::BadSubscript::BadSubscript<char [6]>(YAML::Mark const&, char const (&) [6]) in sched-pipeline.cpp.o
      YAML::BadSubscript::BadSubscript<char [11]>(YAML::Mark const&, char const (&) [11]) in sched-pipeline.cpp.o
      ...
  NOTE: a missing vtable usually means the first non-inline virtual member function has no definition.
  "vtable for YAML::BadConversion", referenced from:
      YAML::BadConversion::BadConversion(YAML::Mark const&) in sched-pipeline.cpp.o
  NOTE: a missing vtable usually means the first non-inline virtual member function has no definition.
  "vtable for YAML::RepresentationException", referenced from:
      YAML::RepresentationException::RepresentationException(YAML::Mark const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) in sched-pipeline.cpp.o
  NOTE: a missing vtable usually means the first non-inline virtual member function has no definition.
  "vtable for YAML::Exception", referenced from:
      YAML::Exception::Exception(YAML::Mark const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) in sched-pipeline.cpp.o
  NOTE: a missing vtable usually means the first non-inline virtual member function has no definition.
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [sched-pipeline] Error 1
make[1]: *** [CMakeFiles/sched-pipeline.dir/all] Error 2
make: *** [all] Error 2

We did previously check the M2 environment and verified that no ARCHFLAGS for CMAKE_* environment variables were set once the conda environment was activated but before running pip to do the build.

I can test manually setting different environment variables for cmake (again forcing x64_64) to imitate what you think py-build-cmake would/should do, if you'd like.

[tttapa]

When forcing x64_64, the result is essentially the same:

Thanks, that's what I expected. I'm afraid that the issue you've encountered could be a fundamental limitation of running CMake under x86_64 emulation on an M2 mac.

The problem here is that CMake does not know that the system libraries (e.g. the ones in /opt/homebrew/lib) are for a different architecture than itself. When CMake is not in cross-compilation mode, it expects that it can use the system libraries without issues, but this assumption breaks under emulation.

There are (at least) two ways to handle this:

1. Linux distributions like Debian use what's called Multiarch, where the libraries are installed in architecture-specific directories, e.g. /usr/lib/aarch64-linux-gnu and /usr/lib/x86_64-linux-gnu instead of just /usr/lib, and CMake can then select the right version.
2. On other systems, you have to cross-compile explicitly, which causes CMake to ignore the system libraries, and only search in the sysroot and other paths you can specify.

In any case, you need to provide a version of the library for the architecture you selected (an x86_64 version of yaml-cpp, in your case). And you have to make sure that CMake is able to locate that library.

I'm not familiar with Homebrew, but from what I can gather, it does not support installing the same package for different architectures: https://github.com/orgs/Homebrew/discussions/2141
So you'll have to install yaml-cpp either from source, or using a different package manager that makes a distinction between architectures. Then you should cross-compile your project, and add the path to the correct yaml-cpp version to the CMAKE_FIND_ROOT_PATH variable in your toolchain file (list(APPEND CMAKE_FIND_ROOT_PATH "/your/x86_64/staging/directory")).
See https://cmake.org/cmake/help/book/mastering-cmake/chapter/Cross%20Compiling%20With%20CMake.html and https://cmake.org/cmake/help/latest/manual/cmake-toolchains.7.html for details about toolchain files and cross-compilation.

I don't own a macOS system myself, so you might get better suggestions over on the CMake forum or the CMake issue tracker (since the problem persists even without py-build-cmake).

[cimes-isi]

Thanks, I expected that, too, after seeing the initial failure through Anaconda. And again we don't actually need a solution to this using py-build-cmake. We're not going to do anything on our end to make this work. I just thought you should be aware of the behavior and I'm happy to help debug it as long as it doesn't turn my system upside down :). In case it wasn't clear, the universal cmake binary works fine from the CLI when we don't force x86_64 mode.

It's also probably the correct behavior for cmake to compile for x86_64 in this case, esp. if any of the code compiled is supposed to link with code running under the same x86_64 Python (Anaconda). It just shouldn't be trying to link against arm64 libraries.

[tttapa]

Thanks, I appreciate it! :)