[Build] Build failure on macOS with Homebrew-installed protobuf

[mcmara]

Describe the issue

Build failure on macOS with Homebrew-installed protobuf

Environment

• OS: macOS 26.1 (Tahoe)
• Hardware: Mac Mini (Apple Silicon)
• Package Manager: Homebrew
• Homebrew protobuf version: 33.x
• Build command:

./build.sh --config Release --build_shared_lib --parallel --use_coreml \
  --cmake_extra_defines CMAKE_POLICY_VERSION_MINIMUM=3.5

Problem Description

When building onnxruntime on macOS with Homebrew installed, the build fails with protobuf version mismatch errors:

In file included from /Users/.../build/MacOS/Release/coreml_proto/ArrayFeatureExtractor.pb.cc:4:
/Users/.../build/MacOS/Release/coreml_proto/ArrayFeatureExtractor.pb.h:12:2: error: This file was generated by a newer version of protoc which is
   12 | #error This file was generated by a newer version of protoc which is
      |  ^
/Users/.../build/MacOS/Release/coreml_proto/ArrayFeatureExtractor.pb.h:13:2: error: incompatible with your Protocol Buffer headers. Please update
   13 | #error incompatible with your Protocol Buffer headers. Please update
      |  ^
/Users/.../build/MacOS/Release/coreml_proto/ArrayFeatureExtractor.pb.h:14:2: error: your headers.
   14 | #error your headers.
      |  ^

Root Cause

The issue occurs because CMake's find_package(Protobuf) discovers and uses Homebrew's protobuf installation (version 33.x) located at /opt/homebrew/lib/cmake/protobuf, while onnxruntime's build system expects to use its own vendored protobuf version. This creates a version mismatch between:

1. The protoc compiler (from Homebrew) that generates .pb.h files
2. The protobuf headers used during compilation (bundled with onnxruntime)

Inspection of CMakeCache.txt confirms:

Protobuf_DIR:PATH=/opt/homebrew/lib/cmake/protobuf

Current Workaround

The only working solution is to temporarily hide Homebrew's protobuf installation:

# Temporarily unlink Homebrew's protobuf
brew unlink protobuf

# Clean and build
rm -rf build/
./build.sh --config Release --build_shared_lib --parallel --use_coreml \
  --cmake_extra_defines CMAKE_POLICY_VERSION_MINIMUM=3.5

# Restore Homebrew's protobuf after build
brew link protobuf

Failed Attempts

The following approaches did not work:

• Setting CMAKE_IGNORE_PATH to exclude Homebrew directories
• Modifying PATH to exclude Homebrew's bin directory
• Setting CMAKE_IGNORE_PREFIX_PATH
• Manually specifying protobuf paths via cmake defines
• Renaming only the protoc binary (CMake still finds the headers/libraries)

Requested Improvement

Could the build system be improved to:

1. Explicitly prefer vendored dependencies: Add a cmake option like -DUSE_SYSTEM_PROTOBUF=OFF (defaulting to OFF) that completely ignores system protobuf installations
2. Better error messages: If a system protobuf is detected, check version compatibility at CMake configure time and provide a clear error message with workaround instructions
3. Document the Homebrew conflict: Add a note to the build documentation about this known issue on macOS with Homebrew

Additional Context

This is a common pattern with Homebrew on macOS where system-wide installations interfere with projects that vendor their dependencies. Other projects handle this by:

• Explicitly setting CMAKE_FIND_PACKAGE_PREFER_CONFIG appropriately
• Using NO_SYSTEM_ENVIRONMENT_PATH in find_package() calls
• Providing clear build flags to prefer vendored dependencies

The current behavior is particularly confusing because the error only manifests during compilation, not during CMake configuration, making it difficult to diagnose.

Also, please note the CMAKE_POLICY_VERSION_MINIMUM=3.5. Without this options, the build fails on this system with:

CMake Error at /Users/.../dev/build/onnxruntime/Release/_deps/psimd-src/CMakeLists.txt:1 (CMAKE_MINIMUM_REQUIRED):
  Compatibility with CMake < 3.5 has been removed from CMake.

This should be addressed as well.

Urgency

No response

Target platform

macOS 26.1

Build script

./build.sh --config Release --build_shared_lib --parallel --use_coreml --cmake_extra_defines CMAKE_POLICY_VERSION_MINIMUM=3.5

Error / output

In file included from /Users/.../build/MacOS/Release/coreml_proto/ArrayFeatureExtractor.pb.cc:4:
/Users/.../build/MacOS/Release/coreml_proto/ArrayFeatureExtractor.pb.h:12:2: error: This file was generated by a newer version of protoc which is
   12 | #error This file was generated by a newer version of protoc which is
      |  ^
/Users/.../build/MacOS/Release/coreml_proto/ArrayFeatureExtractor.pb.h:13:2: error: incompatible with your Protocol Buffer headers. Please update
   13 | #error incompatible with your Protocol Buffer headers. Please update
      |  ^
/Users/.../build/MacOS/Release/coreml_proto/ArrayFeatureExtractor.pb.h:14:2: error: your headers.
   14 | #error your headers.
      |  ^

Visual Studio Version

No response

GCC / Compiler Version

$ gcc --version
Apple clang version 17.0.0 (clang-1700.4.4.1)
Target: arm64-apple-darwin25.1.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin