Skip to content

Commit b1050ee

Browse files
tianleiwutianleiwu
committed
[Build] Fix java macos (#27271)
### Description This PR restores Java support on macOS arm64 and fixes Jar testing failures on the new AcesShared pool. #### Background Commit `5ed340f7a51f3cbdb62577a874daf2b3f23d6a93` (#26252) moved macOS builds to a faster pool (AcesShared) which reduced build time by 85%, but this pool doesn't have JDK installed and ADO's `JavaToolInstaller` doesn't support macOS. As a result, Java binaries for macOS arm64 were temporarily removed. #### Changes 1. Enable Java Builds & Tests on macOS ARM64: * Install JDK 17: Added a script to install JDK 17 via Homebrew if missing on the agent. * Install Maven: Added a fallback to install Maven using curl (since wget is missing on macOS) and configured it to use the * dynamically resolved JAVA_HOME. * Pipeline Updates: Updated jar_package_testing.yml and final-jar-testing-linux.yml to run correctly on AcesShared. 2. Fix C API Tests on macOS ARM64: * Pool Migration: Updated c-api-noopenmp-test-pipelines.yml to use AcesShared with the correct ImageOverride. * Template Enhancements: Updated nuget/templates/test_macos.yml to support dynamic AgentPool and PoolDemands. * Fix Missing Artifact: Modified mac-cpu-packaging-steps.yml to explicitly copy libcustom_op_library.dylib into the testdata folder of the artifact, resolving DllNotFoundException in EndToEndTests. ### Motivation and Context To ensure robust CI coverage for macOS ARM64 (Apple Silicon) for both Java and C APIs effectively using the efficient AcesShared pool. ### Testing - Final_Jar_Testing_MacOS passed: https://dev.azure.com/aiinfra/Lotus/_build/results?buildId=1081961&view=logs&j=f1f8e11e-a9fa-53e5-cd29-3ba2c1988550&t=f4fafe98-de38-519c-0045-d220f6898d47
1 parent 54a1352 commit b1050ee

File tree

7 files changed

+123
-29
lines changed

7 files changed

+123
-29
lines changed

‎tools/ci_build/github/azure-pipelines/jar_package_testing.yml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,8 @@ stages:
2121
- template: templates/final-jar-testing-linux.yml
2222
parameters:
2323
OS: MacOS
24-
PoolName: 'macOS-14'
24+
PoolName: 'AcesShared'
25+
PoolDemands: 'ImageOverride -equals ACES_VM_SharedPool_Sequoia'
2526

2627
- stage: GPU_JAR_Testing
2728
dependsOn: []

‎tools/ci_build/github/azure-pipelines/templates/c-api-cpu.yml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -203,6 +203,10 @@ stages:
203203
- input: pipelineArtifact
204204
artifactName: drop-onnxruntime-java-linux-aarch64
205205
targetPath: '$(Build.BinariesDirectory)\java-artifact\onnxruntime-java-linux-aarch64'
206+
207+
- input: pipelineArtifact
208+
artifactName: drop-onnxruntime-java-osx-arm64
209+
targetPath: '$(Build.BinariesDirectory)\java-artifact\onnxruntime-java-osx-arm64'
206210
outputs:
207211
- output: pipelineArtifact
208212
targetPath: $(Build.BinariesDirectory)\java-artifact\onnxruntime-java-win-x64

‎tools/ci_build/github/azure-pipelines/templates/final-jar-testing-linux.yml‎

Lines changed: 85 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,10 @@ parameters:
88
- name: PoolName
99
type: string
1010

11+
- name: PoolDemands
12+
type: string
13+
default: ''
14+
1115
stages:
1216
- stage: Final_Jar_Testing_${{parameters.OS}}
1317
dependsOn: []
@@ -17,7 +21,16 @@ stages:
1721
clean: all
1822
${{ if eq(parameters.OS, 'MacOS') }}:
1923
pool:
20-
vmImage: 'macOS-15'
24+
# Use PoolName if provided, otherwise fallback to macOS-15
25+
${{ if ne(parameters.PoolName, '') }}:
26+
${{ if contains(parameters.PoolName, '-') }}:
27+
vmImage: ${{ parameters.PoolName }}
28+
${{ else }}:
29+
name: ${{ parameters.PoolName }}
30+
${{ if ne(parameters.PoolDemands, '') }}:
31+
demands: ${{ parameters.PoolDemands }}
32+
${{ else }}:
33+
vmImage: 'macOS-15'
2134
${{ if eq(parameters.OS, 'Linux') }}:
2235
pool:
2336
name: ${{ parameters.PoolName }}
@@ -29,24 +42,41 @@ stages:
2942
- template: set-version-number-variables-step.yml
3043

3144
- bash: |
32-
echo "Downloading and installing Maven $(mavenVersion) for Linux..."
45+
echo "Downloading and installing Maven $(mavenVersion)..."
3346
MAVEN_DIR="$(Agent.TempDirectory)/apache-maven-$(mavenVersion)"
47+
3448
# Download Maven binary
35-
wget https://archive.apache.org/dist/maven/maven-3/$(mavenVersion)/binaries/apache-maven-$(mavenVersion)-bin.tar.gz -O $(Agent.TempDirectory)/maven.tar.gz
49+
if command -v wget &> /dev/null; then
50+
wget https://archive.apache.org/dist/maven/maven-3/$(mavenVersion)/binaries/apache-maven-$(mavenVersion)-bin.tar.gz -O $(Agent.TempDirectory)/maven.tar.gz
51+
else
52+
curl -L -o $(Agent.TempDirectory)/maven.tar.gz https://archive.apache.org/dist/maven/maven-3/$(mavenVersion)/binaries/apache-maven-$(mavenVersion)-bin.tar.gz
53+
fi
3654
3755
# Extract to the temp directory
3856
mkdir -p ${MAVEN_DIR}
3957
tar -xzf $(Agent.TempDirectory)/maven.tar.gz -C $(Agent.TempDirectory)
4058
4159
# Add Maven's bin directory to the PATH for subsequent tasks in the job
4260
echo "##vso[task.prependpath]${MAVEN_DIR}/bin"
43-
displayName: 'Install Maven (Linux)'
44-
condition: and(succeeded(), eq(variables['Agent.OS'], 'Linux'))
61+
displayName: 'Install Maven'
62+
condition: and(succeeded(), in(variables['Agent.OS'], 'Linux', 'Darwin'))
4563
4664
- script: |
4765
echo "Maven is now on the PATH."
4866
mvn --version
4967
68+
- script: |
69+
set -e -x
70+
if ! /usr/libexec/java_home -v 17 >/dev/null 2>&1; then
71+
brew install --cask temurin@17
72+
fi
73+
JAVA_HOME=$(/usr/libexec/java_home -v 17)
74+
echo "JAVA_HOME is set to: $JAVA_HOME"
75+
echo "##vso[task.setvariable variable=JAVA_HOME]$JAVA_HOME"
76+
echo "##vso[task.prependpath]$JAVA_HOME/bin"
77+
displayName: 'Install JDK 17 (macOS)'
78+
condition: and(succeeded(), eq(variables['Agent.OS'], 'Darwin'))
79+
5080
- download: build
5181
artifact: 'onnxruntime-java'
5282
displayName: 'Download Final Jar'
@@ -58,16 +88,17 @@ stages:
5888
goals: 'dependency:copy-dependencies'
5989
options: '-DoutputDirectory=$(Pipeline.Workspace)/build/onnxruntime-java'
6090
publishJUnitTestResults: false
61-
javaHomeOption: 'JDKVersion'
62-
jdkVersionOption: '1.17'
6391
mavenVersionOption: 'Default'
92+
${{ if eq(parameters.OS, 'MacOS') }}:
93+
javaHomeOption: 'Path'
94+
jdkDirectory: '$(JAVA_HOME)'
95+
${{ if eq(parameters.OS, 'Linux') }}:
96+
javaHomeOption: 'JDKVersion'
97+
jdkVersionOption: '1.17'
6498

6599
- task: Bash@3
66-
displayName: 'Run Java Tests on Linux'
67-
# condition: and(succeeded(), in(variables['Agent.OS'], 'Linux', 'Darwin'))
68-
# MacOS packages have been removed from the JAR here:
69-
# https://github.com/microsoft/onnxruntime/commit/5ed340f7a51f3cbdb62577a874daf2b3f23d6a93#diff-a14cc5ea231eb4fa49f13510a242043c47ae48516c860f8a87b0e55762632f49
70-
condition: and(succeeded(), in(variables['Agent.OS'], 'Linux'))
100+
displayName: 'Run Java Tests'
101+
condition: and(succeeded(), in(variables['Agent.OS'], 'Linux', 'Darwin'))
71102
inputs:
72103
targetType: 'inline'
73104
script: |
@@ -83,24 +114,54 @@ stages:
83114
cd ..
84115
mkdir tests
85116
cd tests
117+
# 1. Diagnostics
118+
echo "System Info:"
119+
uname -a
120+
if [[ "$(uname)" == "Darwin" ]]; then arch; fi
121+
echo "Java Version"
122+
java -version
123+
124+
# 2. Extract
86125
jar xf $(Pipeline.Workspace)/build/onnxruntime-java/testing.jar
87126
rm -f $(Pipeline.Workspace)/build/onnxruntime-java/testing.jar
88-
ls $(Pipeline.Workspace)/build/tests
127+
128+
# Identify main jar (avoiding sources and javadoc jars)
129+
MAIN_JAR=$(ls $(Pipeline.Workspace)/build/onnxruntime-java/onnxruntime-*.jar | grep -v 'sources' | grep -v 'javadoc' | head -n 1)
130+
echo "Extracting native libs from $MAIN_JAR"
131+
jar xf $MAIN_JAR ai/onnxruntime/native
132+
133+
ls -R $(Pipeline.Workspace)/build/tests/ai
89134
echo "Java Version"
90135
java -version
91136
92-
# Set the correct library path based on the OS
137+
138+
# 3. Find with robustness
93139
os_name=$(uname)
94-
if [[ "$os_name" == "Linux" ]]; then
95-
echo "Platform: Linux. Setting LD_LIBRARY_PATH."
96-
export LD_LIBRARY_PATH="$(pwd):$LD_LIBRARY_PATH"
97-
java -cp '$(Pipeline.Workspace)/build/tests:$(Pipeline.Workspace)/build/onnxruntime-java/*' org.junit.platform.console.ConsoleLauncher --scan-classpath=$(Pipeline.Workspace)/build/tests \
98-
--fail-if-no-tests --disable-banner --reports-dir "$(Build.ArtifactStagingDirectory)/TestResults"
99-
elif [[ "$os_name" == "Darwin" ]]; then
100-
echo "Platform: macOS. Setting DYLD_LIBRARY_PATH."
101-
export DYLD_LIBRARY_PATH="$(pwd):$DYLD_LIBRARY_PATH"
102-
java -DUSE_WEBGPU=1 -DUSE_COREML=1 -cp '$(Pipeline.Workspace)/build/tests:$(Pipeline.Workspace)/build/onnxruntime-java/*' org.junit.platform.console.ConsoleLauncher --scan-classpath=$(Pipeline.Workspace)/build/tests \
103-
--fail-if-no-tests --disable-banner --reports-dir "$(Build.ArtifactStagingDirectory)/TestResults"
140+
if [[ "$os_name" == "Linux" ]]; then S_FILE="libonnxruntime.so"; else S_FILE="libonnxruntime.dylib"; fi
141+
142+
echo "Searching for $S_FILE in $(pwd)..."
143+
# Exclude .dSYM paths and find actual file
144+
NATIVE_LIB_PATH=$(find $(pwd) -name "$S_FILE" -not -path "*.dSYM*" -type f | head -n 1)
145+
146+
if [[ -n "$NATIVE_LIB_PATH" ]]; then
147+
NATIVE_LIB_DIR=$(dirname "$NATIVE_LIB_PATH")
148+
echo "Found native lib dir: $NATIVE_LIB_DIR"
149+
150+
if [[ "$os_name" == "Linux" ]]; then
151+
echo "Platform: Linux. Setting LD_LIBRARY_PATH."
152+
export LD_LIBRARY_PATH="$NATIVE_LIB_DIR:$(pwd):$LD_LIBRARY_PATH"
153+
java -cp '$(Pipeline.Workspace)/build/tests:$(Pipeline.Workspace)/build/onnxruntime-java/*' org.junit.platform.console.ConsoleLauncher --scan-classpath=$(Pipeline.Workspace)/build/tests \
154+
--fail-if-no-tests --disable-banner --reports-dir "$(Build.ArtifactStagingDirectory)/TestResults"
155+
elif [[ "$os_name" == "Darwin" ]]; then
156+
echo "Platform: macOS. Setting DYLD_LIBRARY_PATH."
157+
export DYLD_LIBRARY_PATH="$NATIVE_LIB_DIR:$(pwd):$DYLD_LIBRARY_PATH"
158+
java -DUSE_WEBGPU=1 -DUSE_COREML=1 -cp '$(Pipeline.Workspace)/build/tests:$(Pipeline.Workspace)/build/onnxruntime-java/*' org.junit.platform.console.ConsoleLauncher --scan-classpath=$(Pipeline.Workspace)/build/tests \
159+
--fail-if-no-tests --disable-banner --reports-dir "$(Build.ArtifactStagingDirectory)/TestResults"
160+
fi
161+
else
162+
echo "Error: $S_FILE not found!"
163+
ls -R ai
164+
exit 1
104165
fi
105166
106167

‎tools/ci_build/github/azure-pipelines/templates/mac-cpu-packaging-steps.yml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,14 @@ steps:
4040
targetPath: '$(Build.ArtifactStagingDirectory)'
4141
artifactName: 'onnxruntime-osx-${{ parameters.MacosArch }}'
4242

43+
- template: java-api-artifacts-package-and-publish-steps-posix.yml
44+
parameters:
45+
arch: 'osx-${{ parameters.MacosArch }}'
46+
buildConfig: 'Release'
47+
artifactName: 'onnxruntime-java-osx-${{ parameters.MacosArch }}'
48+
libraryName: 'libonnxruntime.dylib'
49+
nativeLibraryName: 'libonnxruntime4j_jni.dylib'
50+
4351
- template: nodejs-artifacts-package-and-publish-steps-posix.yml
4452
parameters:
4553
arch: arm64

‎tools/ci_build/github/azure-pipelines/templates/mac-cpu-packing-jobs.yml‎

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -45,9 +45,20 @@ jobs:
4545
set -e -x
4646
export ONNX_ML=1
4747
export CMAKE_ARGS="-DONNX_GEN_PB_TYPE_STUBS=ON -DONNX_WERROR=OFF"
48-
python3 -m pip install -r '$(Build.SourcesDirectory)/tools/ci_build/github/linux/docker/scripts/requirements.txt'
48+
python3 -m pip install -r '$(Build.SourcesDirectory)/tools/ci_build/github/linux/docker/scripts/requirements.txt'
49+
50+
- script: |
51+
set -e -x
52+
if ! /usr/libexec/java_home -v 17 >/dev/null 2>&1; then
53+
brew install --cask temurin@17
54+
fi
55+
JAVA_HOME=$(/usr/libexec/java_home -v 17)
56+
echo "JAVA_HOME is set to: $JAVA_HOME"
57+
echo "##vso[task.setvariable variable=JAVA_HOME]$JAVA_HOME"
58+
echo "##vso[task.prependpath]$JAVA_HOME/bin"
59+
displayName: 'Install JDK 17'
4960
5061
- template: mac-cpu-packaging-steps.yml
5162
parameters:
5263
MacosArch: arm64
53-
AdditionalBuildFlags: ${{ parameters.AdditionalBuildFlags }} --build_nodejs --use_coreml --use_webgpu --cmake_extra_defines CMAKE_OSX_ARCHITECTURES=arm64
64+
AdditionalBuildFlags: ${{ parameters.AdditionalBuildFlags }} --build_java --build_nodejs --use_coreml --use_webgpu --cmake_extra_defines CMAKE_OSX_ARCHITECTURES=arm64

‎tools/ci_build/github/windows/jar_packaging.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -232,6 +232,7 @@ def run_packaging(package_type: str, build_dir: str):
232232
"platforms": [
233233
{"path": "onnxruntime-java-linux-x64", "lib": "libcustom_op_library.so", "archive_lib": True},
234234
{"path": "onnxruntime-java-linux-aarch64", "lib": "libcustom_op_library.so", "archive_lib": False},
235+
{"path": "onnxruntime-java-osx-arm64", "lib": "libcustom_op_library.dylib", "archive_lib": True},
235236
]
236237
},
237238
"gpu": {

‎tools/ci_build/github/windows/jar_packaging_test.py‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -52,14 +52,19 @@ def _setup_test_directory(package_type: str, version_string: str):
5252
create_empty_file(linux_native_dir / "libonnxruntime_providers_cuda.so")
5353
(linux_dir / "_manifest" / "spdx_2.2").mkdir(parents=True, exist_ok=True)
5454

55-
# --- Additional platforms (for CPU test) ---
55+
# --- macOS and other platforms (for CPU test) ---
5656
if package_type == "cpu":
57-
# Add linux-aarch64 for CPU test
57+
# Add linux-aarch64 and osx-arm64 for CPU test
5858
linux_aarch64_dir = java_artifact_dir / "onnxruntime-java-linux-aarch64"
5959
linux_aarch64_native_dir = linux_aarch64_dir / "ai" / "onnxruntime" / "native" / "linux-aarch64"
6060
linux_aarch64_native_dir.mkdir(parents=True, exist_ok=True)
6161
create_empty_file(linux_aarch64_dir / "libcustom_op_library.so")
6262

63+
osx_arm64_dir = java_artifact_dir / "onnxruntime-java-osx-arm64"
64+
osx_arm64_native_dir = osx_arm64_dir / "ai" / "onnxruntime" / "native" / "osx-arm64"
65+
osx_arm64_native_dir.mkdir(parents=True, exist_ok=True)
66+
create_empty_file(osx_arm64_dir / "libcustom_op_library.dylib")
67+
6368
return tmp_path
6469

6570
return _setup_test_directory
@@ -128,9 +133,12 @@ def test_cpu_packaging(directory_setup_factory, version_string):
128133
with zipfile.ZipFile(testing_jar_path, "r") as zf:
129134
jar_contents = zf.namelist()
130135
assert "libcustom_op_library.so" in jar_contents
136+
assert "libcustom_op_library.dylib" in jar_contents
131137

132138
# 3. Verify the custom op libraries were removed from the source directories
133139
linux_dir = temp_build_dir / "java-artifact" / "onnxruntime-java-linux-x64"
134140
linux_aarch64_dir = temp_build_dir / "java-artifact" / "onnxruntime-java-linux-aarch64"
141+
osx_arm64_dir = temp_build_dir / "java-artifact" / "onnxruntime-java-osx-arm64"
135142
assert not (linux_dir / "libcustom_op_library.so").exists()
136143
assert not (linux_aarch64_dir / "libcustom_op_library.so").exists()
144+
assert not (osx_arm64_dir / "libcustom_op_library.dylib").exists()

0 commit comments

Comments
 (0)