Skip to content

Commit c09dd7c

Browse files
jiaqizhojiaqizho
authored
enhance: add macOS C++ CI and tighten cache writes (#554)
The cache changes are meant to stop every PR from creating its own Conan and Rust cache under `refs/pull/.../merge`, while still letting PR jobs reuse caches that already exist. The CI workflows now use explicit restore and save actions: restore runs for normal jobs, but save only runs on main when the primary key was not hit. The cache keys also include runner OS and architecture, so Linux and macOS builds do not share incompatible Conan or Rust artifacts. Release jobs are restore-only, so they can consume existing caches without publishing new ones. This also adds a dedicated macOS C++ build job to cpp-ci.yml. It uses the normal cpp build path and runs make build, which keeps the mac build aligned with how the project is expected to build locally. The job is build-only and does not join the existing Linux unit-test artifact flow, so the current Linux build, upload, and test setup stays unchanged while macOS compatibility gets checked separately. --------- Signed-off-by: jiaqizho <jiaqi.zhou@zilliz.com>
1 parent 7714e4e commit c09dd7c

7 files changed

Lines changed: 274 additions & 48 deletions

File tree

‎.github/workflows/cpp-ci.yml‎

Lines changed: 56 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -36,21 +36,24 @@ jobs:
3636
id: rust-toolchain
3737
uses: dtolnay/rust-toolchain@stable
3838

39-
- name: rust build cache
40-
uses: actions/cache@v4
39+
- name: Restore rust build cache
40+
id: rust-cache
41+
uses: actions/cache/restore@v4
4142
with:
4243
path: |
4344
cpp/build/Release/cargo/build
44-
key: storage-bridge-rust-${{ runner.os }}-${{ steps.rust-toolchain.outputs.cachekey }}-${{ hashFiles('cpp/src/format/bridge/rust/Cargo.lock', 'cpp/src/format/bridge/rust/Cargo.toml') }}
45+
key: storage-bridge-rust-${{ runner.os }}-${{ runner.arch }}-${{ steps.rust-toolchain.outputs.cachekey }}-${{ hashFiles('cpp/src/format/bridge/rust/Cargo.lock', 'cpp/src/format/bridge/rust/Cargo.toml') }}
4546
restore-keys: |
46-
storage-bridge-rust-${{ runner.os }}-${{ steps.rust-toolchain.outputs.cachekey }}-
47+
storage-bridge-rust-${{ runner.os }}-${{ runner.arch }}-${{ steps.rust-toolchain.outputs.cachekey }}-
4748
48-
- name: conan package cache
49-
uses: actions/cache@v4
49+
- name: Restore conan package cache
50+
id: conan-cache
51+
uses: actions/cache/restore@v4
5052
with:
5153
path: ~/.conan2
52-
key: conan-cpp-${{ hashFiles('./cpp/conanfile.py') }}
53-
restore-keys: conan-cpp-
54+
key: conan-cpp-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('./cpp/conanfile.py') }}
55+
restore-keys: |
56+
conan-cpp-${{ runner.os }}-${{ runner.arch }}-
5457
5558
- name: setup conan
5659
run: |
@@ -64,6 +67,23 @@ jobs:
6467
run: |
6568
make build USE_ASAN=True BUILD_TYPE=Release
6669
70+
- name: Save rust build cache
71+
if: github.ref == 'refs/heads/main' && steps.rust-cache.outputs.cache-hit != 'true'
72+
uses: actions/cache/save@v4
73+
continue-on-error: true
74+
with:
75+
path: |
76+
cpp/build/Release/cargo/build
77+
key: ${{ steps.rust-cache.outputs.cache-primary-key }}
78+
79+
- name: Save conan package cache
80+
if: github.ref == 'refs/heads/main' && steps.conan-cache.outputs.cache-hit != 'true'
81+
uses: actions/cache/save@v4
82+
continue-on-error: true
83+
with:
84+
path: ~/.conan2
85+
key: ${{ steps.conan-cache.outputs.cache-primary-key }}
86+
6787
- name: Upload
6888
uses: actions/upload-artifact@v4
6989
with:
@@ -96,21 +116,24 @@ jobs:
96116
id: rust-toolchain
97117
uses: dtolnay/rust-toolchain@stable
98118

99-
- name: rust build cache
100-
uses: actions/cache@v4
119+
- name: Restore rust build cache
120+
id: rust-cache
121+
uses: actions/cache/restore@v4
101122
with:
102123
path: |
103124
cpp/build/Release/cargo/build
104-
key: storage-bridge-rust-${{ runner.os }}-${{ steps.rust-toolchain.outputs.cachekey }}-${{ hashFiles('cpp/src/format/bridge/rust/Cargo.lock', 'cpp/src/format/bridge/rust/Cargo.toml') }}
125+
key: storage-bridge-rust-${{ runner.os }}-${{ runner.arch }}-${{ steps.rust-toolchain.outputs.cachekey }}-${{ hashFiles('cpp/src/format/bridge/rust/Cargo.lock', 'cpp/src/format/bridge/rust/Cargo.toml') }}
105126
restore-keys: |
106-
storage-bridge-rust-${{ runner.os }}-${{ steps.rust-toolchain.outputs.cachekey }}-
127+
storage-bridge-rust-${{ runner.os }}-${{ runner.arch }}-${{ steps.rust-toolchain.outputs.cachekey }}-
107128
108-
- name: conan package cache
109-
uses: actions/cache@v4
129+
- name: Restore conan package cache
130+
id: conan-cache
131+
uses: actions/cache/restore@v4
110132
with:
111133
path: ~/.conan2
112-
key: conan-cpp-${{ hashFiles('./cpp/conanfile.py') }}
113-
restore-keys: conan-cpp-
134+
key: conan-cpp-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('./cpp/conanfile.py') }}
135+
restore-keys: |
136+
conan-cpp-${{ runner.os }}-${{ runner.arch }}-
114137
115138
- name: setup conan
116139
run: |
@@ -124,6 +147,23 @@ jobs:
124147
run: |
125148
make build BUILD_TYPE=Release
126149
150+
- name: Save rust build cache
151+
if: github.ref == 'refs/heads/main' && steps.rust-cache.outputs.cache-hit != 'true'
152+
uses: actions/cache/save@v4
153+
continue-on-error: true
154+
with:
155+
path: |
156+
cpp/build/Release/cargo/build
157+
key: ${{ steps.rust-cache.outputs.cache-primary-key }}
158+
159+
- name: Save conan package cache
160+
if: github.ref == 'refs/heads/main' && steps.conan-cache.outputs.cache-hit != 'true'
161+
uses: actions/cache/save@v4
162+
continue-on-error: true
163+
with:
164+
path: ~/.conan2
165+
key: ${{ steps.conan-cache.outputs.cache-primary-key }}
166+
127167
- name: check-tidy
128168
working-directory: ./cpp
129169
run: |
@@ -208,5 +248,3 @@ jobs:
208248
fail_ci_if_error: true
209249
disable_safe_directory: true
210250
verbose: true
211-
212-

‎.github/workflows/cpp-mac-ci.yml‎

Lines changed: 148 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,148 @@
1+
name: cpp-macos-optional
2+
3+
on:
4+
push:
5+
paths:
6+
- 'cpp/conanfile.py'
7+
- '.github/workflows/cpp-ci.yml'
8+
pull_request:
9+
paths:
10+
- 'cpp/conanfile.py'
11+
- '.github/workflows/cpp-ci.yml'
12+
13+
concurrency:
14+
group: ${{ github.workflow }}-${{ github.ref }}
15+
cancel-in-progress: true
16+
17+
jobs:
18+
mac-build:
19+
runs-on: macos-26
20+
timeout-minutes: 180
21+
steps:
22+
- uses: actions/checkout@v4
23+
24+
- name: Install dependencies
25+
uses: aminya/setup-cpp@v1
26+
with:
27+
conan: 2.25.1
28+
cmake: 3.31.10
29+
30+
- name: Install LLVM 18
31+
run: brew install llvm@18 libomp
32+
33+
- name: Use LLVM 18
34+
run: |
35+
LLVM_PREFIX="$(brew --prefix llvm@18)"
36+
echo "CC=$LLVM_PREFIX/bin/clang" >> "$GITHUB_ENV"
37+
echo "CXX=$LLVM_PREFIX/bin/clang++" >> "$GITHUB_ENV"
38+
echo "$LLVM_PREFIX/bin" >> "$GITHUB_PATH"
39+
40+
- name: Setup Rust
41+
id: rust-toolchain
42+
uses: dtolnay/rust-toolchain@stable
43+
44+
- name: Restore rust build cache
45+
id: rust-cache
46+
uses: actions/cache/restore@v4
47+
with:
48+
path: |
49+
cpp/build/Release/cargo/build
50+
key: storage-bridge-rust-${{ runner.os }}-${{ runner.arch }}-${{ steps.rust-toolchain.outputs.cachekey }}-${{ hashFiles('cpp/src/format/bridge/rust/Cargo.lock', 'cpp/src/format/bridge/rust/Cargo.toml') }}
51+
restore-keys: |
52+
storage-bridge-rust-${{ runner.os }}-${{ runner.arch }}-${{ steps.rust-toolchain.outputs.cachekey }}-
53+
54+
- name: Restore conan package cache
55+
id: conan-cache
56+
uses: actions/cache/restore@v4
57+
with:
58+
path: ~/.conan2
59+
key: conan-cpp-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('./cpp/conanfile.py') }}
60+
restore-keys: |
61+
conan-cpp-${{ runner.os }}-${{ runner.arch }}-
62+
63+
- name: setup conan
64+
run: |
65+
clang --version
66+
SETUP_CPP_CMAKE="$(find "$HOME/cmake" -path '*/CMake.app/Contents/bin/cmake' -type f | head -n 1)"
67+
if [ -z "$SETUP_CPP_CMAKE" ]; then
68+
echo "setup-cpp CMake not found under $HOME/cmake" >&2
69+
exit 1
70+
fi
71+
"$SETUP_CPP_CMAKE" --version
72+
conan profile detect --force
73+
{
74+
echo
75+
echo "[conf]"
76+
echo "tools.cmake:cmake_program=$SETUP_CPP_CMAKE"
77+
echo 'libsodium/*:tools.build:cflags=["-march=armv8-a+crypto+aes"]'
78+
} >> "${CONAN_HOME:-$HOME/.conan2}/profiles/default"
79+
conan profile show -pr:h default -pr:b default
80+
# || true: Conan 2.x errors if the remote already exists (e.g. restored from cache)
81+
conan remote add default-conan-local2 https://milvus01.jfrog.io/artifactory/api/conan/default-conan-local2 || true
82+
conan remote list
83+
84+
- name: Build
85+
working-directory: ./cpp
86+
run: |
87+
make build USE_ASAN=False libcxx_setting="-s:h compiler.libcxx=libc++ -s:h compiler.cppstd=20 -s:h arrow/*:compiler.cppstd=17 -s:b compiler.libcxx=libc++ -s:b compiler.cppstd=20"
88+
89+
- name: Upload mac gtest binary
90+
uses: actions/upload-artifact@v4
91+
with:
92+
name: mac-gtest-binary
93+
path: |
94+
cpp/build/Release/test/milvus_test
95+
cpp/build/Release/test/Test_FFI
96+
cpp/build/Release/libs/
97+
cpp/build/Release/libmilvus-storage*
98+
99+
- name: List gtest tests
100+
working-directory: ./cpp
101+
timeout-minutes: 10
102+
continue-on-error: true
103+
run: |
104+
mkdir -p build/Release/test-diagnostics
105+
otool -L ./build/Release/test/milvus_test > build/Release/test-diagnostics/milvus_test.otool-L.txt
106+
./build/Release/test/milvus_test --gtest_list_tests \
107+
> build/Release/test-diagnostics/gtest-list.stdout \
108+
2> build/Release/test-diagnostics/gtest-list.stderr &
109+
pid=$!
110+
echo "$pid" > build/Release/test-diagnostics/milvus_test.pid
111+
for _ in $(seq 1 300); do
112+
if ! kill -0 "$pid" 2>/dev/null; then
113+
wait "$pid"
114+
exit $?
115+
fi
116+
sleep 1
117+
done
118+
ps -o pid,ppid,stat,etime,command -p "$pid" > build/Release/test-diagnostics/milvus_test.ps.txt || true
119+
sample "$pid" 10 -file build/Release/test-diagnostics/milvus_test.sample.txt || true
120+
kill "$pid" || true
121+
wait "$pid" || true
122+
echo "milvus_test --gtest_list_tests did not finish within 300 seconds" >&2
123+
exit 124
124+
125+
- name: Upload gtest diagnostics
126+
if: always()
127+
uses: actions/upload-artifact@v4
128+
with:
129+
name: mac-gtest-diagnostics
130+
path: cpp/build/Release/test-diagnostics/
131+
if-no-files-found: ignore
132+
133+
- name: Save rust build cache
134+
if: github.ref == 'refs/heads/main' && steps.rust-cache.outputs.cache-hit != 'true'
135+
uses: actions/cache/save@v4
136+
continue-on-error: true
137+
with:
138+
path: |
139+
cpp/build/Release/cargo/build
140+
key: ${{ steps.rust-cache.outputs.cache-primary-key }}
141+
142+
- name: Save conan package cache
143+
if: github.ref == 'refs/heads/main' && steps.conan-cache.outputs.cache-hit != 'true'
144+
uses: actions/cache/save@v4
145+
continue-on-error: true
146+
with:
147+
path: ~/.conan2
148+
key: ${{ steps.conan-cache.outputs.cache-primary-key }}

‎.github/workflows/java-ci.yml‎

Lines changed: 28 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -43,21 +43,24 @@ jobs:
4343
id: rust-toolchain
4444
uses: dtolnay/rust-toolchain@stable
4545

46-
- name: rust build cache
47-
uses: actions/cache@v4
46+
- name: Restore rust build cache
47+
id: rust-cache
48+
uses: actions/cache/restore@v4
4849
with:
4950
path: |
5051
cpp/build/Release/cargo/build
51-
key: storage-bridge-rust-${{ runner.os }}-${{ steps.rust-toolchain.outputs.cachekey }}-${{ hashFiles('cpp/src/format/bridge/rust/Cargo.lock', 'cpp/src/format/bridge/rust/Cargo.toml') }}
52+
key: storage-bridge-rust-${{ runner.os }}-${{ runner.arch }}-${{ steps.rust-toolchain.outputs.cachekey }}-${{ hashFiles('cpp/src/format/bridge/rust/Cargo.lock', 'cpp/src/format/bridge/rust/Cargo.toml') }}
5253
restore-keys: |
53-
storage-bridge-rust-${{ runner.os }}-${{ steps.rust-toolchain.outputs.cachekey }}-
54+
storage-bridge-rust-${{ runner.os }}-${{ runner.arch }}-${{ steps.rust-toolchain.outputs.cachekey }}-
5455
55-
- name: Cache Conan packages
56-
uses: actions/cache@v4
56+
- name: Restore Conan packages
57+
id: conan-cache
58+
uses: actions/cache/restore@v4
5759
with:
5860
path: ~/.conan2
59-
key: conan-java-${{ hashFiles('./cpp/conanfile.py') }}
60-
restore-keys: conan-java-
61+
key: conan-java-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('./cpp/conanfile.py') }}
62+
restore-keys: |
63+
conan-java-${{ runner.os }}-${{ runner.arch }}-
6164
6265
- name: Setup Conan remote
6366
run: |
@@ -71,6 +74,23 @@ jobs:
7174
run: |
7275
make java-lib
7376
77+
- name: Save rust build cache
78+
if: github.ref == 'refs/heads/main' && steps.rust-cache.outputs.cache-hit != 'true'
79+
uses: actions/cache/save@v4
80+
continue-on-error: true
81+
with:
82+
path: |
83+
cpp/build/Release/cargo/build
84+
key: ${{ steps.rust-cache.outputs.cache-primary-key }}
85+
86+
- name: Save Conan packages
87+
if: github.ref == 'refs/heads/main' && steps.conan-cache.outputs.cache-hit != 'true'
88+
uses: actions/cache/save@v4
89+
continue-on-error: true
90+
with:
91+
path: ~/.conan2
92+
key: ${{ steps.conan-cache.outputs.cache-primary-key }}
93+
7494
- name: Copy JNI library
7595
run: |
7696
# Find and copy the library (it may be in different locations)

‎.github/workflows/python-ci.yml‎

Lines changed: 27 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -62,13 +62,14 @@ jobs:
6262
sudo apt-get install -y cmake libaio-dev
6363
pip install conan==${{ env.CONAN_VERSION }}
6464
65-
- name: Cache Conan packages
66-
uses: actions/cache@v4
65+
- name: Restore Conan packages
66+
id: conan-cache
67+
uses: actions/cache/restore@v4
6768
with:
6869
path: ~/.conan2
69-
key: conan-cpp-${{ hashFiles('./cpp/conanfile.py') }}
70+
key: conan-cpp-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('./cpp/conanfile.py') }}
7071
restore-keys: |
71-
conan-cpp-
72+
conan-cpp-${{ runner.os }}-${{ runner.arch }}-
7273
7374
- name: Setup Conan remote
7475
run: |
@@ -81,20 +82,38 @@ jobs:
8182
id: rust-toolchain
8283
uses: dtolnay/rust-toolchain@stable
8384

84-
- name: rust build cache
85-
uses: actions/cache@v4
85+
- name: Restore rust build cache
86+
id: rust-cache
87+
uses: actions/cache/restore@v4
8688
with:
8789
path: |
8890
cpp/build/Release/cargo/build
89-
key: storage-bridge-rust-${{ runner.os }}-${{ steps.rust-toolchain.outputs.cachekey }}-${{ hashFiles('cpp/src/format/bridge/rust/Cargo.lock', 'cpp/src/format/bridge/rust/Cargo.toml') }}
91+
key: storage-bridge-rust-${{ runner.os }}-${{ runner.arch }}-${{ steps.rust-toolchain.outputs.cachekey }}-${{ hashFiles('cpp/src/format/bridge/rust/Cargo.lock', 'cpp/src/format/bridge/rust/Cargo.toml') }}
9092
restore-keys: |
91-
storage-bridge-rust-${{ runner.os }}-${{ steps.rust-toolchain.outputs.cachekey }}-
93+
storage-bridge-rust-${{ runner.os }}-${{ runner.arch }}-${{ steps.rust-toolchain.outputs.cachekey }}-
9294
9395
- name: Build C++ library for Python
9496
working-directory: ./cpp
9597
run: |
9698
make python-lib
9799
100+
- name: Save Conan packages
101+
if: github.ref == 'refs/heads/main' && steps.conan-cache.outputs.cache-hit != 'true'
102+
uses: actions/cache/save@v4
103+
continue-on-error: true
104+
with:
105+
path: ~/.conan2
106+
key: ${{ steps.conan-cache.outputs.cache-primary-key }}
107+
108+
- name: Save rust build cache
109+
if: github.ref == 'refs/heads/main' && steps.rust-cache.outputs.cache-hit != 'true'
110+
uses: actions/cache/save@v4
111+
continue-on-error: true
112+
with:
113+
path: |
114+
cpp/build/Release/cargo/build
115+
key: ${{ steps.rust-cache.outputs.cache-primary-key }}
116+
98117
- name: Set library path
99118
run: |
100119
BUILD_DIR="$GITHUB_WORKSPACE/cpp/build/Release"

0 commit comments

Comments
 (0)