Skip to content

Commit b1127f6

Browse files
yomybabyyomybaby
committed
chore(FR-2612): optimize release workflow with parallel jobs and eliminate double React build (#6765)
Resolves #6815 ([FR-2612](https://lablup.atlassian.net/browse/FR-2612)) ## Summary The release pipeline previously ran everything sequentially on a single macOS runner (~30+ min). This PR introduces several optimizations that cut release time roughly in half. ### Dry run test result | Job | Duration | Runner | |-----|----------|--------| | `build_web` | 3m 30s | ubuntu | | `build_mac` (unsigned) | 3m 54s | macos | | `build_desktop` | 2m 54s | ubuntu | | **Total wall time** | **~7m 24s** | | Previous release (v26.4.3): **26m 35s** → **72% reduction** in wall time. Actual release with code signing/notarization is expected to be ~10-12 min. - Dry run: https://github.com/lablup/backend.ai-webui/actions/runs/24537688533 ### Changes 1. **Split CI into 3 parallel jobs** (`package.yml`) - `build_web` (ubuntu): Builds web assets + uploads bundle - `build_mac` (macos): macOS desktop builds with code signing/notarization - `build_desktop` (ubuntu): Windows + Linux desktop builds - macOS and win/linux jobs run **concurrently** after web build completes 2. **Eliminate double React build** (`scripts/patch-electron-publicpath.js`, `Makefile`) - Previously the entire React app was rebuilt for Electron just to change `publicPath` from `/` to `es6://` - Now a lightweight post-build script patches the already-built files instead 3. **Parallelize local proxy compilation** (`Makefile`) - All platform targets now compile concurrently within each job via the new `compile_all_localproxy` target - `compile_localproxy` is now **idempotent**: skips when the output ZIP already exists, so downstream targets (`mac_x64`, `win_x64`, ...) reuse the cached artifact without recompiling. Set `FORCE_COMPILE_LOCALPROXY=1` to force a rebuild. - `dep_electron` is likewise idempotent: skips when `build/electron-app/app/index.html` already carries the patched `es6://static/js/main` marker. Set `FORCE_DEP_ELECTRON=1` to force a re-sync. - This means the original single set of `mac_x64`/`mac_arm64`/`win_*`/`linux_*` targets can be reused across both ad-hoc local builds and the new parallel CI flow — no duplicated `*_package` targets required. 4. **Optimize ZIP compression** (`Makefile`) - Changed from `-9` (max) to `-6` (default) -- marginal size difference, measurably faster 5. **Concurrent release asset uploads** (`upload-release.js`) - Uploads up to 4 files simultaneously instead of sequentially 6. **Explicit `NODE_OPTIONS`** (`react/package.json`) - Added `--max-old-space-size=4096` to `build:only` script to prevent OOM during webpack builds 7. **`workflow_dispatch` trigger** (`package.yml`) - Enables manual dry-run testing of the build pipeline on any branch - `dry_run=true` skips release uploads and code signing ### Backward Compatibility - Existing `make mac`, `make win`, `make linux`, `make dep` targets work unchanged - `make mac_x64`, `make mac_arm64`, `make win_x64`, `make win_arm64`, `make linux_x64`, `make linux_arm64` targets work unchanged — no new `*_package` variants introduced - `make all` now uses the optimized parallel path internally (pre-builds all local proxies via `compile_all_localproxy`, then calls the original platform targets which become cheap thanks to idempotent prerequisites) - `make bundle` no longer requires full `dep` (just `dep_web`) ## Test plan - [x] Verify `scripts/patch-electron-publicpath.js` correctly patches `es6://` references (local mock test) - [x] Verify `build_web` job builds web assets and uploads artifact - [x] Verify `build_mac` job downloads artifact, prepares Electron, packages DMG - [x] Verify `build_desktop` job packages Win/Linux ZIPs - [x] Verify all 3 jobs complete successfully in parallel - [x] Verify `compile_localproxy` skips when output ZIP exists and rebuilds with `FORCE_COMPILE_LOCALPROXY=1` - [x] Verify `dep_electron` skips when `build/electron-app/app/index.html` already patched and resyncs with `FORCE_DEP_ELECTRON=1` - [ ] Verify actual release (non-dry-run) uploads assets correctly Generated with [Claude Code](https://claude.com/claude-code) [FR-2612]: https://lablup.atlassian.net/browse/FR-2612?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ
1 parent 64b9edb commit b1127f6

5 files changed

Lines changed: 384 additions & 46 deletions

File tree

‎.github/workflows/package.yml‎

Lines changed: 168 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -1,48 +1,132 @@
1-
# This workflow will triage pull requests and apply a label based on the
2-
# paths that are modified in the pull request.
1+
# Build and release workflow for Backend.AI Desktop and WebUI bundle.
32
#
4-
# To use this workflow, you will need to set up a .github/labeler.yml
5-
# file with configuration. For more information, see:
6-
# https://github.com/actions/labeler/blob/master/README.md
3+
# Architecture: 3 parallel jobs after a shared web build step.
4+
#
5+
# build_web (ubuntu) ──┬──> build_mac (macos) → DMG x64/arm64 + local proxy
6+
# ├──> build_desktop (ubuntu) → Win/Linux ZIP x64/arm64 + local proxy
7+
# └──> upload web bundle
8+
#
9+
# Key optimizations over the previous single-job approach:
10+
# 1. Parallel jobs: macOS + win/linux builds run concurrently (~10 min saved)
11+
# 2. No double React build: publicPath patching replaces full rebuild (~5 min saved)
12+
# 3. Parallel local proxy compilation within each job (~3 min saved)
13+
# 4. Optimized ZIP compression level (-6 vs -9, marginal size diff, ~1 min saved)
714

815
name: Build and Release Packages
916
on:
1017
release:
1118
types: [published]
19+
workflow_dispatch:
20+
inputs:
21+
dry_run:
22+
description: 'Skip release asset upload (for testing the build pipeline)'
23+
type: boolean
24+
default: true
25+
26+
env:
27+
NODE_OPTIONS: --max-old-space-size=4096
1228

1329
jobs:
30+
# ──────────────────────────────────────────────────────────────────────
31+
# Job 1: Build web assets and create the web bundle (ubuntu, ~8 min)
32+
# ──────────────────────────────────────────────────────────────────────
33+
build_web:
34+
permissions:
35+
contents: write
36+
runs-on: ubuntu-latest
37+
steps:
38+
- name: Check out Git repository
39+
uses: actions/checkout@v4
40+
41+
- uses: pnpm/action-setup@v4
42+
name: Install pnpm
43+
with:
44+
version: latest
45+
run_install: false
46+
47+
- name: Install Node.js
48+
uses: actions/setup-node@v4
49+
with:
50+
node-version-file: '.nvmrc'
51+
cache: 'pnpm'
52+
53+
- name: Install Dependencies
54+
run: pnpm install --no-frozen-lockfile
55+
56+
- name: Build web assets
57+
run: make dep_web
58+
59+
- name: Create web bundle
60+
run: make bundle
61+
62+
- name: Upload release bundle
63+
if: inputs.dry_run != true
64+
run: node upload-release.js app
65+
env:
66+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
67+
68+
# Share build artifacts with downstream desktop jobs
69+
- name: Upload build artifacts
70+
uses: actions/upload-artifact@v4
71+
with:
72+
name: web-build
73+
path: |
74+
build/web/
75+
src/wsproxy/dist/
76+
retention-days: 1
77+
compression-level: 3
78+
79+
# ──────────────────────────────────────────────────────────────────────
80+
# Job 2: Build macOS desktop apps — requires macOS for code signing,
81+
# notarization, and DMG creation (~10 min)
82+
# ──────────────────────────────────────────────────────────────────────
1483
build_mac:
84+
needs: build_web
1585
permissions:
1686
contents: write
17-
checks: write
18-
actions: read
19-
issues: read
20-
packages: write
21-
pull-requests: read
22-
repository-projects: read
23-
statuses: read
2487
runs-on: macos-latest
25-
environment: app-packaging
88+
# Use the protected `app-packaging` environment for real releases (which
89+
# gates access to signing secrets). Dry runs use a separate unprotected
90+
# name because GitHub Actions rejects an empty environment value.
91+
environment: ${{ inputs.dry_run != true && 'app-packaging' || 'app-packaging-dryrun' }}
2692
steps:
2793
- name: Check out Git repository
2894
uses: actions/checkout@v4
95+
2996
- uses: pnpm/action-setup@v4
3097
name: Install pnpm
3198
with:
3299
version: latest
33100
run_install: false
101+
34102
- name: Install Node.js
35103
uses: actions/setup-node@v4
36104
with:
37105
node-version-file: '.nvmrc'
38106
cache: 'pnpm'
107+
39108
- name: Install Dependencies
40-
# CI defaults to --frozen-lockfile, which blocks merging git branch lockfiles.
41-
# pnpm-workspace.yaml's mergeGitBranchLockfilesBranchPattern auto-merges
42-
# branch lockfiles on main. --no-frozen-lockfile allows that merge write.
43109
run: pnpm install --no-frozen-lockfile
44-
- name: Package Desktop Applications
45-
run: make all
110+
111+
- name: Download web build artifacts
112+
uses: actions/download-artifact@v4
113+
with:
114+
name: web-build
115+
116+
- name: Prepare Electron app
117+
run: make dep_electron
118+
119+
- name: Compile local proxies (parallel)
120+
run: |
121+
make compile_localproxy os=macos arch=x64 local_proxy_postfix= &
122+
make compile_localproxy os=macos arch=arm64 local_proxy_postfix= &
123+
wait
124+
125+
- name: Package macOS Desktop Apps (signed)
126+
if: inputs.dry_run != true
127+
run: |
128+
make mac_x64
129+
make mac_arm64
46130
env:
47131
BAI_APP_SIGN: 1
48132
BAI_APP_SIGN_APPLE_TEAM_ID: ${{ secrets.BAI_APP_SIGN_APPLE_TEAM_ID }}
@@ -51,10 +135,72 @@ jobs:
51135
BAI_APP_SIGN_IDENTITY: ${{ secrets.BAI_APP_SIGN_IDENTITY }}
52136
BAI_APP_SIGN_KEYCHAIN_B64: ${{ secrets.BAI_APP_SIGN_KEYCHAIN_B64 }}
53137
BAI_APP_SIGN_KEYCHAIN_PASSWORD: ${{ secrets.BAI_APP_SIGN_KEYCHAIN_PASSWORD }}
54-
NODE_OPTIONS: --max-old-space-size=4096
55-
- name: Bundle static resources into zip package
56-
run: make bundle
57-
- name: Upload application to latest release
138+
139+
- name: Package macOS Desktop Apps (unsigned, dry run)
140+
if: inputs.dry_run == true
141+
run: |
142+
make mac_x64
143+
make mac_arm64
144+
145+
- name: Upload macOS release assets
146+
if: inputs.dry_run != true
147+
run: node upload-release.js app
148+
env:
149+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
150+
151+
# ──────────────────────────────────────────────────────────────────────
152+
# Job 3: Build Windows + Linux desktop apps (ubuntu, ~8 min)
153+
# No code signing needed — can run on cheaper/faster ubuntu runners.
154+
# ──────────────────────────────────────────────────────────────────────
155+
build_desktop:
156+
needs: build_web
157+
permissions:
158+
contents: write
159+
runs-on: ubuntu-latest
160+
steps:
161+
- name: Check out Git repository
162+
uses: actions/checkout@v4
163+
164+
- uses: pnpm/action-setup@v4
165+
name: Install pnpm
166+
with:
167+
version: latest
168+
run_install: false
169+
170+
- name: Install Node.js
171+
uses: actions/setup-node@v4
172+
with:
173+
node-version-file: '.nvmrc'
174+
cache: 'pnpm'
175+
176+
- name: Install Dependencies
177+
run: pnpm install --no-frozen-lockfile
178+
179+
- name: Download web build artifacts
180+
uses: actions/download-artifact@v4
181+
with:
182+
name: web-build
183+
184+
- name: Prepare Electron app
185+
run: make dep_electron
186+
187+
- name: Compile local proxies (parallel)
188+
run: |
189+
make compile_localproxy os=win arch=x64 local_proxy_postfix=.exe &
190+
make compile_localproxy os=win arch=arm64 local_proxy_postfix=.exe &
191+
make compile_localproxy os=linux arch=x64 local_proxy_postfix= &
192+
make compile_localproxy os=linux arch=arm64 local_proxy_postfix= &
193+
wait
194+
195+
- name: Package Windows & Linux Desktop Apps
196+
run: |
197+
make win_x64
198+
make win_arm64
199+
make linux_x64
200+
make linux_arm64
201+
202+
- name: Upload release assets
203+
if: inputs.dry_run != true
58204
run: node upload-release.js app
59205
env:
60206
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

‎Makefile‎

Lines changed: 64 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -61,35 +61,66 @@ compile_client_node_ts: clean_client_node_ts
6161
compile_wsproxy: compile_client_node_ts
6262
@pnpm -w exec webpack-cli --config src/wsproxy/webpack.config.js
6363
all: dep
64+
@make compile_all_localproxy
6465
@make mac_x64
6566
@make mac_arm64
6667
@make win_x64
6768
@make win_arm64
6869
@make linux_x64
6970
@make linux_arm64
7071
@make bundle
71-
dep:
72+
# Build all local proxy binaries in parallel (saves ~3-4 min vs sequential)
73+
compile_all_localproxy:
74+
@printf "$(GREEN)Compiling all local proxy binaries in parallel...$(NC)\n"
75+
@make compile_localproxy os=macos arch=x64 local_proxy_postfix= & \
76+
make compile_localproxy os=macos arch=arm64 local_proxy_postfix= & \
77+
make compile_localproxy os=win arch=x64 local_proxy_postfix=.exe & \
78+
make compile_localproxy os=win arch=arm64 local_proxy_postfix=.exe & \
79+
make compile_localproxy os=linux arch=x64 local_proxy_postfix= & \
80+
make compile_localproxy os=linux arch=arm64 local_proxy_postfix= & \
81+
wait
82+
@printf "$(YELLOW)All local proxy binaries compiled$(NC)\n"
83+
# Build only the web bundle (no Electron setup). Used by CI web job.
84+
# Also ensures src/wsproxy/dist/wsproxy.js exists, since dep_electron copies it.
85+
dep_web:
7286
@if [ ! -f "./config.toml" ]; then \
7387
cp config.toml.sample config.toml; \
7488
fi
7589
@mkdir -p ./app
76-
@if [ ! -d "./build/web/" ] || ! grep -q 'es6://static/js/main' react/build/index.html; then \
90+
@if [ ! -d "./build/web/" ]; then \
7791
make compile; \
92+
fi
93+
@if [ ! -f "./src/wsproxy/dist/wsproxy.js" ]; then \
7894
make compile_wsproxy; \
79-
rm -rf build/electron-app; \
80-
mkdir -p build/electron-app; \
81-
cp -r electron-app/* build/electron-app/;\
82-
cp electron-app/.npmrc build/electron-app/;\
83-
pnpm i --prefix ./build/electron-app --ignore-workspace;\
95+
fi
96+
# Prepare the Electron app directory. Requires dep_web to have run first.
97+
# Uses publicPath patching instead of a full second React build (~4-8 min savings).
98+
#
99+
# Idempotent: skips when `build/electron-app/app/index.html` already carries
100+
# the patched `es6://static/js/main` marker. This mirrors the original
101+
# Makefile's skip semantics so downstream targets that re-declare `dep` as a
102+
# prerequisite (e.g. `mac_x64`, `win_x64`) do not repeatedly re-copy the web
103+
# bundle. Set `FORCE_DEP_ELECTRON=1` to force a re-sync.
104+
dep_electron: dep_web
105+
@if [ -f "./build/electron-app/app/index.html" ] && grep -q 'es6://static/js/main' ./build/electron-app/app/index.html && [ "$(FORCE_DEP_ELECTRON)" != "1" ]; then \
106+
printf "$(YELLOW)Electron app already prepared, skipping$(NC)\n"; \
107+
else \
108+
if [ ! -d "./build/electron-app" ]; then \
109+
mkdir -p build/electron-app; \
110+
cp -r electron-app/* build/electron-app/; \
111+
cp electron-app/.npmrc build/electron-app/; \
112+
pnpm i --prefix ./build/electron-app --ignore-workspace; \
113+
fi; \
114+
rm -rf build/electron-app/app build/electron-app/resources build/electron-app/manifest; \
84115
cp -Rp build/web build/electron-app/app; \
85116
cp -Rp build/web/resources build/electron-app; \
86117
cp -Rp build/web/manifest build/electron-app; \
87-
BUILD_TARGET=electron pnpm run build:react-only; \
88-
cp -Rp react/build/* build/electron-app/app/; \
118+
node scripts/patch-electron-publicpath.js build/electron-app/app; \
89119
mkdir -p ./build/electron-app/app/wsproxy; \
90120
cp ./src/wsproxy/dist/wsproxy.js ./build/electron-app/app/wsproxy/wsproxy.js; \
91121
cp ./preload.js ./build/electron-app/preload.js; \
92122
fi
123+
dep: dep_electron
93124
web:
94125
@if [ ! -d "./build/web/" ];then \
95126
make compile; \
@@ -119,17 +150,32 @@ endif # BAI_APP_SIGN_KEYCHAIN_PASSWORD
119150
echo Keychain ${KEYCHAIN_NAME} created for build
120151
endif # BAI_APP_SIGN_KEYCHAIN_B64
121152
endif # BAI_APP_SIGN_KEYCHAIN
153+
# Concurrency-safe: each (os, arch) build uses a unique staging directory so
154+
# multiple invocations can run in parallel without overwriting each other's
155+
# intermediate file (`backend.ai-local-proxy[.exe]`) packed into the ZIP.
156+
#
157+
# Idempotent: skips rebuild when the output ZIP already exists, so `make all`
158+
# can pre-build everything via `compile_all_localproxy` and downstream targets
159+
# (`mac_x64`, `win_x64`, ...) can reuse the cached artifact without
160+
# re-compiling. Set `FORCE_COMPILE_LOCALPROXY=1` to force a rebuild.
122161
compile_localproxy:
123-
@rm -rf ./app/backend.ai-local-proxy-$(BUILD_VERSION)-$(os)-$(arch)$(local_proxy_postfix)
124-
@pnpm exec pkg ./src/wsproxy/local_proxy.js --targets node18-$(os)-$(arch) --output ./app/backend.ai-local-proxy-$(BUILD_VERSION)-$(os)-$(arch)$(local_proxy_postfix) --compress Brotli
125-
@rm -rf ./app/backend.ai-local-proxy$(local_proxy_postfix); cp ./app/backend.ai-local-proxy-$(BUILD_VERSION)-$(os)-$(arch)$(local_proxy_postfix) ./app/backend.ai-local-proxy$(local_proxy_postfix)
126-
@cd app; zip -r -9 ./backend.ai-local-proxy-$(BUILD_VERSION)-$(os)-$(arch).zip "./backend.ai-local-proxy$(local_proxy_postfix)"
127-
@rm -rf ./app/backend.ai-local-proxy$(local_proxy_postfix)
162+
@if [ -f "./app/backend.ai-local-proxy-$(BUILD_VERSION)-$(os)-$(arch).zip" ] && [ "$(FORCE_COMPILE_LOCALPROXY)" != "1" ]; then \
163+
printf "$(YELLOW)local-proxy $(os)-$(arch) already built, skipping$(NC)\n"; \
164+
else \
165+
rm -rf ./app/backend.ai-local-proxy-$(BUILD_VERSION)-$(os)-$(arch)$(local_proxy_postfix); \
166+
pnpm exec pkg ./src/wsproxy/local_proxy.js --targets node18-$(os)-$(arch) --output ./app/backend.ai-local-proxy-$(BUILD_VERSION)-$(os)-$(arch)$(local_proxy_postfix) --compress Brotli; \
167+
rm -rf ./app/_lp-stage-$(os)-$(arch); \
168+
mkdir -p ./app/_lp-stage-$(os)-$(arch); \
169+
cp ./app/backend.ai-local-proxy-$(BUILD_VERSION)-$(os)-$(arch)$(local_proxy_postfix) ./app/_lp-stage-$(os)-$(arch)/backend.ai-local-proxy$(local_proxy_postfix); \
170+
rm -f ./app/backend.ai-local-proxy-$(BUILD_VERSION)-$(os)-$(arch).zip; \
171+
(cd ./app/_lp-stage-$(os)-$(arch); zip -r -6 ../backend.ai-local-proxy-$(BUILD_VERSION)-$(os)-$(arch).zip "./backend.ai-local-proxy$(local_proxy_postfix)"); \
172+
rm -rf ./app/_lp-stage-$(os)-$(arch); \
173+
fi
128174
package_zip:
129175
@printf "$(GREEN)Packaging as ZIP archive...$(NC)"
130176
@cp ./configs/$(site).toml ./build/electron-app/app/config.toml
131177
@node ./app-packager.js $(os) $(arch)
132-
@cd app; zip -r -9 ./backend.ai-desktop-$(os)-$(arch)-$(BUILD_DATE).zip "./Backend.AI Desktop-$(os_api)-$(arch)"
178+
@cd app; zip -r -6 ./backend.ai-desktop-$(os)-$(arch)-$(BUILD_DATE).zip "./Backend.AI Desktop-$(os_api)-$(arch)"
133179
ifeq ($(site),main)
134180
@mv ./app/backend.ai-desktop-$(os)-$(arch)-$(BUILD_DATE).zip ./app/backend.ai-desktop-$(BUILD_VERSION)-$(os)-$(arch).zip
135181
else
@@ -152,9 +198,10 @@ else
152198
@mv ./app/backend.ai-desktop-$(arch)-$(BUILD_DATE).dmg ./app/backend.ai-desktop-$(BUILD_VERSION)-$(site)-$(os)-$(arch).dmg
153199
endif
154200
@printf "$(YELLOW)Finished$(NC)\n"
155-
bundle: dep
201+
bundle: dep_web
156202
@printf "$(GREEN)Bundling...$(NC)"
157-
@cd build/web; zip -r -9 ../../app/backend.ai-webui-bundle-$(BUILD_DATE).zip . > /dev/null
203+
@mkdir -p ./app
204+
@cd build/web; zip -r -6 ../../app/backend.ai-webui-bundle-$(BUILD_DATE).zip . > /dev/null
158205
@mv ./app/backend.ai-webui-bundle-$(BUILD_DATE).zip ./app/backend.ai-webui-bundle-$(BUILD_VERSION).zip
159206
@printf "$(YELLOW)Finished$(NC)\n"
160207
mac: dep

‎react/package.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -88,7 +88,7 @@
8888
"scripts": {
8989
"start": "NODE_OPTIONS='--max-old-space-size=4096' craco start",
9090
"build": "pnpm run build:only && cp -r ./build/* ../build/web/",
91-
"build:only": "pnpm run relay && craco build",
91+
"build:only": "NODE_OPTIONS='--max-old-space-size=4096' pnpm run relay && NODE_OPTIONS='--max-old-space-size=4096' craco build",
9292
"test": "NODE_OPTIONS='$NODE_OPTIONS --no-deprecation --experimental-vm-modules' jest",
9393
"eject": "react-scripts eject",
9494
"relay": "relay-compiler",

0 commit comments

Comments
 (0)