Skip to content

Commit 19d129b

Browse files
hsyl20claude
hsyl20
and
claude
committed
Run plinth-test in its own CI job consuming a bindist
Split the Linux CI into two jobs for a separate status check and timeout budget: plinth-build-linux builds the compiler and plinth-test-linux runs the test against it. The test job can't run in parallel (it needs the built compiler), so it depends on the build job and downloads the artifact instead of rebuilding GHC. To hand the compiler between jobs, plinth-build.sh gains a BINDIST=1 mode that produces the fixed-up uplc-ghc bindist + .tar.xz archive while keeping the lean dev flavour; previously this only happened for RELEASE=1, which also pulled in docs and the release flavour. The build job uploads the archive and the test job installs it with the usual configure + make install (the bindist's wrappers/ dir, written by the fixup, makes this produce bin/uplc-ghc), then points plinth-test.sh at it via the now-overridable GHC variable. The lean dev flavour is plain "release" (no +assertions): an assertions GHC is built with -DDEBUG, which slows compilation of GHC itself and of every package built with it (the Plinth test project). Compiling the test packages with the Plinth plugin (Core -> PLC) is memory-heavy. The test's cabal parallelism is configurable via JOBS (default "-j", all cores); the CI test job sets JOBS=1 to build serially. Even so, a single makeLift-heavy module (PlutusLedgerApi.V3.Contexts) spikes past the ~16 GB runner RAM, so the test job adds a 16 GB swapfile on /mnt to let the spike spill to swap instead of being OOM-killed. The artifact name is keyed by matrix.os + matrix.ghc so build matrix legs never collide within a run; the test job mirrors the build matrix so the keys line up per leg. The shared Linux setup (free disk, /mnt bind-mount, submodules, GHC/cabal) moves into a .github/actions/prepare composite action. The GITHUB_TOKEN is restricted to contents: read (least privilege, flagged by CodeQL). Now that the plutus submodule (ghc-plinth-plutus) is public, its .gitmodules URL is switched from SSH to HTTPS and all the SSH plumbing (SUBMODULE_SSH_KEY secret, ssh-keyscan setup) is removed; every submodule is now an anonymous HTTPS clone. Jobs are renamed for coherence: plinth-build-linux, plinth-test-linux, plinth-build-windows (plinth-test-windows to follow). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent d02de8c commit 19d129b

5 files changed

Lines changed: 206 additions & 85 deletions

File tree

‎.github/actions/prepare/action.yml‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
name: Prepare Linux build environment
2+
description: >-
3+
Free disk space, bind-mount the large /mnt disk as _build, fetch submodules
4+
and set up GHC/cabal. Shared by the cabal (build) and plinth-test jobs. The
5+
repo must already be checked out so the runner can find this local action.
6+
7+
inputs:
8+
ghc-version:
9+
description: GHC version passed to haskell-actions/setup
10+
required: true
11+
12+
runs:
13+
using: composite
14+
steps:
15+
16+
- name: Free disk space
17+
uses: jlumbroso/free-disk-space@main
18+
with:
19+
# this might remove tools that are actually needed,
20+
# if set to "true" but frees about 6 GB
21+
tool-cache: false
22+
23+
# all of these default to true, but feel free to set to
24+
# "false" if necessary for your workflow
25+
android: true
26+
dotnet: true
27+
haskell: false
28+
# left off: it is slow (~1-2 min) and the bulk of the build now lives
29+
# on /mnt, so reclaiming large packages on / is not needed.
30+
large-packages: false
31+
docker-images: true
32+
swap-storage: true
33+
34+
# The runner's workspace disk (/) has ~21 GB free; the ephemeral disk at
35+
# /mnt has ~70 GB. Build everything under _build on /mnt to avoid running
36+
# out of space. We bind-mount (rather than symlink) so _build stays a real
37+
# directory: a symlinked build root can confuse path tracking in
38+
# Hadrian/Cabal.
39+
- name: Build on the large /mnt disk
40+
shell: bash
41+
run: |
42+
sudo mkdir -p /mnt/_build
43+
mkdir -p "$GITHUB_WORKSPACE/_build"
44+
sudo mount --bind /mnt/_build "$GITHUB_WORKSPACE/_build"
45+
sudo chown "$USER" "$GITHUB_WORKSPACE/_build"
46+
47+
- name: Init and update submodules
48+
shell: bash
49+
run: |
50+
git submodule sync
51+
# Shallow (single-commit) submodule clones: the build never needs
52+
# submodule history, and these 29 submodules (notably plutus) carry a
53+
# lot of it. --jobs fetches them in parallel. GitHub allows fetching
54+
# the pinned SHA directly, so depth 1 works even when it isn't a
55+
# branch tip.
56+
git submodule update --init --recursive --depth 1 --jobs 4
57+
58+
- uses: haskell-actions/setup@v2
59+
name: Setup Haskell tools
60+
with:
61+
ghc-version: ${{ inputs.ghc-version }}
62+
cabal-version: "latest"
63+
cabal-update: true

‎.github/workflows/ci.yml‎

Lines changed: 112 additions & 67 deletions
Original file line numberDiff line numberDiff line change
@@ -7,9 +7,15 @@ on:
77
branches: [master, ghc-9.6-plinth]
88
workflow_dispatch:
99

10+
# Least-privilege GITHUB_TOKEN: jobs only check out code (contents: read).
11+
# Submodules use a dedicated SSH key and artifacts use the Actions runtime
12+
# token, so no write scopes are needed.
13+
permissions:
14+
contents: read
15+
1016
jobs:
11-
cabal:
12-
name: ${{ matrix.os }} / ghc ${{ matrix.ghc }}
17+
plinth-build-linux:
18+
name: plinth-build / ${{ matrix.os }} / ghc ${{ matrix.ghc }}
1319
runs-on: ${{ matrix.os }}
1420
strategy:
1521
fail-fast: false
@@ -19,73 +25,119 @@ jobs:
1925

2026
steps:
2127

22-
- name: Free disk space
23-
uses: jlumbroso/free-disk-space@main
24-
with:
25-
# this might remove tools that are actually needed,
26-
# if set to "true" but frees about 6 GB
27-
tool-cache: false
28-
29-
# all of these default to true, but feel free to set to
30-
# "false" if necessary for your workflow
31-
android: true
32-
dotnet: true
33-
haskell: false
34-
# left off: it is slow (~1-2 min) and the bulk of the build now lives
35-
# on /mnt, so reclaiming large packages on / is not needed.
36-
large-packages: false
37-
docker-images: true
38-
swap-storage: true
39-
28+
# checkout first: the local composite action (.github/actions/prepare) must
29+
# be on disk before the runner can resolve it.
4030
- uses: actions/checkout@v4
4131
with:
4232
submodules: false
4333

44-
# The runner's workspace disk (/) has ~21 GB free; the ephemeral disk at
45-
# /mnt has ~70 GB. Build everything under _build on /mnt to avoid running
46-
# out of space. Both plinth-build.sh and plinth-test.sh write under _build.
47-
# We bind-mount (rather than symlink) so _build stays a real directory:
48-
# a symlinked build root can confuse path tracking in Hadrian/Cabal.
49-
- name: Build on the large /mnt disk
50-
run: |
51-
sudo mkdir -p /mnt/_build
52-
mkdir -p "$GITHUB_WORKSPACE/_build"
53-
sudo mount --bind /mnt/_build "$GITHUB_WORKSPACE/_build"
54-
sudo chown "$USER" "$GITHUB_WORKSPACE/_build"
34+
- uses: ./.github/actions/prepare
35+
with:
36+
ghc-version: ${{ matrix.ghc }}
5537

56-
- name: Setup SSH for submodules
57-
run: |
58-
mkdir -p ~/.ssh
59-
echo "${{ secrets.SUBMODULE_SSH_KEY }}" > ~/.ssh/id_rsa
60-
chmod 600 ~/.ssh/id_rsa
61-
ssh-keyscan github.com >> ~/.ssh/known_hosts
38+
# BINDIST=1 produces the fixed-up uplc-ghc bindist (+ .tar.xz archive) while
39+
# keeping the lean dev flavour. The plinth-test job consumes that archive.
40+
- name: Build uplc-ghc bindist
41+
run: BINDIST=1 ./plinth-build.sh
6242

63-
- name: Init and update submodules
64-
run: |
65-
git submodule sync
66-
# Shallow (single-commit) submodule clones: the build never needs
67-
# submodule history, and these 29 submodules (notably plutus) carry a
68-
# lot of it. --jobs fetches them in parallel. GitHub allows fetching
69-
# the pinned SHA directly, so depth 1 works even when it isn't a
70-
# branch tip.
71-
git submodule update --init --recursive --depth 1 --jobs 4
43+
- name: Upload uplc-ghc bindist
44+
uses: actions/upload-artifact@v4
45+
with:
46+
# key by platform + GHC so multiple build matrix legs (or another job)
47+
# never collide on the artifact name within a single run. See the
48+
# matching download in the plinth-test job, which shares this matrix.
49+
name: plinth-bindist-${{ matrix.os }}-${{ matrix.ghc }}
50+
path: _build/bindist/*.tar.xz
51+
retention-days: 1
52+
# the archive is already xz-compressed; don't re-zip it
53+
compression-level: 0
54+
55+
# plinth-test runs in its own job for a separate status check and timeout
56+
# budget. It depends on `plinth-build-linux` (it can't run in parallel: it
57+
# needs the built compiler), downloading the bindist archive instead of
58+
# rebuilding GHC. Its matrix mirrors the build so the artifact-name key
59+
# matches per leg.
60+
plinth-test-linux:
61+
name: plinth-test / ${{ matrix.os }} / ghc ${{ matrix.ghc }}
62+
needs: plinth-build-linux
63+
runs-on: ${{ matrix.os }}
64+
strategy:
65+
fail-fast: false
66+
matrix:
67+
os: [ubuntu-latest]
68+
ghc: ['9.6.7']
7269

73-
- uses: haskell-actions/setup@v2
74-
id: setup
75-
name: Setup Haskell tools
70+
steps:
71+
72+
# checkout first: the local composite action (.github/actions/prepare) must
73+
# be on disk before the runner can resolve it. The prepare action also
74+
# fetches submodules, which the test needs: plinth/test/cabal.project pulls
75+
# plutus-tx/plutus-core/plutus-ledger-api from the plutus submodule.
76+
- uses: actions/checkout@v4
7677
with:
77-
ghc-version: ${{ matrix.ghc }}
78-
cabal-version: "latest"
79-
cabal-update: true
78+
submodules: false
8079

81-
- name: Build uplc-ghc
82-
run: ./plinth-build.sh
80+
- uses: ./.github/actions/prepare
81+
with:
82+
ghc-version: ${{ matrix.ghc }}
8383

84+
# A single Plinth plugin (Core -> PLC) compile of a makeLift-heavy module
85+
# (PlutusLedgerApi.V3.Contexts) spikes past the ~16 GB runner RAM and gets
86+
# OOM-killed even with JOBS=1. Add a 16 GB swapfile on the large /mnt disk
87+
# so the spike spills to swap and the compile completes.
88+
- name: Add 16 GB swap
89+
run: |
90+
sudo fallocate -l 16G /mnt/swapfile
91+
sudo chmod 600 /mnt/swapfile
92+
sudo mkswap /mnt/swapfile
93+
sudo swapon /mnt/swapfile
94+
swapon --show
95+
free -h
96+
97+
- name: Download uplc-ghc bindist
98+
uses: actions/download-artifact@v4
99+
with:
100+
# matches the plinth-build upload name; both jobs share the same matrix.
101+
name: plinth-bindist-${{ matrix.os }}-${{ matrix.ghc }}
102+
path: _build/artifact
103+
104+
# Install the bindist with the usual configure + make install; the bindist's
105+
# wrappers/ dir (added by the BINDIST fixup) makes this produce bin/uplc-ghc.
106+
# Each step is guarded so a missing artifact or failed install fails loudly
107+
# here, not with a confusing error deep in the test.
108+
- name: Install uplc-ghc bindist
109+
run: |
110+
shopt -s nullglob
111+
archives=(_build/artifact/*.tar.xz)
112+
if [ ${#archives[@]} -ne 1 ]; then
113+
echo "error: expected exactly one bindist archive, found: ${archives[*]:-none}"
114+
exit 1
115+
fi
116+
mkdir -p _build/bindist
117+
echo "extracting ${archives[0]}"
118+
tar --xz -xf "${archives[0]}" -C _build/bindist
119+
bindirs=(_build/bindist/ghc-*)
120+
if [ ${#bindirs[@]} -ne 1 ]; then
121+
echo "error: expected one extracted bindist dir, found: ${bindirs[*]:-none}"
122+
exit 1
123+
fi
124+
( cd "${bindirs[0]}" && ./configure --prefix="$GITHUB_WORKSPACE/_build/install" && make install )
125+
uplc="$GITHUB_WORKSPACE/_build/install/bin/uplc-ghc"
126+
if [ ! -x "$uplc" ]; then
127+
echo "error: $uplc missing or not executable after make install"
128+
ls -l "$GITHUB_WORKSPACE/_build/install/bin" || true
129+
exit 1
130+
fi
131+
rm -rf _build/artifact
132+
133+
# Build packages serially (JOBS=1): compiling with the Plinth plugin
134+
# (Core -> PLC) is memory-heavy, and concurrent GHC processes OOM-kill the
135+
# runner (seen mid-compile on PlutusLedgerApi.V3).
84136
- name: Test uplc-ghc
85-
run: ./plinth-test.sh
137+
run: JOBS=1 GHC="$GITHUB_WORKSPACE/_build/install/bin/uplc-ghc" ./plinth-test.sh
86138

87-
windows:
88-
name: x86_64-windows / ghc ${{ matrix.ghc }}
139+
plinth-build-windows:
140+
name: plinth-build / windows-latest / ghc ${{ matrix.ghc }}
89141
runs-on: windows-latest
90142
strategy:
91143
fail-fast: false
@@ -107,16 +159,9 @@ jobs:
107159
with:
108160
submodules: false
109161

110-
# The pre-MSYS2 steps run in the default Git-for-Windows bash because the
111-
# MSYS2 environment is not set up yet.
112-
- name: Setup SSH for submodules
113-
shell: bash
114-
run: |
115-
mkdir -p ~/.ssh
116-
echo "${{ secrets.SUBMODULE_SSH_KEY }}" > ~/.ssh/id_rsa
117-
chmod 600 ~/.ssh/id_rsa
118-
ssh-keyscan github.com >> ~/.ssh/known_hosts
119-
162+
# The pre-MSYS2 step runs in the default Git-for-Windows bash because the
163+
# MSYS2 environment is not set up yet. All submodules are public HTTPS, so
164+
# no credentials are needed.
120165
- name: Init and update submodules
121166
shell: bash
122167
run: |

‎.gitmodules‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -112,4 +112,4 @@
112112
url = https://gitlab.haskell.org/ghc/packages/exceptions.git
113113
[submodule "plutus"]
114114
path = plutus
115-
url = git@github.com:input-output-hk/ghc-plinth-plutus.git
115+
url = https://github.com/input-output-hk/ghc-plinth-plutus.git

‎plinth-build.sh‎

Lines changed: 20 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -14,12 +14,14 @@ DEV_HADRIAN_ARGS="--docs=none"
1414
RELEASE_FLAVOUR="release"
1515
# Note [Lean CI flavour]
1616
# ~~~~~~~~~~~~~~~~~~~~~~~
17-
# debug_info/debug_ghc compile GHC and its libraries with DWARF (-g3), which
18-
# inflates the build tree 2-3x and is the main disk-space hog on CI runners.
19-
# They only help when debugging GHC itself, so the dev/CI flavour keeps just
20-
# "assertions" (cheap, still catches GHC/plugin bugs). Use RELEASE=1 for a
21-
# clean release build.
22-
DEV_FLAVOUR="release+assertions"
17+
# The dev/CI flavour is just "release":
18+
# - debug_info/debug_ghc compile GHC and its libraries with DWARF (-g3), which
19+
# inflates the build tree 2-3x and is the main disk-space hog on CI runners;
20+
# - +assertions builds GHC with -DDEBUG, which slows compilation of GHC itself
21+
# *and* of every package built with the resulting compiler (i.e. the Plinth
22+
# test project). Dropping it keeps CI within its time budget.
23+
# Use RELEASE=1 for a clean release build (adds docs etc.).
24+
DEV_FLAVOUR="release"
2325

2426
# platform-specific default configure arguments
2527
case "$UNAME_S" in
@@ -36,6 +38,7 @@ DEV_CONFIGURE_ARGS="$DEFAULT_CONFIGURE_ARGS"
3638

3739
: ${REBUILD:=0} # set to 1 to force rebuild
3840
: ${RELEASE:=0} # set to 1 to build release version including documentation (more build dependencies)
41+
: ${BINDIST:=0} # set to 1 to produce the fixed-up uplc-ghc bindist + archive while keeping the dev (assertions) flavour
3942

4043
# program locations
4144
: ${GHC:=$(command -v ghc-9.6.7 2>/dev/null || true)}
@@ -237,10 +240,12 @@ DEST_UPLC_GHC="$BASE/_build/stage1/bin/uplc-ghc${EXE_EXT}"
237240
# add to build dir (needed for testing, so always done)
238241
cp "$SRC_UPLC_GHC" "$DEST_UPLC_GHC"
239242

240-
# The bindist fixup below only matters for the shippable bindist, which is
241-
# only produced for RELEASE=1. Skipping it on CI avoids duplicating the
242-
# whole bindist tree on disk. See Note [Lean CI flavour].
243-
if [ "$RELEASE" -ne 1 ]; then
243+
# The bindist fixup below only matters for the shippable bindist, produced
244+
# for RELEASE=1 (release flavour + docs) or BINDIST=1 (current flavour, e.g.
245+
# the lean CI flavour - see Note [Lean CI flavour]). The CI test job consumes
246+
# this bindist, so it builds with BINDIST=1. Otherwise skip the fixup to avoid
247+
# duplicating the whole bindist tree on disk.
248+
if [ "$RELEASE" -ne 1 ] && [ "$BINDIST" -ne 1 ]; then
244249
exit 0
245250
fi
246251

@@ -285,11 +290,12 @@ DEST_UPLC_GHC="$BASE/_build/stage1/bin/uplc-ghc${EXE_EXT}"
285290
done
286291
)
287292

288-
# create bindist archive after fixup (release only: the tar|xz is slow and
289-
# needs extra disk; CI only needs _build/stage1/bin/uplc-ghc for testing).
290-
if [ "$RELEASE" -eq 1 ]; then
293+
# create bindist archive after fixup. The tar|xz is slow and needs extra disk,
294+
# so only do it for a shippable bindist: RELEASE=1, or BINDIST=1 to hand the
295+
# bindist to the CI test job. -T0 lets xz use all cores to keep CI time down.
296+
if [ "$RELEASE" -eq 1 ] || [ "$BINDIST" -eq 1 ]; then
291297
echo "creating bindist archive..."
292298
BINDIST_NAME="ghc-$VERSION-$TARGET_PLATFORM"
293-
(cd "$BASE/_build/bindist" && "$TAR" -cf - "$BINDIST_NAME" | "$XZ" > "$BINDIST_NAME.tar.xz")
299+
(cd "$BASE/_build/bindist" && "$TAR" -cf - "$BINDIST_NAME" | "$XZ" -T0 > "$BINDIST_NAME.tar.xz")
294300
fi
295301

‎plinth-test.sh‎

Lines changed: 10 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,11 @@
22
set -euo pipefail
33

44
: ${CLEAN:=0} # set to 1 to force rebuild
5+
# Number of parallel cabal jobs. Empty means "-j" (all cores). Compiling with
6+
# the Plinth plugin (Core -> PLC) is memory-heavy, so on RAM-constrained
7+
# machines (e.g. CI runners) several concurrent GHC processes can OOM. Set
8+
# JOBS=1 there to build packages serially.
9+
: ${JOBS:=}
510

611

712
# build Plinth test project
@@ -13,10 +18,12 @@ case "$UNAME_S" in
1318
*) EXE_EXT="" ;;
1419
esac
1520

16-
GHC="$PWD/_build/stage1/bin/uplc-ghc${EXE_EXT}"
21+
# Default to the in-tree build; override GHC to point at an installed bindist
22+
# (e.g. the CI test job consumes the bindist produced by `BINDIST=1 plinth-build.sh`).
23+
: ${GHC:=$PWD/_build/stage1/bin/uplc-ghc${EXE_EXT}}
1724

1825
if [ ! -x "$GHC" ]; then
19-
echo "Plinth GHC not found. Please run plinth-build.sh first."
26+
echo "Plinth GHC not found ($GHC). Please run plinth-build.sh first."
2027
exit 1
2128
fi
2229

@@ -32,7 +39,7 @@ CABAL_ARGS="\
3239
--logs-dir=_build/logs"
3340

3441
CABAL_BUILD_ARGS="\
35-
-j -w ${GHC} \
42+
-j${JOBS} -w ${GHC} \
3643
--builddir=_build/build \
3744
--ghc-options=\"-fhide-source-paths\""
3845

0 commit comments

Comments
 (0)