Skip to content

Commit e73f75e

Browse files
ryanking13pre-commit-ci[bot]
ryanking13
and
pre-commit-ci[bot]
authored
TST Add integration tests for building recipes (#16)
This adds an integration test for `pyodide-build`, which builds a few selected recipes. Unlike unit test, integration tests are not run in CI by default. So users ned to run it locally, or include `[integration]` in the commit message to trigger the test. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
1 parent fac0109 commit e73f75e

File tree

10 files changed

+231
-1
lines changed

10 files changed

+231
-1
lines changed

‎.github/workflows/main.yml

+55
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,61 @@ jobs:
6464
with:
6565
fail_ci_if_error: false
6666

67+
check-integration-test-trigger:
68+
name: test-integration-test-trigger
69+
runs-on: ubuntu-latest
70+
outputs:
71+
run-integration-test: ${{ steps.check-integration-test-trigger.outputs.trigger }}
72+
73+
steps:
74+
- uses: actions/checkout@v4
75+
with:
76+
ref: ${{ github.event.pull_request.head.sha }}
77+
78+
- id: check-integration-test-trigger
79+
name: Check integration test trigger
80+
run: bash tools/check_integration_test_trigger.sh
81+
82+
integration-test:
83+
runs-on: ubuntu-latest
84+
needs: [check-integration-test-trigger]
85+
if: needs.check-integration-test-trigger.outputs.run-integration-test
86+
steps:
87+
- uses: actions/checkout@v4
88+
with:
89+
# include tags so that hatch-vcs can infer the version
90+
fetch-depth: 0
91+
92+
- name: Setup Python
93+
uses: actions/setup-python@v4
94+
with:
95+
python-version: "3.12"
96+
97+
- name: Set up Node.js
98+
uses: actions/setup-node@v4
99+
with:
100+
node-version: "20"
101+
102+
- name: Install the package
103+
run: |
104+
python -m pip install --upgrade pip
105+
python -m pip install .
106+
107+
- name: Install xbuildenv
108+
run: |
109+
pyodide xbuildenv install
110+
echo EMSCRIPTEN_VERSION=$(pyodide config get emscripten_version) >> $GITHUB_ENV
111+
112+
- name: Install Emscripten
113+
uses: mymindstorm/setup-emsdk@v12
114+
with:
115+
version: ${{ env.EMSCRIPTEN_VERSION }}
116+
117+
- name: Run the integration tests (recipes)
118+
run: |
119+
cd integration_tests
120+
make test-recipe
121+
67122
publish:
68123
name: >-
69124
Publish to PyPI

‎.gitignore

+7-1
Original file line numberDiff line numberDiff line change
@@ -21,10 +21,16 @@ build
2121
ccache
2222
dist*/
2323
docs/_build/
24-
emsdk/emsdk
24+
emsdk
2525
geckodriver.log
2626
node_modules
2727
dist/
2828
pyodide_build/**/build.log
2929
xbuildenv/
3030
.pyodide-xbuildenv*
31+
pyodide/
32+
venv/
33+
34+
# integration tests
35+
integration_tests/**/build.log
36+
integration_tests/**/.libs

‎integration_tests/Makefile

+26
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
all:
2+
@echo "Please specify a target"
3+
@exit 1
4+
5+
.PHONY: test-recipe
6+
test-recipe: check
7+
@echo "... Running integration tests for building recipes"
8+
9+
pyodide build-recipes --recipe-dir=recipes --force-rebuild "*"
10+
11+
@echo "... Passed"
12+
13+
.PHONY: check
14+
check:
15+
@echo "... Checking dependencies"
16+
17+
@which pyodide > /dev/null || (echo "pyodide-build is not installd"; exit 1)
18+
@which emsdk > /dev/null || (echo "emscripten is not installed"; exit 1)
19+
20+
@echo "... Passed"
21+
22+
.PHONY: clean
23+
clean:
24+
rm -rf .pyodide-xbuildenv*
25+
rm -rf recipes/*/build
26+
rm -rf dist

‎integration_tests/README.md

+11
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
This directory contains assets to run integration tests for pyodide-build.
2+
3+
Tests in this directory are not run by default in CI. To run these tests, add `[integration]` in the commit message.
4+
5+
## Running integration tests locally
6+
7+
To run the integration tests locally, use `make` command in this directory.
8+
9+
```bash
10+
make test-recipe
11+
```

‎integration_tests/recipes/README.md

+13
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
This directory contains a few curated recipes to test the build process of pyodide-build.
2+
3+
### List of recipes and their purpose
4+
5+
- numpy: The most popular and widely used library in the scientific computing community.
6+
- orjson: Tests rust extension modules.
7+
- zlib: Tests static libraries.
8+
- geos: Tests shared libraries.
9+
10+
### For maintainers
11+
12+
- Do not put too many recipes in this directory. It is meant to be a small collection of recipes that are representative of the build process.
13+
- The recipes in this directory is originally copied from `pyodide/pyodide`. It does not need to be updated frequently unless there is a change in the build process.

‎integration_tests/recipes/geos/meta.yaml

+28
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
package:
2+
name: geos
3+
version: 3.12.1
4+
tag:
5+
- library
6+
source:
7+
url: https://github.com/libgeos/geos/releases/download/3.12.1/geos-3.12.1.tar.bz2
8+
sha256: d6ea7e492224b51193e8244fe3ec17c4d44d0777f3c32ca4fb171140549a0d03
9+
10+
build:
11+
type: shared_library
12+
script: |
13+
LDFLAGS="${SIDE_MODULE_LDFLAGS}" emcmake cmake \
14+
-DDISABLE_GEOS_INLINE=ON \
15+
-DBUILD_TESTING=OFF \
16+
-DBUILD_BENCHMARKS=OFF \
17+
-DBUILD_DOCUMENTATION=OFF \
18+
-DBUILD_GEOSOP=OFF \
19+
-DCMAKE_C_FLAGS="-fPIC" \
20+
-DCMAKE_CXX_FLAGS="-fPIC" \
21+
-DCMAKE_INSTALL_PREFIX=${WASM_LIBRARY_DIR} \
22+
./
23+
24+
emmake make -j ${PYODIDE_JOBS:-3}
25+
emmake make install
26+
27+
mkdir -p dist
28+
cp ${WASM_LIBRARY_DIR}/lib/libgeos* dist/

‎integration_tests/recipes/numpy/meta.yaml

+33
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
package:
2+
name: numpy
3+
version: 1.26.4
4+
tag:
5+
- min-scipy-stack
6+
top-level:
7+
- numpy
8+
source:
9+
url: https://files.pythonhosted.org/packages/65/6e/09db70a523a96d25e115e71cc56a6f9031e7b8cd166c1ac8438307c14058/numpy-1.26.4.tar.gz
10+
sha256: 2a02aba9ed12e4ac4eb3ea9421c420301a0c6460d9830d74a9df87efa4912010
11+
12+
build:
13+
# numpy uses vendored meson, so we need to pass the cross file manually
14+
backend-flags: |
15+
setup-args=-Dallow-noblas=true
16+
setup-args=--cross-file=${MESON_CROSS_FILE}
17+
# numpy creates numpy/distutils/__pycache__ directory during the build.
18+
# It breaks our test because there is a .pyc in the directory.
19+
post: |
20+
rm -rf numpy/distutils/__pycache__
21+
cflags: |
22+
-Wno-return-type
23+
cross-build-env: true
24+
cross-build-files:
25+
- numpy/core/include/numpy/numpyconfig.h
26+
- numpy/core/include/numpy/_numpyconfig.h
27+
- numpy/core/lib/libnpymath.a
28+
- numpy/random/lib/libnpyrandom.a
29+
about:
30+
home: https://www.numpy.org
31+
PyPI: https://pypi.org/project/numpy
32+
summary: NumPy is the fundamental package for array computing with Python.
33+
license: BSD

‎integration_tests/recipes/orjson/meta.yaml

+20
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
package:
2+
name: orjson
3+
version: 3.10.1
4+
top-level:
5+
- orjson
6+
source:
7+
url: https://files.pythonhosted.org/packages/f5/af/0daa12a907215a5af6d97db8adf301ef14a1b1c651f7e176ee04e0998433/orjson-3.10.1.tar.gz
8+
sha256: a883b28d73370df23ed995c466b4f6c708c1f7a9bdc400fe89165c96c7603204
9+
requirements:
10+
executable:
11+
- rustup
12+
build:
13+
script: export RUSTFLAGS="-Z link-native-libraries=yes"
14+
about:
15+
home: https://github.com/ijl/orjson
16+
PyPI: https://pypi.org/project/orjson
17+
summary:
18+
Fast, correct Python JSON library supporting dataclasses, datetimes, and
19+
numpy
20+
license: Apache-2.0 OR MIT

‎integration_tests/recipes/zlib/meta.yaml

+25
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
package:
2+
name: zlib
3+
version: 1.3.1
4+
tag:
5+
- library
6+
source:
7+
sha256: 9a93b2b7dfdac77ceba5a558a580e74667dd6fede4585b91eefb60f03b72df23
8+
url: https://github.com/madler/zlib/releases/download/v1.3.1/zlib-1.3.1.tar.gz
9+
10+
build:
11+
type: static_library
12+
script: |
13+
mkdir -p "build"
14+
pushd "build"
15+
LDFLAGS="${SIDE_MODULE_LDFLAGS}" emcmake cmake \
16+
-DBUILD_TESTING=OFF \
17+
-DBUILD_BENCHMARKS=OFF \
18+
-DBUILD_DOCUMENTATION=OFF \
19+
-DCMAKE_C_FLAGS="-fPIC -Wno-deprecated-non-prototype" \
20+
-DCMAKE_CXX_FLAGS="-fPIC -Wno-deprecated-non-prototype" \
21+
-DCMAKE_INSTALL_PREFIX=${WASM_LIBRARY_DIR} \
22+
../
23+
emmake make -j ${PYODIDE_JOBS:-3}
24+
emmake make install
25+
popd

‎tools/check_integration_test_trigger.sh

+13
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
#!/bin/bash
2+
3+
set -e
4+
set -x
5+
6+
COMMIT_MSG=$(git log --no-merges -1 --oneline)
7+
8+
# The integration tests will be triggered on push or on pull_request when the commit
9+
# message contains "[integration]"
10+
if [[ "$GITHUB_EVENT_NAME" == push ||
11+
"$COMMIT_MSG" =~ \[integration\] ]]; then
12+
echo "trigger=true" >> "$GITHUB_OUTPUT"
13+
fi

0 commit comments

Comments
 (0)