Tests

v0.2.0 (honest beta): merge train complete, gates green, real jobs verified #299

Workflow file for this run

	name: Tests

	on:
	push:
	branches: [ master, main, develop ]
	pull_request:
	branches: [ master, main, develop ]

	permissions:
	contents: write
	pull-requests: read

	jobs:
	test:
	runs-on: ${{ matrix.os }}
	# 15 minutes was set when this job ran only tests/unit -- about 350 tests.
	# It now runs the whole non-billable suite, ~1790 tests (issue #113).
	# Linux finishes in ~10 minutes and macOS in ~14; Windows reached 96% and
	# was cancelled mid-run, because process-spawning tests (the serialization
	# round-trips each start a fresh interpreter) are markedly slower there.
	# This is a bound on a genuinely longer job, not a relaxed check: every
	# test still has to pass, and pytest's own --timeout=120 still bounds any
	# individual test that wedges.
	timeout-minutes: 30
	strategy:
	# Show every platform's failures in one run. With the default
	# fail-fast, one job failing cancelled the other six, so a
	# Python-3.12-only problem looked like a total outage and hid whatever
	# the other combinations would have reported.
	fail-fast: false
	matrix:
	os: [ubuntu-latest, windows-latest, macos-latest]
	# 3.8 and 3.9 were never actually supported: click, requests,
	# huggingface_hub and python-dotenv all require >=3.10 at their current
	# versions, so those jobs installed years-old dependencies or failed
	# outright. 3.8 has been end-of-life since October 2024. (See #110.)
	python-version: ['3.10', '3.11', '3.12']
	exclude:
	# Reduce matrix size by excluding some combinations
	- os: windows-latest
	python-version: '3.10'
	- os: macos-latest
	python-version: '3.10'

	steps:
	- uses: actions/checkout@v4

	- name: Set up Python ${{ matrix.python-version }}
	uses: actions/setup-python@v5
	with:
	python-version: ${{ matrix.python-version }}

	- name: Install dependencies
	run: \|
	python -m pip install --upgrade pip
	pip install -e ".[dev,test,widget]"

	# Linting is version-independent, so run it once on the primary target
	# rather than on every matrix combination.
	- name: Lint with black
	if: matrix.os == 'ubuntu-latest' && matrix.python-version == '3.11'
	run: \|
	black --check clustrix/ tests/ scripts/

	# No --exit-zero and no inline --extend-ignore: the flags made this step
	# incapable of failing, and the inline list was a second, divergent copy
	# of the project's lint policy. .flake8 is the single source of truth.
	- name: Lint with flake8
	if: matrix.os == 'ubuntu-latest' && matrix.python-version == '3.11'
	run: \|
	flake8 clustrix/ tests/ scripts/

	- name: Test with pytest
	run: \|
	# Runs the whole non-billable suite, not just tests/unit/ (issue
	# #113). tests/real_world/conftest.py auto-applies the real_world
	# marker to everything under that directory, so -m "not real_world"
	# alone would already exclude it -- the --ignore is kept anyway as a
	# second, independent guard. tests/integration/ provisions real
	# billable AWS resources (#109) and must never run in ordinary CI.
	pytest tests/ -v --cov=clustrix --cov-report=xml --cov-report=html --cov-report=term-missing --cov-report=json -m "not real_world" --ignore=tests/real_world --ignore=tests/integration

	- name: Update coverage badge
	if: matrix.os == 'ubuntu-latest' && matrix.python-version == '3.11' && github.ref == 'refs/heads/master'
	run: \|
	python .github/scripts/update_coverage_badge.py

	- name: Commit coverage badge update
	if: matrix.os == 'ubuntu-latest' && matrix.python-version == '3.11' && github.ref == 'refs/heads/master'
	run: \|
	git config --local user.email "action@github.com"
	git config --local user.name "GitHub Action"
	git add README.md
	# Only push when there is something to push. The README carries no
	# coverage badge, so there never is; pushing anyway asked the runner to
	# resolve a branch it may not be on, for no change.
	if git diff --staged --quiet; then
	echo "No coverage badge change to commit."
	else
	git commit -m "Update coverage badge [skip ci]"
	git push
	fi
	env:
	GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

	# Must run BEFORE "Test installation": that step does a non-editable
	# `pip install .`, after which `import clustrix` resolves to
	# site-packages and this checker would be reading a different copy of
	# the code than the docs in this checkout describe. Executes every code
	# block the documentation publishes (#124) -- the script existed and
	# passed, but nothing ran it, so the docs were correct only as long as
	# someone remembered to check by hand. Run once on the primary target;
	# the examples are not version-specific.
	- name: Check documentation examples
	if: matrix.os == 'ubuntu-latest' && matrix.python-version == '3.11'
	run: \|
	python scripts/check_docs_examples.py

	- name: Test installation
	run: \|
	pip install .
	python -c "import clustrix; print('Clustrix installed successfully')"

	- name: Test CLI
	run: \|
	clustrix --help
	clustrix config --help

	integration-test:
	runs-on: ubuntu-latest
	timeout-minutes: 10
	needs: test

	steps:
	- uses: actions/checkout@v4

	- name: Set up Python 3.11
	uses: actions/setup-python@v5
	with:
	python-version: '3.11'

	- name: Install dependencies
	run: \|
	python -m pip install --upgrade pip
	pip install -e ".[dev,test,widget]"

	- name: Run integration tests
	run: \|
	# Run only local integration tests that don't require external resources
	pytest tests/ -k "integration and local and not (gpu or aws or azure or gcp or ssh or slurm or pbs or sge)" -v -x -m "not real_world"

	- name: Test example scripts
	run: \|
	python -c "
	from clustrix import cluster, configure
	configure(cluster_host=None) # Local execution

	@cluster(cores=2)
	def test_example():
	return sum(range(100))

	result = test_example()
	assert result == 4950
	print('Example test passed!')
	"

	docs-test:
	runs-on: ubuntu-latest
	timeout-minutes: 10

	steps:
	- uses: actions/checkout@v4

	- name: Set up Python 3.11
	uses: actions/setup-python@v5
	with:
	python-version: '3.11'

	- name: Install system dependencies
	run: \|
	# pandoc is required: nbsphinx renders 30 notebook pages with it, so
	# dropping it would silently lose them from the built docs.
	#
	# The Acquire::*::Timeout options below are not sufficient on their
	# own -- this job burned its whole 10-minute cap on
	# "Get:5 noble-security InRelease" despite them, twice. So try the
	# install against the runner image's existing package lists first,
	# and only refresh them if that fails, under a hard wall-clock bound
	# that apt cannot ignore.
	#
	# `\|\| ...` here does not hide a failure: if pandoc is still missing
	# afterwards the verification below exits non-zero and the job fails.
	sudo apt-get install -y --no-install-recommends pandoc \|\| {
	echo "pandoc not in the cached lists; refreshing (bounded)"
	sudo timeout 120 apt-get update \
	-o Acquire::Retries=3 \
	-o Acquire::http::Timeout=15 \
	-o Acquire::https::Timeout=15 \|\| \
	echo "apt-get update did not finish in 120s; trying the install anyway"
	sudo apt-get install -y --no-install-recommends pandoc
	}
	# Fail loudly and immediately if pandoc is still absent, rather than
	# letting sphinx produce notebook-less documentation that looks fine.
	pandoc --version \| head -1

	- name: Install dependencies
	run: \|
	python -m pip install --upgrade pip
	pip install -e ".[docs]"

	- name: Test documentation build
	run: \|
	cd docs
	make html

	# Sphinx inline markup does not nest: a ``literal`` inside a bold
	# span renders as plain text, backticks and all, and `sphinx -W` builds
	# it happily. Only the built HTML shows it, and only the built HTML has
	# the .rst, the docstring and the nbsphinx-converted notebook cases in
	# one place -- so this checks the output of the step above.
	- name: Check documentation for nested inline markup
	run: python scripts/check_docs_markup.py docs/build/html

	- name: Test notebook execution
	run: \|
	pip install jupyter nbconvert
	jupyter nbconvert --to script docs/source/notebooks/basic_usage.ipynb
	# Note: Full notebook execution would require cluster setup
	# A single stable name for branch protection to require. Without it,
	# protection has to list all seven `test (os, version)` jobs individually
	# and silently stops covering any combination added later -- and `master`
	# had no protection at all, so a red run could always be merged past.
	#
	# `if: always()` so this job still runs when a dependency fails; without it
	# the gate would be skipped rather than failing, and a skipped required
	# check does not block a merge.
	tests-status:
	name: Tests Status
	runs-on: ubuntu-latest
	needs: [test, integration-test, docs-test]
	if: always()
	steps:
	- name: Check status
	run: \|
	# Every job in `needs` must be checked by name. An aggregator that
	# forgets one reports success while that one burns -- the exact
	# defect fixed in fast_ci.yml's status-check, which had
	# security-scan in `needs` but not in its condition.
	failed=0
	for job in test integration-test docs-test; do
	case "$job" in
	test) result="${{ needs.test.result }}" ;;
	integration-test) result="${{ needs.integration-test.result }}" ;;
	docs-test) result="${{ needs.docs-test.result }}" ;;
	esac
	if [ "$result" != "success" ]; then
	echo "::error::$job: $result"
	failed=1
	else
	echo "$job: success"
	fi
	done
	if [ "$failed" -ne 0 ]; then
	echo "One or more test jobs did not succeed."
	exit 1
	fi
	echo "All test jobs passed."

Navigation Menu

Uh oh!

v0.2.0 (honest beta): merge train complete, gates green, real jobs verified #299

Workflow file

v0.2.0 (honest beta): merge train complete, gates green, real jobs verified #299

Uh oh!

Workflow file for this run