diff --git a/.github/CI-MAINTAINER.md b/.github/CI-MAINTAINER.md new file mode 100644 index 000000000..01be87151 --- /dev/null +++ b/.github/CI-MAINTAINER.md @@ -0,0 +1,188 @@ +# CI Maintainer Notes + +Operational steps that a `gramps-project/addons-source` maintainer +needs to handle once PR 820 (and the branch-neutral follow-up) lands. +The pipeline is otherwise self-driving — this document is only about +the rough edges. + +For day-to-day addon-release maintenance see [MAINTAINERS.md](../MAINTAINERS.md); +for the contributor-facing summary of what a green CI check means on +an unreleased branch see [CONTRIBUTING.md](../CONTRIBUTING.md#work-towards-a-merge). + +## Contents + +1. [One-time setup when the PR first merges](#one-time-setup-when-the-pr-first-merges) +2. [Creating a new maintenance branch](#creating-a-new-maintenance-branch) +3. [When a Gramps minor release lands on PyPI](#when-a-gramps-minor-release-lands-on-pypi) +4. [Image lifecycle](#image-lifecycle) +5. [Diagnostic log markers](#diagnostic-log-markers) +6. [Optional future-proofing knobs](#optional-future-proofing-knobs) + +## One-time setup when the PR first merges + +### 1. Make the `gramps-ci` GHCR package public + +`docker-build.yml` pushes images to +`ghcr.io/gramps-project/addons-source/gramps-ci:` using the +workflow's `GITHUB_TOKEN`. GHCR creates the package as **private** the +first time. Same-repo CI keeps working (token covers own packages), +but **fork PRs cannot pull the image** because their `GITHUB_TOKEN` +has no read access to private packages in `gramps-project`. Fork-PR +container jobs would fail at "Initialize containers" with an +authentication error. + +Fix once, immediately after the first `Build Docker Images` run +finishes: + +1. Go to +2. Under "Danger Zone" → "Change visibility" → set to **Public** + +Every existing and future `gramps-ci:` tag inherits public +visibility from this single setting. + +### 2. Expect the first-push race on `maintenance/gramps60` + +The first push event after merge fires both workflows in parallel: + +- `Build Docker Images` builds and pushes `gramps-ci:gramps60` + (~5 min cold). +- `CI` runs `setup` (~2 s), then its container jobs try to pull the + image. + +Because both start on the same push, the CI container jobs race the +image push and may fail at "Initialize containers" the first time. +This race only happens once per branch: + +1. Wait for `Build Docker Images` to complete. +2. Open the failed CI run → click "Re-run failed jobs". +3. Subsequent pushes find the image already in GHCR — no race. + +This is also explained in the header comment of `ci.yml`. + +## Creating a new maintenance branch + +When a new Gramps minor series goes into development and addons need +a corresponding branch (e.g. `maintenance/gramps62` once 6.2 opens): + +``` +git branch maintenance/gramps62 maintenance/gramps61 +git push origin maintenance/gramps62 +``` + +No *workflow* edits required — the workflows derive everything from +`github.ref_name`. The first push fires the same race described +above — re-run the failed CI jobs once `Build Docker Images` +finishes. + +One file is NOT auto-derived and must be bumped by hand on the new +branch: **`.github/environment.yml`**'s gramps pin (`gramps>=6.0,<6.1`). +The conda Windows lane installs gramps from PyPI through that pin, so +until you bump it to the new series (e.g. `>=6.2,<6.3`) — and the +series is actually published on PyPI — the Windows lane keeps +validating addons against the old series. `ci.yml`'s "Report +gramps-vs-branch series" step prints a loud `::warning::` while the two +diverge; it does not fail. + +The setup job's regex (`gramps[0-9][0-9]`) requires a two-digit +suffix. When Gramps 10.0 opens this regex needs updating in two +places (`ci.yml` setup job and `docker-build.yml` params step). + +## When a Gramps minor release lands on PyPI + +The hybrid Dockerfile auto-detects PyPI availability: + +- `pip install "gramps==X.Y.*"` succeeds → image installs the tagged + PyPI release (`::notice::` log line). +- pip reports "No matching distribution found" → image falls back to a + SHA-pinned `git clone` of `gramps-project/gramps@maintenance/grampsNN` + at the SHA captured by `docker-build.yml`'s params step + (`::warning::` log line). + +When `gramps==6.1.0` is finally published to PyPI, the next image +rebuild on `maintenance/gramps61` silently switches from "git tip" to +"PyPI release" — no maintainer action needed. + +To **immediately** rebuild against the new PyPI release without +waiting for the next push: + +1. Open the `Build Docker Images` workflow in the Actions tab. +2. "Run workflow" → select `maintenance/grampsNN` → Run. + +The `::notice::` line in the build log confirms the switch. + +## Image lifecycle + +`docker-build.yml` owns the `gramps-ci:` image. How it rebuilds: + +- **On push** to `maintenance/gramps**` — builds and pushes the image + for that branch. This is the normal path. +- **On pull_request** touching `.github/docker/**` or + `docker-build.yml` — builds the image **without pushing** (the login + step is skipped and `push:` is false), so a Dockerfile change is + validated before merge. A fork PR never touches the push credential. + A green PR build proves "the image still builds"; it does **not** + update the tag CI pulls. +- **workflow_dispatch** with **`no-cache: true`** — a from-scratch + rebuild that ignores the buildx layer cache. Use this to pick up a + new base-image (apt security) update or a newly published gramps + patch release that the cache key would otherwise skip. +- **Weekly `schedule`** — the `weekly-rebuild` job fans out a + `no-cache` dispatch to every `maintenance/grampsNN` branch. + **Caveat:** GitHub runs scheduled workflows only from the repo's + **default branch**, so this is inert until these workflows exist + there (upstream's default is `maintenance/gramps61`). That is by + design — remove/adjust it if the default branch never carries the + pipeline. + +**Known one-push image lag (not fixed, be aware):** `ci.yml` and +`docker-build.yml` both fire on the same push, and `ci.yml` pulls the +moving `gramps-ci:` tag while the rebuild is still running. So a +push that changes the image (e.g. a Dockerfile edit) has its CI run +against the **previous** image. After the `Build Docker Images` run for +that push finishes, **re-run the affected CI jobs** to test against the +new image. For the same reason, the `ruff` version pinned in the +Dockerfile takes effect one push late. + +## Diagnostic log markers + +When investigating a CI failure, the install-step output in +`Build Docker Images` carries these annotations: + +| Annotation | Meaning | +| --- | --- | +| `::notice::installed gramps==X.Y.* from PyPI` | Released-path install. CI is testing against a tagged release. | +| `::warning::no gramps==X.Y.* on PyPI; installing from gramps-project/gramps@maintenance/grampsNN at ` | Unreleased-branch fallback. CI is testing against the upstream branch tip at ``. Visible to contributors via the CONTRIBUTING.md note. | +| `::error::no gramps==X.Y.* on PyPI and GRAMPS_FALLBACK_SHA is unset` | `git ls-remote` in `docker-build.yml`'s params step returned no SHA for `maintenance/grampsNN` on `gramps-project/gramps`. The matching upstream branch is missing, or the addons-source branch is misnamed. | +| `::error::pip install gramps failed (non-version reason)` | pip failed for a network/registry reason, not because the version is missing. Captured stderr is dumped after the line. The build does **not** fall back to git in this case — by design, so a transient PyPI hiccup cannot silently flip a released branch into "git tip" mode. | + +Other useful entry points: + +- `ci.yml` setup job log shows the derived `branch_suffix` and + `ci_image`. A failure here means the branch name doesn't match + `maintenance/gramps[0-9][0-9]`. +- `docker-build.yml` params step log shows the captured + `fallback_sha`. An empty value means upstream `gramps-project/gramps` + has no matching maintenance branch (warning issued; image build will + fail iff the fallback is needed). + +## Optional future-proofing knobs + +Not needed today; record here in case they ever come up. + +### Upstream gramps repo URL + +The Dockerfile hardcodes `https://github.com/gramps-project/gramps.git` +as the fallback source. If the project ever reorgs or renames, this +URL must change in one place (`.github/docker/gramps-ci/Dockerfile`). +It could be parameterised as a build arg (`ARG GRAMPS_UPSTREAM_REPO`) +with the current URL as default, but a hardcoded value keeps the +Dockerfile simpler and a rename is a sufficiently large event that +editing one string is not the bottleneck. + +### GHCR tag retention + +`docker-build.yml` pushes both a moving `gramps-ci:` tag (e.g. +`gramps60`) and a per-commit `gramps-ci:-` tag. +The moving tag is always overwritten; the SHA tags accumulate over +time. Set a retention policy on the GHCR package settings page if +the count becomes inconvenient — the moving tags are what CI consumes. diff --git a/.github/docker/gramps-ci/Dockerfile b/.github/docker/gramps-ci/Dockerfile new file mode 100644 index 000000000..95284d427 --- /dev/null +++ b/.github/docker/gramps-ci/Dockerfile @@ -0,0 +1,143 @@ +# .github/docker/gramps-ci/Dockerfile +# +# Gramps CI image. The Gramps minor series (6.0, 6.1, …) is picked at +# build time via the GRAMPS_SERIES build arg, so the same Dockerfile +# produces gramps-ci:gramps60, gramps-ci:gramps61, … without per-branch +# edits. Includes everything jobs need: +# - Python + pip-installed Gramps, PyGObject, pycairo +# - GTK typelibs (so addon modules that `from gi.repository import Gtk` +# at module load time are importable — widgets still need xvfb to render) +# - intltool/gettext/git for make.py builds +# - ruff, dbf for lint/test tooling (tests use stdlib unittest per AGENTS.md) +# - xvfb + xauth for tests that actually render (wrap with `xvfb-run`) +# +# No display server runs by default — the image is headless unless a command +# explicitly invokes `xvfb-run`. When running with docker locally, pass +# `--init` (or use a container runtime that injects tini) because xvfb-run +# hangs if it inherits PID 1. +# +# Gramps install path. PyPI is tried first; when no gramps==${SERIES}.* +# release exists (e.g. while 6.1 is still in development) the build +# falls back to a SHA-pinned git clone of +# gramps-project/gramps@maintenance/gramps${SERIES_NODOT}. Other pip +# failures (network, etc.) are NOT silently retried so a transient +# PyPI outage cannot flip a normally-released branch to "test against +# moving tip" mode. docker-build.yml captures the upstream SHA via +# git ls-remote and passes it in as GRAMPS_FALLBACK_SHA; the SHA +# becomes part of the buildx cache key so a moved upstream tip +# actually re-runs the install layer. +# +# Local build (released series): +# docker build --build-arg GRAMPS_SERIES=6.0 .github/docker/gramps-ci +# +# Local build (unreleased series, e.g. 6.1): +# sha=$(git ls-remote https://github.com/gramps-project/gramps.git \ +# refs/heads/maintenance/gramps61 | awk '{print $1}') +# docker build --build-arg GRAMPS_SERIES=6.1 \ +# --build-arg GRAMPS_FALLBACK_SHA=$sha \ +# .github/docker/gramps-ci +# +ARG PYTHON_VERSION=3.12 +FROM python:${PYTHON_VERSION}-slim + +# Gramps minor series to install (e.g. 6.0, 6.1). No default — must be +# passed explicitly so a wrong default cannot silently produce an image +# for the wrong Gramps series. docker-build.yml derives this from the +# branch ref. +ARG GRAMPS_SERIES +# Commit SHA on gramps-project/gramps@maintenance/grampsNN. Only used +# when no gramps==${GRAMPS_SERIES}.* release exists on PyPI. Ignored +# otherwise. Empty default is intentional — on released branches the +# fallback never fires. +ARG GRAMPS_FALLBACK_SHA="" +RUN [ -n "$GRAMPS_SERIES" ] || { echo "GRAMPS_SERIES is required (e.g. 6.0)"; exit 1; } + +LABEL org.opencontainers.image.source="https://github.com/gramps-project/addons-source" +LABEL org.opencontainers.image.description="Gramps ${GRAMPS_SERIES} CI image (Python, Gramps, GTK typelibs, xvfb)" + +RUN apt-get update && apt-get install -y --no-install-recommends \ + libgirepository-2.0-dev \ + gir1.2-glib-2.0 \ + gir1.2-gtk-3.0 \ + gir1.2-pango-1.0 \ + gir1.2-gdkpixbuf-2.0 \ + gir1.2-atk-1.0 \ + gir1.2-gexiv2-0.10 \ + gcc \ + g++ \ + pkg-config \ + python3-dev \ + libcairo2-dev \ + libicu-dev \ + intltool \ + gettext \ + git \ + xvfb \ + xauth \ + && rm -rf /var/lib/apt/lists/* + +# Addon runtime deps (dbf, networkx, lxml, svgwrite, boto3, etc.) are +# NOT baked in here — ci.yml's "Install addon runtime deps (derived from +# requires_mod)" step pip-installs them at CI runtime from every +# .gpr.py's requires_mod list, matching what Gramps' Addon Manager does +# for an end user. Keeps .gpr.py the single source of truth. +# ruff is PINNED: an unpinned ruff would change the lint verdict on rebuild (a +# new rule in the E9/F63/F7/F82 selection flips a lane red — or silently stops +# flagging — with no repo change). Bump deliberately, not by rebuild drift. +# +# PyICU is REQUIRED, not optional. Gramps degrades localization without it +# ("ICU not loaded ... Localization will be impaired"), and worse, gramps 6.0's +# gramps/plugins/webreport/common.py cannot even be IMPORTED without it: when +# neither `icu` nor `PyICU` resolves, its outer `except ImportError: pass` +# leaves `localAlphabeticIndex` unbound and the module-level +# `AlphabeticIndex = localAlphabeticIndex` raises NameError. Every addon that +# imports that module (e.g. TimePedigreeHTML) then fails to load, so CI without +# PyICU reports an addon defect that does not exist for real users. Installing +# it also makes the image match a normal Gramps environment. (Built from source +# against libicu-dev above.) +RUN pip install --no-cache-dir PyGObject pycairo orjson PyICU ruff==0.15.22 + +# Install gramps: PyPI first, SHA-pinned git clone as fallback. +RUN /bin/bash -eo pipefail <<'BASH' +suffix_nodot="${GRAMPS_SERIES//./}" +if pip install --no-cache-dir "gramps==${GRAMPS_SERIES}.*" 2>/tmp/pip.err; then + echo "::notice::installed gramps==${GRAMPS_SERIES}.* from PyPI" +elif grep -q "No matching distribution found for gramps==" /tmp/pip.err; then + if [ -z "${GRAMPS_FALLBACK_SHA}" ]; then + echo "::error::no gramps==${GRAMPS_SERIES}.* on PyPI and GRAMPS_FALLBACK_SHA is unset" + exit 1 + fi + echo "::warning::no gramps==${GRAMPS_SERIES}.* on PyPI; installing from gramps-project/gramps@maintenance/gramps${suffix_nodot} at ${GRAMPS_FALLBACK_SHA}" + mkdir -p /tmp/gramps + cd /tmp/gramps + git init -q + git remote add origin https://github.com/gramps-project/gramps.git + git fetch --depth 1 origin "${GRAMPS_FALLBACK_SHA}" + git checkout FETCH_HEAD + pip install --no-cache-dir . + cd / + rm -rf /tmp/gramps +else + echo "::error::pip install gramps failed (non-version reason); aborting rather than silently switching to git tip" + cat /tmp/pip.err + exit 1 +fi +BASH + +# NOTE: the build toolchain (gcc, python3-dev, pkg-config) is deliberately KEPT +# in the image — it is not purged here. Addon requires_mod are pip-installed at CI +# runtime (ci.yml "Install addon runtime deps"), and the source-built ones with no +# wheel on the apt lane (pygraphviz, psycopg2) compile a C extension during that +# step. Purging gcc/pkg-config left those builds with no compiler, so they failed +# and were swallowed by the install step's "|| echo … (continuing)" — silently +# degrading the affected addon's coverage while the job stayed green. The per- +# package system headers those builds link against (libgraphviz-dev, libpq-dev on +# apt) are derived from the addons' .gpr.py and installed at CI runtime by the same +# single-source map as requires_gi/requires_exe — see .github/scripts/ +# addon_system_deps.py (MOD_BUILD_PACKAGES) and ci.yml's "Install addon system +# deps" step — so .gpr.py stays the single source of truth. + +RUN python -c "from gramps.gen.const import VERSION; print('Gramps', VERSION)" \ + && python -c "import gi; gi.require_version('Gtk', '3.0'); from gi.repository import Gtk; print('GTK OK')" + +WORKDIR /workspace diff --git a/.github/environment.yml b/.github/environment.yml new file mode 100644 index 000000000..0d73f7cbd --- /dev/null +++ b/.github/environment.yml @@ -0,0 +1,23 @@ +name: addons-ci +channels: + - conda-forge +dependencies: + - python=3.12 + - pygobject + - gtk3 + - pip + # Addon runtime deps (dbf, networkx, lxml, svgwrite, boto3, etc.) are + # installed at CI runtime by ci.yml's auto-derive step from .gpr.py + # requires_mod — single source of truth. Keep only the stable base + # here (Gramps + orjson for plugin registration). + # + # NOTE: gramps is installed from PyPI via the `pip:` block below (conda-forge + # is NOT the source), and the pin is a deliberate per-series bound. `<6.1` + # can never resolve 6.1, so on a maintenance/gramps61 (or later) branch this + # file MUST be bumped (e.g. to ">=6.1,<6.2") — the workflows derive the series + # automatically, but this pin does not and will otherwise validate addons + # against 6.0.x forever (see .github/CI-MAINTAINER.md and ci.yml's "Report + # gramps-vs-branch series" step, which warns loudly when the two diverge). + - pip: + - "gramps>=6.0,<6.1" + - orjson diff --git a/.github/scripts/active_addons.py b/.github/scripts/active_addons.py new file mode 100644 index 000000000..757862c0d --- /dev/null +++ b/.github/scripts/active_addons.py @@ -0,0 +1,127 @@ +#!/usr/bin/env python3 +"""Single source of truth for the *active addon* rule (include_in_listing). + +An addon is **active** — built and released by ``make.py``, so CI gates on it — +when at least one ``register()`` call in its ``.gpr.py`` file(s) would be listed. +``make.py`` reads each registration's ``include_in_listing`` with a default of +``True`` (``p.get("include_in_listing", True)``), so a plugin is listed unless +its registration explicitly sets ``include_in_listing=False``. An addon whose +EVERY registration sets it to ``False`` is inactive; CI skips it in lint, the +structure check, compile, and the test runners. + +This module is the parser behind ``active_addons.sh``'s ``is_active()``; the +shell helper calls ``--list`` once and greps the result. The rule here is: + +* **per-register**, matching ``make.py`` — one ``register(..., include_in_listing + =False)`` does not make the addon inactive if a sibling ``register()`` omits + the flag (defaults True) or sets it truthy. (The previous grep helper was + file-granular: any ``include_in_listing=`` with no ``=True`` in the whole file + read as inactive, disagreeing with make.py on that mixed shape.) +* **comment-proof** — a flag mentioned only in a ``#`` comment is ignored, + because the source is parsed with ``ast`` rather than grepped. +* **tolerant** — a ``.gpr.py`` that fails to parse, or a dir whose gpr has no + ``register()`` call at all, counts as ACTIVE. The default never silently drops + an addon from CI; the worst case is gating one that make.py would not build. + +Pure stdlib, no gramps import, never executes the ``.gpr.py`` (ast.parse only). + +CLI:: + + active_addons.py --list [ROOT] # active addon dir names, one per line + active_addons.py --check DIR # exit 0 if DIR is active, 1 if not +""" + +from __future__ import annotations + +import argparse +import ast +import glob +import os +import sys + + +def _register_calls(tree: ast.AST): + """Yield every ``register(...)`` call node in a parsed .gpr.py.""" + for node in ast.walk(tree): + if isinstance(node, ast.Call): + func = node.func + if isinstance(func, ast.Name) and func.id == "register": + yield node + + +def _register_is_listed(call: ast.Call) -> bool: + """Whether one ``register()`` call would be listed by make.py. + + Listed unless it carries ``include_in_listing=`` set to the literal + ``False``. Omitted → default True → listed. A non-literal value (a variable + / expression we cannot evaluate statically) is treated as listed, so the + tolerant default never hides an addon. + """ + for kw in call.keywords: + if kw.arg == "include_in_listing": + value = kw.value + if isinstance(value, ast.Constant) and value.value is False: + return False + return True + return True # omitted → make.py default True + + +def addon_is_active(addon_dir: str) -> bool: + """Whether an addon directory is active (built/released → CI gates on it).""" + gprs = sorted(glob.glob(os.path.join(addon_dir, "*.gpr.py"))) + if not gprs: + return False # not an addon (no descriptor) — nothing to gate + saw_register = False + for gpr in gprs: + try: + with open(gpr, encoding="utf-8") as fh: + tree = ast.parse(fh.read(), filename=gpr) + except (OSError, SyntaxError, ValueError): + return True # cannot analyse → tolerant active + for call in _register_calls(tree): + saw_register = True + if _register_is_listed(call): + return True + # gpr(s) exist but declared no register() at all → tolerant active; + # otherwise every register set include_in_listing=False → inactive. + return not saw_register + + +def active_addons(root: str) -> list[str]: + """Sorted names of the active addon directories directly under *root*.""" + names: list[str] = [] + for gpr in sorted(glob.glob(os.path.join(root, "*", "*.gpr.py"))): + addon_dir = os.path.dirname(gpr) + name = os.path.basename(addon_dir) + if name not in names and addon_is_active(addon_dir): + names.append(name) + return sorted(names) + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser( + description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter + ) + group = parser.add_mutually_exclusive_group(required=True) + group.add_argument( + "--list", + nargs="?", + const=".", + metavar="ROOT", + help="print active addon dir names under ROOT (default: cwd)", + ) + group.add_argument( + "--check", + metavar="DIR", + help="exit 0 if the addon DIR is active, 1 if not", + ) + args = parser.parse_args(argv) + + if args.check is not None: + return 0 if addon_is_active(args.check) else 1 + print("\n".join(active_addons(args.list))) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/.github/scripts/active_addons.sh b/.github/scripts/active_addons.sh new file mode 100644 index 000000000..c1c2cca30 --- /dev/null +++ b/.github/scripts/active_addons.sh @@ -0,0 +1,37 @@ +# Single source of truth for the is_active() addon filter, sourced by every +# ci.yml job step that gates on include_in_listing. +# +# An addon is "active" (built and released by make.py, so CI gates on it) when +# at least one register() in its .gpr.py would be listed — include_in_listing +# omitted (make.py default True) or set to anything but the literal False. Only +# an addon whose EVERY register() sets include_in_listing=False is inactive. +# Those are skipped by lint, the structure check, compile, and the unit / +# integration test runners. Previously each of those ~6 job steps inlined an +# identical copy of this function; this file is the one place it now lives, so a +# change to the active-addon rule is a one-site edit (per PR #820; Gary Griffin). +# +# The rule itself lives in active_addons.py (ast-based, per-register, +# comment-proof — a grep cannot tell one register's flag from another's, nor a +# real flag from one in a comment). This helper computes the active set ONCE at +# source time and is_active() is a membership test, so a sourcing step starts +# one interpreter, not one per addon. python3, falling back to python for the +# conda Windows lane which ships only `python`. A helper failure aborts the +# sourcing step (its shell runs under set -e) rather than silently marking +# every addon inactive. +# +# Source it from a `shell: bash` step (the function uses `local`, a bashism): +# source .github/scripts/active_addons.sh +_AA_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +if command -v python3 >/dev/null 2>&1; then _AA_PY=python3; else _AA_PY=python; fi +_ACTIVE_ADDONS="$("$_AA_PY" "$_AA_DIR/active_addons.py" --list .)" || { + echo "::error::active_addons.py failed — cannot determine active addons" >&2 + return 1 2>/dev/null || exit 1 +} + +is_active() { + # Accept a dir name or path; compare on the basename (strip trailing slash + # then any leading path), matching the names active_addons.py --list prints. + local name="${1%/}" + name="${name##*/}" + printf '%s\n' "$_ACTIVE_ADDONS" | grep -qxF "$name" +} diff --git a/.github/scripts/addon_python_deps.py b/.github/scripts/addon_python_deps.py new file mode 100644 index 000000000..cb9ce4d35 --- /dev/null +++ b/.github/scripts/addon_python_deps.py @@ -0,0 +1,318 @@ +#!/usr/bin/env python3 +"""Single source of truth for addon *Python* dependencies (``requires_mod``). + +Sibling to ``addon_system_deps.py`` (which covers the *system* deps — +``requires_gi`` typelibs and ``requires_exe`` executables). This module covers +the third dependency kind a Gramps addon declares in its ``.gpr.py``: + +* ``requires_mod`` — importable Python module names (e.g. ``["psycopg2"]``, + ``["life_line_chart", "svgwrite"]``). pip-installable. + +ci.yml previously inlined an *identical* ``requires_mod`` derivation heredoc in +three jobs (``unit-test-linux``, ``unit-test-windows``, ``integration-test``) +to build the install union, plus a near-identical ``find_spec`` validator +heredoc in each. That copy-paste is the drift this module removes: the +``.gpr.py`` files stay the single source of truth and every job derives the +list from one place. A one-line change is now a one-site edit. + +Self-contained on purpose: pure stdlib, no Gramps import and no third-party +import, so it runs at image-build time before Gramps is installed and on the +bare Windows conda runner. It does NOT depend on any external project. + +Scanning mirrors ``addon_system_deps.py`` deliberately: a regex finds each +``requires_mod = [...]`` assignment and ``ast.literal_eval`` parses the +bracketed list (no executing the ``.gpr.py``). Every real ``requires_mod`` in +addons-source is a flat list of string literals, so a literal-eval parse covers +them all; a non-literal or unreadable declaration is skipped tolerantly (with a +note to stderr) rather than aborting the batch — mirroring the old inline +behaviour. + +``requires_mod`` is the *importable module* name Gramps verifies at runtime +(``gramps/gen/utils/requirements.py`` ``Requirements.check_mod`` — bare +``find_spec`` on gramps 6.0; ``find_spec`` plus a real import on 6.1+, gramps +PR #2308). ``pip install`` wants the *distribution* name, which differs for a +few packages, so the install union maps the known import→distribution cases +(``PIL`` → ``Pillow``) — single-sourced from Gramps' own ``_IMPORT_TO_PYPI`` +table when the installed gramps ships one (6.1+), with the local +``_IMPORT_TO_DISTRIBUTION`` mirror as fallback. The map is INSTALL-ONLY: +``--check-resolves`` validates the *raw* declared import name, exactly as +Gramps does, so an addon that declares the PyPI distribution name by mistake +(``requires_mod=["Pillow"]`` when the import name is ``"PIL"``) is still +caught. + +CLI:: + + addon_python_deps.py --install-list ROOT # space-separated sorted union + addon_python_deps.py --check-resolves ROOT # fail if a declared import name + # pip-installs but does not import +""" + +from __future__ import annotations + +import argparse +import ast +import glob +import os +import re +import sys +from collections.abc import Callable + +# Sibling module in the same directory: the wheel-only vs source-built +# classification of every declared requires_mod. Running this file by path +# already puts its dir on sys.path[0]; the insert also covers being imported +# (the tests, and any embedding). Pure stdlib either way — no gramps import. +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from addon_system_deps import WHEEL_ONLY_MODS # noqa: E402 + +# Matches a single-line ``requires_mod = ["a", "b"]`` assignment — the same +# shape addon_system_deps.py's _GI_RE/_EXE_RE assume, and the only shape that +# occurs in addons-source. +_MOD_RE = re.compile(r"requires_mod\s*=\s*(\[[^\]]*\])") + +# ``requires_mod`` holds the *importable module* name (declare "PIL", the import +# name, NOT "Pillow", the PyPI distribution — Gramps verifies it with +# ``importlib.util.find_spec``). But ``pip install`` wants the *distribution* +# name, which differs for some packages, so installing the raw import name fails +# (``pip install PIL`` → no such distribution; the package is ``Pillow``). Map +# the known import→distribution cases so the derived install list resolves on +# PyPI. Addons stay correct (import name); only the install side translates. +# +# FALLBACK MIRROR of Gramps' own install-time table, ``_IMPORT_TO_PYPI`` in +# ``gramps/gen/utils/pypi.py`` (gramps PR #2308, merged into gramps 6.1 as +# 7f94428b13; not on 6.0) — the authority Gramps uses to install +# ``requires_mod`` deps in frozen/Flatpak/pip-less environments. At lookup time +# ``_distribution_map()`` prefers that table from the installed gramps, so on +# 6.1+ lanes new upstream entries take effect without touching this file; this +# mirror serves lanes where gramps is absent or predates 6.1 (the gramps60 +# image, the conda-forge 6.0.x Windows lane, bare runners). The sync-guard test +# ``tests/test_addon_python_deps.py::GrampsTableSync`` pins mirror == authority +# wherever the authority is importable. +_IMPORT_TO_DISTRIBUTION = { + "PIL": "Pillow", + "cv2": "opencv-python", + "sklearn": "scikit-learn", + "yaml": "PyYAML", + "dateutil": "python-dateutil", + "bs4": "beautifulsoup4", + "serial": "pyserial", + "usb": "pyusb", + "nacl": "PyNaCl", + "Crypto": "pycryptodome", + "OpenSSL": "pyOpenSSL", + "wx": "wxPython", +} + + +def _distribution_map() -> dict[str, str]: + """The import→distribution table, preferring the authority over the mirror. + + The installed Gramps (6.1+) ships the authoritative table, + ``_IMPORT_TO_PYPI`` in ``gramps/gen/utils/pypi.py``; read it so CI installs + exactly the distribution Gramps' own installer would. Fall back to the + local ``_IMPORT_TO_DISTRIBUTION`` mirror when gramps is absent (image + build, bare runner) or predates 6.1. The table dict is read directly rather + than calling ``resolve_pypi_name()``: same data, but without one + ``LOG.warning`` per translated name in every CI step — and that warning + advises declaring the PyPI name, the opposite of the import-name contract + ``--check-resolves`` enforces. + + A half-installed gramps can raise ``SystemExit`` at import (ResourcePath + aborts when its resources are missing), so the guard is broader than + ``ImportError``. + """ + try: + from gramps.gen.utils import pypi as _gramps_pypi + except (Exception, SystemExit): + return _IMPORT_TO_DISTRIBUTION + table = getattr(_gramps_pypi, "_IMPORT_TO_PYPI", None) + return dict(table) if table else _IMPORT_TO_DISTRIBUTION + + +def _module_checker() -> tuple[str, Callable[[str], bool]]: + """A ``(label, check)`` pair implementing Gramps' requires_mod gate. + + Delegates to the installed gramps' ``Requirements().check_mod`` so the + gate matches whichever series this lane ships: bare ``find_spec`` on 6.0, + ``find_spec`` plus a real import on 6.1+ (gramps PR #2308). The stdlib + ``find_spec`` fallback only applies where gramps is not importable (a dev + box, never a CI lane) and deliberately stays find_spec-only: the gate + exists to catch *declaration* bugs, and really importing every declared + mod on an arbitrary machine is slow and side-effectful. + """ + try: + from gramps.gen.utils.requirements import Requirements + except (Exception, SystemExit): + from importlib.util import find_spec + + return ( + "stdlib find_spec (gramps not importable)", + lambda name: find_spec(name) is not None, + ) + return "gramps Requirements().check_mod", Requirements().check_mod + + +def _gpr_files(root: str) -> list[str]: + return sorted(glob.glob(os.path.join(root, "*", "*.gpr.py"))) + + +def _declared_mods(text: str, path: str) -> list[str]: + """The *raw* (un-mapped) ``requires_mod`` import names declared in one + ``.gpr.py`` body. Tolerant: a non-literal value is skipped with a note to + stderr, mirroring the old inline behaviour.""" + out: list[str] = [] + for m in _MOD_RE.finditer(text): + try: + value = ast.literal_eval(m.group(1)) + except (ValueError, SyntaxError): + print( + f"addon_python_deps: skipping non-literal requires_mod " + f"in {path}: {m.group(1)!r}", + file=sys.stderr, + ) + continue + for mod in value: + if not mod: + continue + if not isinstance(mod, str): + # e.g. requires_mod=[("psycopg2", ">=2")] — an author copying + # the requires_gi tuple shape. A non-str entry would crash the + # later sorted()/mapping; skip it tolerantly with a note. + print( + f"addon_python_deps: skipping non-string requires_mod entry " + f"in {path}: {mod!r}", + file=sys.stderr, + ) + continue + out.append(mod) + return out + + +def declared_mods(root: str) -> set[str]: + """The sorted-by-caller set of *raw* ``requires_mod`` import names declared + across every addon's ``.gpr.py`` under *root* — the names Gramps verifies + via ``find_spec`` (NOT the install-mapped distribution names). Unreadable + files are skipped with a note to stderr.""" + names: set[str] = set() + for path in _gpr_files(root): + try: + with open(path, encoding="utf-8") as fh: + text = fh.read() + except OSError as exc: + print( + f"addon_python_deps: skipping unreadable {path}: {exc}", + file=sys.stderr, + ) + continue + names.update(_declared_mods(text, path)) + return names + + +def install_list(root: str) -> list[str]: + """Return the sorted union of ``requires_mod`` across every addon's + ``.gpr.py`` under *root*, mapped to pip *distribution* names. This is the + list the ci.yml "Install addon runtime deps" steps pip-install. Best-effort + and tolerant — a file that cannot be read, or a non-literal value, is + skipped (not fatal), so the step installs what it can resolve.""" + table = _distribution_map() + return sorted(table.get(mod, mod) for mod in declared_mods(root)) + + +def check_resolves(root: str) -> int: + """Validate that every declared ``requires_mod`` import name which actually + pip-installed also passes Gramps' own dependency gate, + ``Requirements().check_mod`` — bare ``find_spec`` on gramps 6.0, + ``find_spec`` plus a real import on 6.1+ (gramps PR #2308) — so the gate + matches whichever series this lane ships. The *raw* declared name is + checked (NOT the install-mapped distribution name), because that is what + Gramps imports; installed-ness however is probed with the *mapped* + distribution name, because that is the only name pip knows (``pip show + PIL`` fails even with Pillow installed). + + Run this *after* the install union has been pip-installed. A name whose + distribution pip never installed is judged by its category: a **wheel-only** + module (``WHEEL_ONLY_MODS``) ships a pure/binary wheel that installs on + every CI platform, so a miss is a real provisioning regression and FAILS + the gate; a **source-built** module (``MOD_BUILD_PACKAGES``, e.g. pygraphviz + / psycopg2) can legitimately miss on an image/system gap and stays an + advisory skip. A name that pip-installed yet still fails the gate is a wrong + declaration (e.g. the PyPI distribution ``"Pillow"`` instead of the import + name ``"PIL"``) — or, on 6.1+, a module that installs but cannot import — + and fails the run. Returns 1 if any bad or missing-wheel name, else 0.""" + import subprocess + + label, check = _module_checker() + print(f"dep gate: {label}") + table = _distribution_map() + bad: list[str] = [] + missing_wheels: list[str] = [] + for name in sorted(declared_mods(root)): + dist = table.get(name, name) + installed = ( + subprocess.run( + [sys.executable, "-m", "pip", "show", dist], + capture_output=True, + ).returncode + == 0 + ) + if not installed: + if name in WHEEL_ONLY_MODS: + missing_wheels.append(name) + print(f"x {name} (wheel-only, but pip never installed {dist})") + else: + print( + f"~ {name} (pip never installed {dist}, skipping — " + "source-built/system-dep)" + ) + continue + if not check(name): + bad.append(name) + print(f"x {name} (installed as {dist} but fails Gramps' dep gate)") + else: + print(f"ok {name}") + + if bad: + print() + print(f"::error::Wrong requires_mod names: {bad}") + print("These pip-install but are not importable. requires_mod is") + print("consumed by gramps' check_mod() — find_spec, plus a real import") + print("on gramps 6.1+ — so the importable module name is required") + print("(e.g. 'PIL', not 'Pillow').") + if missing_wheels: + print() + print(f"::error::Wheel-only requires_mod never pip-installed: {missing_wheels}") + print("These declare a pure/binary wheel that installs on every CI") + print("platform, so a failed install is a provisioning regression, not") + print("an environment gap — investigate the install step's output above.") + return 1 if (bad or missing_wheels) else 0 + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser( + description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + group = parser.add_mutually_exclusive_group(required=True) + group.add_argument( + "--install-list", + metavar="ROOT", + help="print the sorted, pip-installable union of requires_mod across " + "ROOT/*/*.gpr.py (import names mapped to distribution names)", + ) + group.add_argument( + "--check-resolves", + metavar="ROOT", + help="verify every declared requires_mod import name that pip-installed " + "also passes Gramps' dep gate (Requirements().check_mod when gramps is " + "importable, stdlib find_spec otherwise); exit 1 if any installs but " + "does not import", + ) + args = parser.parse_args(argv) + + if args.install_list is not None: + print(" ".join(install_list(args.install_list))) + return 0 + return check_resolves(args.check_resolves) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/.github/scripts/addon_system_deps.py b/.github/scripts/addon_system_deps.py new file mode 100644 index 000000000..2ff13d02d --- /dev/null +++ b/.github/scripts/addon_system_deps.py @@ -0,0 +1,289 @@ +#!/usr/bin/env python3 +"""Single source of truth for addon *system* dependencies in CI. + +Addons declare three dependency kinds in their ``.gpr.py``: + +* ``requires_mod`` — importable Python modules. pip-installable; ci.yml + auto-derives these from the ``.gpr.py`` files. Most ship a plain wheel and need + nothing more, but a few have no wheel on a CI platform and build from / link + against a system library — ``pygraphviz`` (graphviz), ``psycopg2`` / ``psycopg`` + (libpq). Those need a system package before the pip step can satisfy them, + mapped in ``MOD_BUILD_PACKAGES`` below. +* ``requires_gi`` — GObject-introspection typelibs (e.g. ``GooCanvas``). +* ``requires_exe`` — system executables (e.g. ``dot`` from graphviz). + +All three system kinds above (the GI typelibs, the executables, and the +source-built ``requires_mod`` system packages) are not pip-installable as named, +are named differently per platform, and Gramps' own ``Requirements`` only +*checks* ``requires_mod`` (never installs its system side). This module maps each +declared ``requires_gi`` namespace / ``requires_exe`` name / source-built +``requires_mod`` to its package on each CI platform and scans the addons for what +they declare, so ci.yml derives the install list from one place instead of a +hand-kept list. + +Platform availability is asymmetric and encoded here: the GTK 3 addon libs +(goocanvas, osm-gps-map, gexiv2) exist on Debian/apt but **not on conda-forge**, +so the conda (Windows) lane cannot install them — addons needing them skip there +by necessity. A ``conda`` value of ``None`` records that. + +Pure stdlib so it runs anywhere in CI without bootstrapping. + +CLI:: + + addon_system_deps.py --platform apt # space-separated install list + addon_system_deps.py --platform conda # (only packages available there) + addon_system_deps.py --unmapped . # declared GI/exe/mod with no map entry; exit 1 if any +""" + +# ------------------------ +# Python modules +# ------------------------ +from __future__ import annotations + +import argparse +import ast +import glob +import os +import re +import sys + +# --------------------------------------------------------------------------- +# The map. Keys are what addons declare; values give the package per platform. +# A None value means "no package provides this on that platform" (so it is not +# installed there and an addon needing it is expected to skip). +# --------------------------------------------------------------------------- + +# requires_gi namespace -> package providing the typelib, per platform. +GI_PACKAGES: dict[str, dict[str, str | None]] = { + "GExiv2": {"apt": "gir1.2-gexiv2-0.10", "conda": None}, + "GooCanvas": {"apt": "gir1.2-goocanvas-2.0", "conda": None}, + "OsmGpsMap": {"apt": "gir1.2-osmgpsmap-1.0", "conda": None}, + # PlaceCoordinateGramplet declares GeocodeGlib 1.0, but modern distros ship + # only the 2.0 typelib and conda-forge ships none; the addon has no tests. + # Recorded so the drift-guard recognises the namespace; not installed. + "GeocodeGlib": {"apt": None, "conda": None}, +} + +# requires_exe executable -> package providing it, per platform. +EXE_PACKAGES: dict[str, dict[str, str | None]] = { + "dot": {"apt": "graphviz", "conda": "graphviz"}, +} + +# Source-built / system-library requires_mod -> the package that makes the +# module installable+importable, per platform. A requires_mod belongs here ONLY +# when a plain ``pip install `` cannot satisfy it by itself on a CI platform: +# * apt — no wheel, so pip compiles the C extension from source against a -dev +# header (``pygraphviz`` -> libgraphviz-dev; ``psycopg2`` -> libpq-dev), or the +# pure-Python build links a system shared library at import time (``psycopg`` / +# psycopg3 -> libpq, provided by libpq-dev). The generic compiler toolchain +# (gcc, python3-dev, pkg-config) those source builds need stays in the CI +# image (.github/docker/gramps-ci/Dockerfile no longer purges it). +# * conda — conda-forge ships the whole binding prebuilt, so the value is the +# module's own conda-forge package; the conda lane ``mamba install``s it and +# the later pip step finds it already satisfied. Verified present on +# conda-forge win-64: pygraphviz, psycopg2, psycopg. +# Both platforms must PROVISION the dep or fail the install step honestly — a +# value the platform cannot resolve aborts the job (apt-get install / mamba +# install) rather than letting ci.yml's "|| echo … (continuing)" swallow a failed +# build into a silently-degraded green. A genuinely unprovisionable module would +# be a None, but pygraphviz/psycopg2/psycopg are available on both apt and +# conda-forge. +MOD_BUILD_PACKAGES: dict[str, dict[str, str | None]] = { + "pygraphviz": {"apt": "libgraphviz-dev", "conda": "pygraphviz"}, + "psycopg2": {"apt": "libpq-dev", "conda": "psycopg2"}, + "psycopg": {"apt": "libpq-dev", "conda": "psycopg"}, # psycopg3 +} + +# requires_mod that ship a plain pip wheel needing no system package on any CI +# platform. Listed explicitly so the drift guard can tell a wheel-only module +# (nothing to map) from a source-built one that was forgotten: every declared +# requires_mod must be classified as exactly one of WHEEL_ONLY_MODS or +# MOD_BUILD_PACKAGES, else a newly-added source-built dep could silently lose +# coverage again (the build-toolchain gap this module closes). ``--unmapped`` +# fails CI on any requires_mod that is in neither set. +WHEEL_ONLY_MODS: frozenset[str] = frozenset( + { + "PIL", # EditExifMetadata — Pillow ships binary wheels on every CI + # platform; declared by import name, addon_python_deps maps + # PIL→Pillow on the install side + "boto3", # S3MediaUploader — pure-Python AWS SDK wheel + "dbf", # TMGimporter — pure-Python wheel + "life_line_chart", # LifeLineChartView — pure-Python wheel + "litellm", # ChatWithTree / GrampsChat — pure-Python wheel + "networkx", # NetworkChart — pure-Python wheel + "pymongo", # MongoDB — ships binary wheels on every CI platform + "svgwrite", # LifeLineChartView — pure-Python wheel + } +) + +PLATFORMS = ("apt", "conda") + + +# ------------------------------------------------------------ +# +# scanning +# +# ------------------------------------------------------------ +_GI_RE = re.compile(r"requires_gi\s*=\s*(\[[^\]]*\])") +_EXE_RE = re.compile(r"requires_exe\s*=\s*(\[[^\]]*\])") +_MOD_RE = re.compile(r"requires_mod\s*=\s*(\[[^\]]*\])") + + +def _gpr_files(root: str) -> list[str]: + return sorted(glob.glob(os.path.join(root, "*", "*.gpr.py"))) + + +def _literal(src: str): + try: + return ast.literal_eval(src) + except (ValueError, SyntaxError): + return [] + + +def _scan(root: str, pattern: re.Pattern, first_of_tuple: bool) -> set[str]: + found: set[str] = set() + for path in _gpr_files(root): + try: + text = open(path, encoding="utf-8").read() + except OSError: + continue + for match in pattern.finditer(text): + for entry in _literal(match.group(1)): + if first_of_tuple and isinstance(entry, (tuple, list)): + entry = entry[0] if entry else None + if not entry: + continue + if not isinstance(entry, str): + # A non-str entry (e.g. requires_mod=[("psycopg2", ">=2")], + # or a nested list) would be unhashable for .add() or crash a + # later sorted()/lookup — skip it tolerantly with a note. + print( + f"addon_system_deps: skipping non-string requires_* " + f"entry in {path}: {entry!r}", + file=sys.stderr, + ) + continue + found.add(entry) + return found + + +def scan_gi_namespaces(root: str) -> set[str]: + return _scan(root, _GI_RE, first_of_tuple=True) + + +def scan_executables(root: str) -> set[str]: + return _scan(root, _EXE_RE, first_of_tuple=False) + + +def scan_modules(root: str) -> set[str]: + return _scan(root, _MOD_RE, first_of_tuple=False) + + +def addon_requirements(addon_dir: str) -> tuple[set[str], set[str]]: + """Return (gi_namespaces, executables) declared by a single addon dir.""" + gi: set[str] = set() + exe: set[str] = set() + for path in sorted(glob.glob(os.path.join(addon_dir, "*.gpr.py"))): + try: + text = open(path, encoding="utf-8").read() + except OSError: + continue + for match in _GI_RE.finditer(text): + for entry in _literal(match.group(1)): + ns = entry[0] if isinstance(entry, (tuple, list)) else entry + if isinstance(ns, str) and ns: + gi.add(ns) + for match in _EXE_RE.finditer(text): + for entry in _literal(match.group(1)): + if isinstance(entry, str) and entry: + exe.add(entry) + return gi, exe + + +# ------------------------------------------------------------ +# +# derivation +# +# ------------------------------------------------------------ +def packages(platform: str) -> list[str]: + """All install-by-name packages available for a platform (full mapped set).""" + pkgs: list[str] = [] + for table in (GI_PACKAGES, EXE_PACKAGES, MOD_BUILD_PACKAGES): + for entry in table.values(): + pkg = entry.get(platform) + if pkg: + pkgs.append(pkg) + return sorted(set(pkgs)) + + +def unmapped(root: str) -> tuple[set[str], set[str], set[str]]: + """Declared deps with no entry in the maps at all (drift). + + The third element is every declared ``requires_mod`` classified as *neither* + a wheel-only module nor a source-built one — an unclassified module that, if + it turns out to need a system package, would silently lose coverage. CI fails + on it so a human must classify it (add to ``WHEEL_ONLY_MODS`` or + ``MOD_BUILD_PACKAGES``). + """ + return ( + scan_gi_namespaces(root) - set(GI_PACKAGES), + scan_executables(root) - set(EXE_PACKAGES), + scan_modules(root) - WHEEL_ONLY_MODS - set(MOD_BUILD_PACKAGES), + ) + + +def addon_satisfiable_on(addon_dir: str, platform: str) -> bool: + """ + True if every system dep the addon declares has a package on this platform. + + Used by the test runner to tell an *expected* platform skip (a declared dep + that simply is not packaged here, e.g. goocanvas on conda) from a suspicious + all-skip that should fail. + """ + gi, exe = addon_requirements(addon_dir) + for ns in gi: + entry = GI_PACKAGES.get(ns) + if entry is None or entry.get(platform) is None: + return False + for name in exe: + entry = EXE_PACKAGES.get(name) + if entry is None or entry.get(platform) is None: + return False + return True + + +# ------------------------------------------------------------ +# +# CLI +# +# ------------------------------------------------------------ +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--platform", choices=PLATFORMS) + parser.add_argument( + "--unmapped", + metavar="ROOT", + help="print declared GI/exe/mod deps with no map entry; exit 1 if any", + ) + args = parser.parse_args(argv) + + if args.unmapped is not None: + gi, exe, mod = unmapped(args.unmapped) + for ns in sorted(gi): + print(f"gi:{ns}") + for name in sorted(exe): + print(f"exe:{name}") + for name in sorted(mod): + print(f"mod:{name}") + return 1 if (gi or exe or mod) else 0 + + if args.platform: + print(" ".join(packages(args.platform))) + return 0 + + parser.error("nothing to do: pass --platform or --unmapped") + return 2 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/.github/scripts/gi_bootstrap/sitecustomize.py b/.github/scripts/gi_bootstrap/sitecustomize.py new file mode 100644 index 000000000..9f5cb090d --- /dev/null +++ b/.github/scripts/gi_bootstrap/sitecustomize.py @@ -0,0 +1,29 @@ +"""Pin the GObject-introspection versions, like the Gramps GUI launcher. + +Put this directory on ``PYTHONPATH`` for a test step and the interpreter imports +this ``sitecustomize`` at startup — before any test (or subprocess it spawns) +imports a ``gramps.gui`` module. + +Why: gramps pins its GI versions in the GUI launcher (``gramps/gui/grampsgui.py`` +calls ``gi.require_version`` for Pango/PangoCairo/Gtk at import). A test that +imports a ``gramps.gui.*`` module directly never runs that launcher, so Gtk/Pango +get imported with no version pinned first — emitting a ``PyGIWarning`` and, on a +host where GTK 4 is the default, risking the wrong stack. This shim performs the +same bootstrap, so tests run under the supported GTK 3 stack. + +Used for the discover-based / subprocess-loading steps (e.g. plugin +registration), where the bootstrap must be inherited via ``PYTHONPATH`` by every +spawned interpreter. The in-process unit/integration runner +(``run_addon_tests.py``) does the same ``require_version`` itself. +""" + +try: + import gi + + for _ns, _ver in (("Pango", "1.0"), ("PangoCairo", "1.0"), ("Gtk", "3.0")): + try: + gi.require_version(_ns, _ver) + except (ValueError, AttributeError): + pass +except ImportError: + pass diff --git a/.github/scripts/run_addon_tests.py b/.github/scripts/run_addon_tests.py new file mode 100644 index 000000000..f096ae5ae --- /dev/null +++ b/.github/scripts/run_addon_tests.py @@ -0,0 +1,420 @@ +#!/usr/bin/env python3 +"""Run per-addon unit tests with a GI bootstrap, a timeout, and honest skips. + +Replaces a bare ``python -m unittest `` in CI. It does three things +plain unittest does not: + +1. **GI version bootstrap.** Before any test imports a ``gramps.gui`` module, it + calls ``gi.require_version`` for Pango/PangoCairo/Gtk — the set the Gramps + GUI launcher (``gramps/gui/grampsgui.py``) pins at startup. A direct test + import never runs that launcher, so without this the first + ``from gi.repository import Gtk`` (in gramps core) warns and risks the wrong + GTK on a host where GTK 4 is the default. + +2. **A per-module timeout.** Each module runs in its own subprocess with a wall + clock. A test that hangs (e.g. a DB import that blocks on a platform) would + otherwise hang the whole CI job indefinitely — neither plain unittest nor + xmlrunner has a timeout. A module that exceeds the limit is killed and + reported as a FAILURE, so the job stays bounded and names the culprit. The + worker runs in its own process group (POSIX) so the timeout reaps any + children it spawned, not just the worker — otherwise a hung grandchild + holding the stdout pipe defeats the timeout. + +3. **Honest skip accounting.** unittest exits 0 when every test SKIPS, and also + when a module collects ZERO tests — both read as a pass. This runner FAILS a + wholly-skipped or zero-test module, UNLESS the addon's declared system deps + are unavailable on this platform (e.g. goocanvas/osm-gps-map are not on + conda-forge), in which case the skip is expected and tolerated (the map + lives in ``addon_system_deps.py``). A module that fails to LOAD is excused as + a platform skip only when the failure is dependency-shaped (ImportError, or + gi's absent-typelib ValueError); a SyntaxError or a bug in the addon's own + import-time code is a real defect and always FAILS, on every platform. A + module that raises ``SkipTest`` at import is opting out explicitly and is + always honoured as a skip. + +Usage:: + + run_addon_tests.py --platform apt Addon.tests.test_x Other.tests.test_y + run_addon_tests.py --platform conda Addon.tests.test_x + +Exit code is non-zero if any module is a hard failure (test failure/error, +timeout, or an unexpected all-skip on a platform where the addon's deps are +available). +""" + +# ------------------------ +# Python modules +# ------------------------ +from __future__ import annotations + +import argparse +import importlib +import os +import re +import signal +import subprocess +import sys +import unittest + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +import addon_system_deps as deps # noqa: E402 + + +def _module_timeout() -> int: + """Per-module wall clock (seconds). Generous enough for a legitimate + DB-backed suite, small enough that a hung test is caught promptly instead of + running to the job cap. Overridable via ``RUN_ADDON_TESTS_TIMEOUT``; a + non-integer value is ignored with a note rather than crashing the runner.""" + raw = os.environ.get("RUN_ADDON_TESTS_TIMEOUT", "") + if raw: + try: + return int(raw) + except ValueError: + print( + f"run_addon_tests: ignoring non-integer " + f"RUN_ADDON_TESTS_TIMEOUT={raw!r}; using default 300", + file=sys.stderr, + ) + return 300 + + +MODULE_TIMEOUT_S = _module_timeout() + +# Markers the worker prints so the parent can read the outcome without needing +# xmlrunner (820's env has only stdlib unittest). +_OK = "__RESULT__ ok" +_LOADERROR = "__RESULT__ loaderror" + + +def _terminal_exc_name(text: str) -> str | None: + """The exception class name on the last ``Type: message`` line of a + traceback string (``ModuleNotFoundError``, ``SyntaxError``, ``SkipTest``…), + or None. Scans from the end, so the first match is the terminal line.""" + for line in reversed(text.strip().splitlines()): + m = re.match(r"^([A-Za-z_][\w.]*)\s*:", line.strip()) + if m: + return m.group(1).rsplit(".", 1)[-1] + return None + + +def _is_skip_request(exc: BaseException) -> bool: + """Whether a module opted out at import time via ``raise SkipTest(...)``. + + A module-level SkipTest is an explicit, deliberate opt-out (the addon's own + guard for "this needs a display / PyGObject / a backend I don't have"), so it + is honoured as a skip on every platform — never a failure, and never + dependent on the addon's declared system deps.""" + if isinstance(exc, unittest.SkipTest): + return True + # unittest's loader may wrap it (see _dep_shaped) — check the terminal type. + return isinstance(exc, ImportError) and _terminal_exc_name(str(exc)) == "SkipTest" + + +def _dep_shaped(exc: BaseException) -> bool: + """Whether a module load failure is a missing-dependency shape. + + Only these may be excused as an expected platform skip when the addon's + declared deps are unavailable: a missing module (ImportError / + ModuleNotFoundError), or the ``ValueError`` ``gi.require_version`` raises for + an absent typelib (``Namespace X not available``). A SyntaxError or a bug in + the addon's own import-time code is a real defect and must never be excused. + + Complication: ``unittest``'s loader wraps EVERY import-time exception — + including SyntaxError — into an ``ImportError`` whose message embeds the + original traceback (``Failed to import test module: …``). So for that + wrapper we cannot go by type; inspect the terminal exception in the embedded + traceback instead. + """ + if isinstance(exc, ValueError): + return "not available" in str(exc) + if isinstance(exc, ImportError): + msg = str(exc) + if "Failed to import test module" in msg: + terminal = _terminal_exc_name(msg) + # Unknown terminal → treat as dep (tolerant: a missing dep is the + # common case and we would rather skip than falsely fail). + return terminal in (None, "ImportError", "ModuleNotFoundError") + return True # a bare ImportError (not the loader wrapper) + return False + + +def _load_kind(exc: BaseException) -> str: + """Classify a module load failure: skip | dep | other.""" + if _is_skip_request(exc): + return "skip" + return "dep" if _dep_shaped(exc) else "other" + + +def _load_failure_exception(suite: unittest.TestSuite): + """The exception behind a deferred import failure, or None. + + ``loadTestsFromName`` wraps a module that fails to import in a + ``unittest.loader._FailedTest`` placeholder carrying the original exception + in ``_exception``. Walk the suite for one and return that exception, so the + caller can classify the failure instead of running a placeholder that errors + anonymously. Private-API tolerant: if the internals ever change, return None + and let the suite run (the pre-taxonomy behaviour).""" + try: + from unittest.loader import _FailedTest + except Exception: + return None + + def _walk(test): + if isinstance(test, unittest.TestSuite): + for child in test: + found = _walk(child) + if found is not None: + return found + return None + if isinstance(test, _FailedTest): + return getattr(test, "_exception", None) or ImportError( + "module failed to load" + ) + return None + + return _walk(suite) + + +def _bootstrap_gi() -> None: + """Pin the GI versions the Gramps GUI launcher pins, before tests import.""" + try: + import gi + except ImportError: + return + for namespace, version in (("Pango", "1.0"), ("PangoCairo", "1.0"), ("Gtk", "3.0")): + try: + gi.require_version(namespace, version) + except (ValueError, AttributeError): + pass + + +# ------------------------------------------------------------ +# +# worker: runs ONE module in this (sub)process +# +# ------------------------------------------------------------ +def _run_worker(modname: str, root: str = ".") -> int: + """Run a single module; print a machine-readable result line; exit 0. + + The parent classifies pass/fail from the printed counts and its own platform + knowledge, so the worker always exits 0 (a non-zero exit would be + indistinguishable from an interpreter crash). + """ + _bootstrap_gi() + # Put the addon's own directory on sys.path, mirroring how Gramps' plugin + # loader (gramps/gen/plug/_manager.py) inserts the addon dir before importing + # a plugin. APPEND (not prepend) so the repo-root shared `tests` environment + # still takes precedence: this lets a nested-package addon's top-level + # imports (e.g. ``from name_processor… import``) resolve without shadowing + # the shared Gramps-emulation test env. The module is still loaded by its + # full dotted name from the repo root, so package-relative imports keep + # working too. + addon = modname.split(".", 1)[0] + addon_dir = os.path.join(root, addon) + # Resolve the addon PACKAGE (its directory, from the repo root) before the + # addon dir joins sys.path. Many addons ship a top-level module named after + # the addon itself (/.py); the moment / is on sys.path + # that regular module wins the bare name over the namespace package, + # and the dotted test name .tests.test_x then dies with "module + # '' has no attribute 'tests'". Importing the package first pins it in + # sys.modules so the dotted load resolves, while the addon dir added below + # still serves the tests' bare sibling imports (e.g. `from models import …`). + try: + importlib.import_module(addon) + except Exception: + # Not importable as a package (single-file addon, or its __init__ needs + # deps): leave it; the dotted load below reports the real failure. + pass + if addon_dir not in sys.path: + sys.path.append(addon_dir) + # Load via unittest (NOT a bare import_module probe): an addon whose + # top-level module shares the addon's directory name (e.g. + # CalculateEstimatedDates/CalculateEstimatedDates.py) is shadowed by + # addon_dir being on sys.path, which breaks a dotted import_module but not + # unittest's own resolution. A load failure surfaces two ways depending on + # the Python version and error kind — loadTestsFromName may RAISE it, or + # defer it into a _FailedTest placeholder that errors only when run (so the + # parent would otherwise see an anonymous `broke=1` with the shape lost). + # Handle both, and classify the real exception (dependency-shaped vs a code + # bug) either way. + try: + suite = unittest.defaultTestLoader.loadTestsFromName(modname) + except Exception as exc: # raised import-time failure + kind = _load_kind(exc) + print(f"{_LOADERROR} kind={kind} {exc!r}", flush=True) + return 0 + load_exc = _load_failure_exception(suite) # deferred (_FailedTest) failure + if load_exc is not None: + kind = _load_kind(load_exc) + print(f"{_LOADERROR} kind={kind} {load_exc!r}", flush=True) + return 0 + result = unittest.TextTestRunner(verbosity=2).run(suite) + broke = len(result.failures) + len(result.errors) + print( + f"{_OK} tests={result.testsRun} skipped={len(result.skipped)} broke={broke}", + flush=True, + ) + return 0 + + +# ------------------------------------------------------------ +# +# parent: spawns a timed worker per module and classifies the outcome +# +# ------------------------------------------------------------ +def _classify(modname: str, platform: str, root: str) -> tuple[bool, str]: + """Run one module in a timed subprocess. Return (is_hard_failure, summary).""" + addon = modname.split(".", 1)[0] + satisfiable = deps.addon_satisfiable_on(os.path.join(root, addon), platform) + + proc = subprocess.Popen( + [ + sys.executable, + os.path.abspath(__file__), + "--worker", + modname, + "--root", + root, + ], + stdout=subprocess.PIPE, + stderr=None, # stream the test output straight to the CI log + text=True, + # Own process group so a timeout can reap the worker AND any children it + # spawned. Without this, proc.kill() kills only the worker and the + # follow-up communicate() blocks until a grandchild that inherited the + # stdout pipe exits — a hung grandchild defeats the timeout entirely. + start_new_session=(os.name == "posix"), # setsid; raises on Windows + ) + try: + # communicate() enforces the wall clock and reaps the process. + stdout, _ = proc.communicate(timeout=MODULE_TIMEOUT_S) + except subprocess.TimeoutExpired: + if os.name == "posix": + try: + os.killpg(proc.pid, signal.SIGKILL) # group id == worker pid + except (ProcessLookupError, PermissionError): + proc.kill() + else: + proc.kill() + try: + # Bounded: a surviving grandchild holding the stdout pipe must not + # hang the run a second time. + proc.communicate(timeout=10) + except subprocess.TimeoutExpired: + proc.kill() + return True, f" FAIL {modname} — timed out after {MODULE_TIMEOUT_S}s (hung)" + + out_lines = stdout.splitlines() + for line in out_lines: # echo the worker's result marker into the log + print(line) + + result_line = next( + (ln for ln in reversed(out_lines) if ln.startswith("__RESULT__")), "" + ) + + if result_line.startswith(_LOADERROR): + kind_m = re.search(r"\bkind=(\w+)", result_line) + kind = kind_m.group(1) if kind_m else "" + if kind == "skip": + # The module raised SkipTest at import: an explicit opt-out, honoured + # on every platform regardless of declared-dep satisfiability. + return False, f" skip {modname} — module opted out (SkipTest at import)" + dep_shaped = kind == "dep" + if not dep_shaped: + # A non-dependency load failure (SyntaxError, a bug in the addon's + # import-time code) is a real defect on every platform — never + # excusable as a "deps unavailable here" skip. + return True, ( + f" FAIL {modname} — load error (not dependency-shaped; " + "a code bug, not a platform skip)" + ) + if satisfiable: + return True, f" FAIL {modname} — load error" + return False, ( + f" skip {modname} — not loadable on {platform} " + f"(addon system deps unavailable here)" + ) + + if not result_line.startswith(_OK): + return True, f" FAIL {modname} — no result (worker crashed)" + + fields = dict(tok.split("=", 1) for tok in result_line.split()[2:] if "=" in tok) + ran = int(fields.get("tests", 0)) + skipped = int(fields.get("skipped", 0)) + broke = int(fields.get("broke", 0)) + + if broke: + return True, f" FAIL {modname} — {broke} failed/errored" + if ran == 0 and skipped == 0: + # The module loaded but collected NOTHING (a class not subclassing + # TestCase, a mistyped method name, a refactor that broke collection). + # unittest exits 0 on this, so it would read as green — fail it. Note + # the skipped==0 guard: a class-level skip (setUpClass raising SkipTest) + # reports tests=0 with skips recorded, and that is a skipped module, not + # an empty one — it falls through to the all-skipped rule below. + return True, ( + f" FAIL {modname} — module loaded but collected zero tests " + "(empty or misnamed test module reads as green)" + ) + if skipped >= ran: + # Everything skipped — including the ran==0/skipped>0 class-level case. + total = max(ran, skipped) + if satisfiable: + return True, ( + f" FAIL {modname} — all {total} test(s) skipped " + f"(degraded coverage; deps ARE available on {platform})" + ) + return False, ( + f" skip {modname} — all {total} skipped, expected " + f"(addon system deps unavailable on {platform})" + ) + if skipped: + return False, f" ok {modname} — {ran} tests, {skipped} skipped" + return False, f" ok {modname} — {ran} tests" + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--platform", choices=deps.PLATFORMS) + parser.add_argument( + "--root", + default=".", + help="addons-source root holding the / dirs (default: cwd)", + ) + parser.add_argument( + "--worker", + metavar="MODULE", + help="internal: run this single module and print its result line", + ) + parser.add_argument("modules", nargs="*", help="dotted test modules to run") + args = parser.parse_args(argv) + + if args.worker: + return _run_worker(args.worker, args.root) + + if not args.platform: + parser.error("--platform is required in parent mode") + if not args.modules: + print("No per-addon unit test modules found") + return 0 + + hard_failures: list[str] = [] + summary: list[str] = [] + for modname in args.modules: + failed, line = _classify(modname, args.platform, args.root) + summary.append(line) + if failed: + hard_failures.append(modname) + + print("\n=== addon test summary ===") + for line in summary: + print(line) + + if hard_failures: + print(f"\n{len(hard_failures)} module(s) failed: {hard_failures}") + return 1 + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 000000000..b00efc63b --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,605 @@ +name: CI + +# Branch-neutral workflow: the image tag, make.py argument, and Gramps +# series pin are all derived from the branch ref at runtime, so the same +# file runs unchanged on maintenance/gramps60, maintenance/gramps61, and +# any future maintenance/grampsNN branch. +# +# NOTE for maintainers — first push to a new maintenance branch: the +# corresponding gramps-ci: image does not exist in GHCR yet, so +# the container jobs in this workflow will fail at "Initialize containers" +# on the very first run. The companion docker-build.yml workflow fires +# on the same push and builds/pushes the image (~5 min cold). After it +# finishes, re-run the failed CI jobs (Actions tab → this run → "Re-run +# failed jobs") and they will pull the now-existing image. This race +# happens only on initial branch creation — every subsequent push to +# that branch finds the image already in GHCR. +# +# Full operational runbook (GHCR visibility, PyPI-release transitions, +# diagnostic log markers, etc.): .github/CI-MAINTAINER.md + +on: + push: + branches: [maintenance/gramps**] + pull_request: + branches: [maintenance/gramps**] + +# Least privilege. Every job only reads the repo; the CI image is a public +# GHCR package, so pulling it needs no `packages` scope. This workflow runs +# addon-influenced code (pip package hooks derived from requires_mod, and +# addon import-time code in the load/test lanes), so the ambient token it +# hands that code must not carry write scopes. Presumes the gramps-ci package +# is public (the one-time visibility flip in .github/CI-MAINTAINER.md). +permissions: + contents: read + +jobs: + # ----------------------------------------------------------------- + # Setup — derive the branch suffix (gramps60 / gramps61 / …) from + # the ref. On push events github.ref_name is the branch being + # pushed; on pull_request events github.base_ref is the target + # branch. Either way the suffix is what follows "maintenance/". + # ----------------------------------------------------------------- + setup: + name: Setup + runs-on: ubuntu-latest + outputs: + branch_suffix: ${{ steps.compute.outputs.branch_suffix }} + ci_image: ${{ steps.compute.outputs.ci_image }} + steps: + - id: compute + shell: bash + run: | + ref="${{ github.base_ref || github.ref_name }}" + suffix="${ref#maintenance/}" + case "$suffix" in + gramps[0-9][0-9]) ;; + *) echo "::error::unexpected ref '$ref' (suffix '$suffix')"; exit 1 ;; + esac + echo "branch_suffix=$suffix" >> "$GITHUB_OUTPUT" + # Lowercase the owner/name: GHCR refs must be lowercase, and + # docker-build's metadata-action lowercases what it PUSHES, so the + # pull side must match — otherwise a fork whose owner has uppercase + # letters pulls an invalid (mixed-case) reference and every container + # job fails at init. ${VAR,,} is a bash lowercasing expansion; there + # is no expression-level equivalent in Actions. + repo_lc="${GITHUB_REPOSITORY,,}" + echo "ci_image=ghcr.io/${repo_lc}/gramps-ci:$suffix" >> "$GITHUB_OUTPUT" + + # ----------------------------------------------------------------- + # Lint (ci container) + # ----------------------------------------------------------------- + lint: + name: Lint + needs: setup + runs-on: ubuntu-latest + container: + image: ${{ needs.setup.outputs.ci_image }} + steps: + - uses: actions/checkout@v4 + + - name: Run ruff (syntax and import errors only) + # Skip addon directories whose every register() in .gpr.py sets + # include_in_listing=False — those addons are not built or released + # by make.py, so CI does not gate on their lint state (per Gary + # Griffin's request on PR #820). To re-enable lint gating for an + # addon, set include_in_listing=True on at least one register() + # call in its descriptor (or remove the field — Gramps' default + # is True). Repeated inline rather than centralised so each job + # step stays self-contained. + shell: bash + run: | + source .github/scripts/active_addons.sh + excludes="" + for d in */; do + d="${d%/}" + ls "$d"/*.gpr.py >/dev/null 2>&1 || continue + is_active "$d" || excludes="$excludes --exclude=$d" + done + ruff check --select=E9,F63,F7,F82 --no-fix --exclude='*.gpr.py' $excludes . + + - name: Check trailing whitespace in Python files + run: | + # Use PCRE (-P): in BRE/ERE the bracket expression [ \t] is the + # set { space, backslash, 't' } — git grep matches anything ending + # in 't', not just whitespace. -P makes \t a tab. + if git --no-pager grep --color -n --full-name -P '[ \t]+$' -- '*.py'; then + echo "::error::Trailing whitespace found in Python files" + exit 1 + fi + + # ----------------------------------------------------------------- + # Addon structure (bare runner — just bash, no deps needed) + # ----------------------------------------------------------------- + addon-structure: + name: Addon Structure + runs-on: ubuntu-latest + # Non-blocking until the four addons missing po/template.pot are fixed + # in a follow-up PR. Flip this off in that PR. + continue-on-error: true + steps: + - uses: actions/checkout@v4 + + - name: Check all listed addons have po/template.pot + # Skip include_in_listing=False addons (see lint job for rationale). + shell: bash + run: | + source .github/scripts/active_addons.sh + failed=0 + for gpr in */*.gpr.py; do + addon_dir="$(dirname "$gpr")" + is_active "$addon_dir" || continue + if [ ! -d "$addon_dir/po" ]; then + echo "::error::$addon_dir is missing po/ directory" + failed=1 + elif [ ! -f "$addon_dir/po/template.pot" ]; then + echo "::error::$addon_dir is missing po/template.pot" + failed=1 + fi + done + if [ "$failed" -eq 0 ]; then + echo "All listed addons have po/template.pot" + fi + exit $failed + + # ----------------------------------------------------------------- + # Compile check (ci container) + # ----------------------------------------------------------------- + compile-check: + name: Compile Check + needs: setup + runs-on: ubuntu-latest + container: + image: ${{ needs.setup.outputs.ci_image }} + steps: + - uses: actions/checkout@v4 + + - name: Compile all Python files in listed addons (excluding .gpr.py) + # Skip include_in_listing=False addons (see lint job for rationale). + shell: bash + run: | + source .github/scripts/active_addons.sh + skipped="" + for d in */; do + d="${d%/}" + ls "$d"/*.gpr.py >/dev/null 2>&1 || continue + is_active "$d" || skipped="$skipped $d" + done + failed=0 + while IFS= read -r f; do + skip=0 + for s in $skipped; do + case "$f" in ./$s/*) skip=1; break;; esac + done + [ "$skip" = 1 ] && continue + if ! python3 -m py_compile "$f" 2>&1; then + failed=1 + fi + done < <(find . -name '*.py' ! -name '*.gpr.py' ! -path './.git/*' ! -path '*/__pycache__/*') + exit $failed + + # ----------------------------------------------------------------- + # Unit tests — Linux (ci container) + # ----------------------------------------------------------------- + unit-test-linux: + name: Unit Tests (Linux) + needs: setup + runs-on: ubuntu-latest + container: + image: ${{ needs.setup.outputs.ci_image }} + steps: + - uses: actions/checkout@v4 + + - name: Install addon system deps (requires_gi / requires_exe / source-built requires_mod) + # System deps (GI typelibs, executables, and the -dev headers / libpq that + # source-built requires_mod link against — pygraphviz→libgraphviz-dev, + # psycopg2/psycopg→libpq-dev) are not pip-installable as named and gramps + # only *checks* requires_mod, so the image cannot bake them generically + # (its build context excludes addons-source). Derive the apt set from every + # .gpr.py via the single-source map and install it — the container runs as + # root. MUST run before "Install addon runtime deps" so the headers are + # present when pip compiles the source-built modules; the compiler toolchain + # those builds need lives in the CI image (no longer purged there). + shell: bash + run: | + pkgs=$(python3 .github/scripts/addon_system_deps.py --platform apt) + if [ -n "$pkgs" ]; then + echo "→ addon system deps (apt): $pkgs" + apt-get update + apt-get install -y --no-install-recommends $pkgs + else + echo "no requires_gi / requires_exe / source-built requires_mod declarations found" + fi + + - name: Validate addon system deps are mapped + # Every requires_gi / requires_exe an addon declares must have an entry in + # addon_system_deps.py, and every requires_mod must be classified as either + # wheel-only or source-built (with a system package), so the install list + # never silently drifts from what addons declare — the build-toolchain + # coverage gap returns the moment a new source-built requires_mod is added + # without a mapping. (The system-dep analogue of the requires_mod dep + # gate below.) + shell: bash + run: | + python3 .github/scripts/addon_system_deps.py --unmapped . || { + echo "::error::Addon(s) declare requires_gi/requires_exe/requires_mod with no entry in .github/scripts/addon_system_deps.py — add a mapping row (or, for a wheel-only module, list it in WHEEL_ONLY_MODS)." + exit 1 + } + + - name: Install addon runtime deps (derived from requires_mod) + # Auto-derive the union of requires_mod across every .gpr.py in + # the repo. Mirrors Gramps' Addon Manager install path + # (gramps/gui/plug/_windows.py __on_install_clicked → req.install → + # gen/utils/requirements.py). Keeps .gpr.py files as the single + # source of truth for addon deps — no parallel list to maintain + # in the image or workflow. Import→distribution names (PIL→Pillow) + # come from the installed gramps' own _IMPORT_TO_PYPI table + # (gen/utils/pypi.py, gramps 6.1+, PR #2308), with a local mirror + # in addon_python_deps.py for lanes on 6.0. Best-effort: a package + # needing exotic + # system deps (pygraphviz → graphviz-dev, psycopg2 → libpq-dev) + # may fail here; the affected addon's tests will skip or fail in + # isolation without blocking the rest. + shell: bash + run: | + addon_mods=$(python3 .github/scripts/addon_python_deps.py --install-list .) + if [ -n "$addon_mods" ]; then + echo "→ addon deps: $addon_mods" + for mod in $addon_mods; do + pip install "$mod" || echo "× $mod failed to install (continuing)" + done + else + echo "no requires_mod declarations found" + fi + + - name: Validate requires_mod names against Gramps' dep gate + # Cross-check: every requires_mod entry that pip successfully + # installed in the previous step must also pass Gramps' own dep + # gate — the script delegates to the installed gramps' + # Requirements().check_mod (gramps/gen/utils/requirements.py): + # find_spec on 6.0; find_spec plus a real import on 6.1+ (gramps + # PR #2308), so the gate matches whichever series this lane + # ships. A name that pip-installs but does not import is a + # declaration bug — e.g. requires_mod=["Pillow"] when the + # importable name is "PIL". Pip-installed-ness is probed by the + # mapped distribution name (pip only knows "Pillow", not "PIL"). + # A name whose distribution never installed is judged by category: + # a wheel-only module (WHEEL_ONLY_MODS) ships a wheel that installs + # everywhere, so a miss is a provisioning regression and FAILS the + # gate; a source-built module (pygraphviz/psycopg2) can miss on an + # image/system gap and stays an advisory skip. + shell: bash + run: | + python3 .github/scripts/addon_python_deps.py --check-resolves . + + - name: Run per-addon unit tests + # Filename convention (all OSes): + # test_*.py — general (any OS) + # test_linux_*.py — Linux-only + # test_windows_*.py — Windows-only + # test_integration_*.py — Linux-only, full-pipeline/DB-backed + # The Linux job runs test_*.py except the Windows-only and + # integration buckets. Integration tests run in their own job. + # + # shell: bash — the container's default shell is /bin/sh + # (dash on python:3.12-slim), which does not support the + # ${var//pattern/repl} and ${var%.py} parameter expansions + # used below. + shell: bash + env: + PYTHONPATH: . + run: | + source .github/scripts/active_addons.sh + modules="" + shopt -s globstar # also match nested-package tests//test_*.py + # Glob is scoped to */tests/**/ ON PURPOSE. The only root-level test + # module in the tree, DynamicWeb/test_dynamicweb.py, is a nose-era dev + # harness (nose is gone on py3.12; it asserts a USER_PLUGINS install and + # drives Gramps.py from a source checkout) — unrunnable here, and it + # declares no requires_gi, so it would hard-fail if matched. Its + # CI-shaped sibling DynamicWeb/tests/test_dwr_tree_names.py IS matched, + # so DynamicWeb keeps coverage. The runner can load a root-level module, + # so this is a policy choice, not a limitation. + for f in */tests/**/test_*.py; do + [ -f "$f" ] || continue + addon="${f%%/*}" + is_active "$addon" || continue + case "$(basename "$f")" in + test_integration*) continue ;; + test_windows_*) continue ;; + esac + case "$f" in + # Sqlite/tests/test_sqlite.py predates this pipeline: it needs + # GRAMPS_RESOURCES + example/gramps/example.gramps from a gramps + # SOURCE checkout (the pip/conda wheels ship only gramps/ + share, + # not example/) and writes fixed /tmp paths with no cleanup. Un- + # excluding is a follow-up: provide example.gramps in the lane and + # port the test to tempfile. + Sqlite/tests/test_sqlite.py) continue ;; + esac + mod="${f%.py}" + mod="${mod//\//.}" + modules="$modules $mod" + done + if [ -n "$modules" ]; then + echo "Running unit tests:$modules" + # xvfb-run: some addons create a Gtk style context at import and + # need a display (else a hard Gtk-ERROR abort, not a clean skip). + # run_addon_tests.py: pins the GI versions like gramps' launcher + # (so gramps.gui imports load GTK 3, no PyGIWarning) and fails a + # wholly-skipped module unless the addon's deps are unavailable on + # this platform. + xvfb-run -a --server-args="-screen 0 1920x1080x24" \ + python3 .github/scripts/run_addon_tests.py --platform apt --root . $modules + else + echo "No per-addon unit test modules found" + fi + + # ----------------------------------------------------------------- + # Unit tests — Windows (conda-forge: bundles PyGObject + GTK + Gramps) + # ----------------------------------------------------------------- + unit-test-windows: + name: Unit Tests (Windows) + needs: setup + runs-on: windows-latest + defaults: + run: + shell: bash -el {0} + steps: + - uses: actions/checkout@v4 + + - name: Set up Miniforge + uses: conda-incubator/setup-miniconda@v3 + with: + miniforge-version: latest + activate-environment: addons-ci + environment-file: .github/environment.yml + use-mamba: true + + - name: Verify environment + run: | + mamba info + mamba list | head -30 + python -c "import gramps, gi; print('deps OK')" + + - name: Report gramps-vs-branch series (Windows lane caveat) + # The Linux lane runs the branch's exact gramps in its CI image + # (.github/docker/gramps-ci/Dockerfile, PyPI-first / git-tip fallback). + # The conda Windows lane installs gramps from PyPI via environment.yml's + # `pip:` block (NOT from conda-forge), pinned `gramps>=6.0,<6.1`. So on a + # maintenance/gramps61 (or later) branch it validates addons against + # 6.0.x, not the branch's series — for two independent reasons: the pin's + # `<6.1` excludes 6.1, and 6.1 is not published on PyPI yet anyway. This + # step surfaces the mismatch honestly; it does NOT fail. Addon tests that + # depend on series-exact gramps behaviour skip themselves on Windows + # (e.g. TMGimporter's real-DB import tests) and run on the Linux lane + # instead. The caveat clears only when BOTH hold: gramps 6.1 is on PyPI, + # AND environment.yml's pin has been bumped on that branch (it does not + # self-heal — see .github/CI-MAINTAINER.md). + run: | + suffix="${{ needs.setup.outputs.branch_suffix }}" # e.g. gramps61 + digits="${suffix#gramps}" # e.g. 61 + want="${digits:0:1}.${digits:1}" # e.g. 6.1 + have="$(python -c 'from gramps.version import major_version; print(major_version)')" + if [ "$want" = "$have" ]; then + echo "conda-forge gramps $have matches branch series $want — addons tested against the branch's gramps" + else + echo "::warning::Windows lane: branch targets gramps $want but conda-forge ships $have; addons here are validated against $have. Full $want coverage is on the Linux lane (its CI image git-builds $want). See step comment for why conda-Windows cannot build $want." + fi + + - name: Install addon system deps (requires_gi / requires_exe / source-built requires_mod) + # The conda-forge-available subset of the single-source map. This lane + # provisions its OWN deps and must mirror the apt lane: source-built + # requires_mod that conda-forge ships prebuilt — pygraphviz, psycopg2, + # psycopg — are installed here as full conda-forge packages + # (MOD_BUILD_PACKAGES' conda side), so the later pip step finds them already + # satisfied and the affected addons' suites RUN on Windows instead of + # silently skipping. `mamba install` fails the job if a mapped package + # cannot be resolved, so a provisioning gap aborts honestly — never a silent + # green. The GTK 3 addon GI libs (goocanvas/osm-gps-map/gexiv2) are + # genuinely NOT on conda-forge, so the map keeps them at None and they are + # not installed; addons needing only those skip on Windows by necessity, + # which run_addon_tests tolerates for declared GI deps (--platform conda). + run: | + pkgs=$(python .github/scripts/addon_system_deps.py --platform conda) + if [ -n "$pkgs" ]; then + echo "→ addon system deps (conda): $pkgs" + mamba install -y -c conda-forge $pkgs + else + echo "no conda-available addon system deps to install" + fi + + - name: Validate addon system deps are mapped + # Same drift gate as unit-test-linux (see its comment). This lane runs + # it independently and BEFORE its own pip step: unit-test-windows only + # `needs: setup`, not unit-test-linux, so it must not lean on the Linux + # gate — otherwise a novel requires_mod would reach `pip install` + # unguarded here (arbitrary package execution) while Linux blocks it. + # `python` (conda-forge env) to match the surrounding Windows style. + run: | + python .github/scripts/addon_system_deps.py --unmapped . || { + echo "::error::Addon(s) declare requires_gi/requires_exe/requires_mod with no entry in .github/scripts/addon_system_deps.py — add a mapping row (or, for a wheel-only module, list it in WHEEL_ONLY_MODS)." + exit 1 + } + + - name: Install addon runtime deps (derived from requires_mod) + # See unit-test-linux for rationale. Uses `python` (conda-forge + # env) to match the surrounding Windows job style. + run: | + addon_mods=$(python .github/scripts/addon_python_deps.py --install-list .) + if [ -n "$addon_mods" ]; then + echo "→ addon deps: $addon_mods" + for mod in $addon_mods; do + pip install "$mod" || echo "× $mod failed to install (continuing)" + done + else + echo "no requires_mod declarations found" + fi + + - name: Validate requires_mod names against Gramps' dep gate + # See unit-test-linux for rationale. Uses `python` (conda-forge + # env) to match the surrounding Windows job style. + run: | + python .github/scripts/addon_python_deps.py --check-resolves . + + - name: Run per-addon unit tests + # See filename-convention note in unit-test-linux. The Windows + # job runs test_*.py except test_linux_* and test_integration_*. + env: + PYTHONPATH: . + run: | + source .github/scripts/active_addons.sh + modules="" + # Glob scoped to */tests/**/ (see unit-test-linux for why DynamicWeb's + # root-level test module is deliberately not matched). + shopt -s globstar # also match nested-package tests//test_*.py + for f in */tests/**/test_*.py; do + [ -f "$f" ] || continue + addon="${f%%/*}" + is_active "$addon" || continue + case "$(basename "$f")" in + test_integration*) continue ;; + test_linux_*) continue ;; + esac + case "$f" in + # excluded — see unit-test-linux for the rationale (needs a gramps + # source checkout's example.gramps; a follow-up). + Sqlite/tests/test_sqlite.py) continue ;; + esac + mod="${f%.py}" + mod="${mod//\//.}" + modules="$modules $mod" + done + if [ -n "$modules" ]; then + echo "Running unit tests:$modules" + # No xvfb on Windows (GTK renders natively). run_addon_tests pins + # the GI versions and tolerates addons whose GI deps are not on + # conda-forge (they skip here by platform necessity). + python .github/scripts/run_addon_tests.py --platform conda --root . $modules + else + echo "No per-addon unit test modules found" + fi + + # ----------------------------------------------------------------- + # Integration tests — Gramps (ci container, xvfb available) + # ----------------------------------------------------------------- + integration-test: + name: Integration Tests (Gramps) + runs-on: ubuntu-latest + needs: [setup, unit-test-linux] + container: + image: ${{ needs.setup.outputs.ci_image }} + options: --init + steps: + - uses: actions/checkout@v4 + + - name: Install addon system deps (requires_gi / requires_exe / source-built requires_mod) + # Same as unit-test-linux: derive the apt set from the single-source + # map and install it (container runs as root). The plugin registration + # test subprocess-loads each addon module, so the GI typelibs those + # modules import — and the -dev headers / libpq their source-built + # requires_mod compile/link against — must be present here too. (Mapping is + # drift-guarded in unit-test-linux, which this job needs:, so no duplicate + # gate here.) + shell: bash + run: | + pkgs=$(python3 .github/scripts/addon_system_deps.py --platform apt) + if [ -n "$pkgs" ]; then + echo "→ addon system deps (apt): $pkgs" + apt-get update + apt-get install -y --no-install-recommends $pkgs + fi + + - name: Install addon runtime deps (derived from requires_mod) + # See unit-test-linux for rationale. The plugin registration test + # subprocess-loads each addon's module, which imports its + # requires_mod packages. + shell: bash + run: | + addon_mods=$(python3 .github/scripts/addon_python_deps.py --install-list .) + if [ -n "$addon_mods" ]; then + echo "→ addon deps: $addon_mods" + for mod in $addon_mods; do + pip install "$mod" || echo "× $mod failed to install (continuing)" + done + else + echo "no requires_mod declarations found" + fi + + - name: Validate requires_mod names against Gramps' dep gate + # See unit-test-linux for rationale. + shell: bash + run: | + python3 .github/scripts/addon_python_deps.py --check-resolves . + + - name: Run plugin registration tests + # shell: bash for consistency with the surrounding steps; the + # current command uses no bashisms, but keeps this block safe + # against future edits. Container default is /bin/sh → dash. + # + # gi_bootstrap on PYTHONPATH pins the GI versions (like the gramps GUI + # launcher) for this process AND the addon-module subprocesses this test + # spawns, so gramps.gui imports load GTK 3 without a PyGIWarning. + # + # NOT run under xvfb: this test only *loads* (imports) addon modules in + # subprocesses and tolerates load failures; it does not render. Giving it + # a display made an addon load hang on the (absent) AT-SPI accessibility + # bus until the per-load timeout. Imports that build a Gtk style context + # are exercised under xvfb in the unit/integration test runs instead. + shell: bash + env: + PYTHONPATH: .github/scripts/gi_bootstrap:. + run: | + python3 -m unittest discover -s tests -p "test_*.py" -t . -v + + - name: Run per-addon integration tests + # shell: bash — see unit-test-linux for rationale; the + # ${var//pattern/repl} and ${var%.py} expansions below are + # bash-only. + shell: bash + env: + PYTHONPATH: . + run: | + source .github/scripts/active_addons.sh + modules="" + shopt -s globstar # also match nested-package tests//test_integration*.py + for f in */tests/**/test_integration*.py; do + [ -f "$f" ] || continue + addon="${f%%/*}" + is_active "$addon" || continue + mod="${f%.py}" + mod="${mod//\//.}" + modules="$modules $mod" + done + if [ -n "$modules" ]; then + echo "Running per-addon integration tests:$modules" + xvfb-run -a --server-args="-screen 0 1920x1080x24" \ + python3 .github/scripts/run_addon_tests.py --platform apt --root . $modules + else + echo "No per-addon integration test modules found" + fi + + # ----------------------------------------------------------------- + # Build (ci container) + # ----------------------------------------------------------------- + build: + name: Build + needs: setup + runs-on: ubuntu-latest + container: + image: ${{ needs.setup.outputs.ci_image }} + steps: + - uses: actions/checkout@v4 + + - name: Determine GRAMPSPATH + id: gramps-path + run: | + GPATH=$(python3 -c "import gramps, os; print(os.path.dirname(os.path.dirname(gramps.__file__)))") + echo "path=$GPATH" >> "$GITHUB_OUTPUT" + + - name: Build all addons + env: + GRAMPSPATH: ${{ steps.gramps-path.outputs.path }} + run: | + mkdir -p ../download + python3 make.py "${{ needs.setup.outputs.branch_suffix }}" build all diff --git a/.github/workflows/docker-build.yml b/.github/workflows/docker-build.yml new file mode 100644 index 000000000..4534a9d9f --- /dev/null +++ b/.github/workflows/docker-build.yml @@ -0,0 +1,143 @@ +name: Build Docker Images + +on: + push: + # No paths filter on purpose. Buildx layer cache makes the rebuild + # a ~20-30 s no-op when nothing under .github/docker/ has changed, + # in exchange for guaranteeing gramps-ci: exists on the + # first push to any newly-created maintenance branch. + branches: [maintenance/gramps**] + pull_request: + # Build-only validation (no push — see the build step's `push:` and the + # login step's `if:`) when a PR touches the image definition, so a broken + # Dockerfile is caught before it merges instead of on the next push. + branches: [maintenance/gramps**] + paths: + - ".github/docker/**" + - ".github/workflows/docker-build.yml" + schedule: + # Weekly no-cache refresh so base-image (apt security) updates and new + # gramps patch releases enter the image even when nothing changes the + # buildx cache key. Fans out to every maintenance branch (see the + # weekly-rebuild job). NOTE: scheduled runs execute only from the repo's + # DEFAULT branch — inert until these workflows exist there. + - cron: "23 4 * * 1" + workflow_dispatch: + inputs: + no-cache: + description: "Rebuild without the buildx layer cache (fresh base image + gramps)" + type: boolean + default: false + +env: + REGISTRY: ghcr.io + REPO: ${{ github.repository }} + +permissions: + contents: read + packages: write + +jobs: + build-ci: + name: Build gramps-ci + # The schedule event is handled by weekly-rebuild (fan-out); this job runs + # for push / pull_request / workflow_dispatch. + if: github.event_name != 'schedule' + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Log in to GHCR + # Skip on pull_request: a fork PR's token is read-only and this build + # does not push, so the push credential is never exercised on a PR. + if: github.event_name != 'pull_request' + uses: docker/login-action@v3 + with: + registry: ghcr.io + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v3 + + - name: Compute branch parameters + # Derive the image-tag suffix, Gramps minor series, and upstream + # fallback SHA from the branch ref. Same validation as ci.yml's + # setup job: anything outside maintenance/grampsNN fails fast. + # The fallback SHA is the current tip of gramps-project/gramps + # at the matching maintenance branch; the Dockerfile only uses + # it when no gramps==${series}.* release exists on PyPI. The + # SHA is part of the buildx cache key so a moved upstream tip + # actually re-runs the install layer (otherwise gramps61 CI + # would stay on the same stale gramps revision build after + # build). + id: params + shell: bash + run: | + ref="${{ github.base_ref || github.ref_name }}" + suffix="${ref#maintenance/}" + case "$suffix" in + gramps[0-9][0-9]) ;; + *) echo "::error::unexpected ref '$ref' (suffix '$suffix')"; exit 1 ;; + esac + # gramps60 → 6.0, gramps61 → 6.1, gramps62 → 6.2, … + series="${suffix:6:1}.${suffix:7}" + fallback_sha=$(git ls-remote https://github.com/gramps-project/gramps.git "refs/heads/maintenance/${suffix}" | awk '{print $1}') + if [ -z "$fallback_sha" ]; then + echo "::warning::upstream gramps-project/gramps has no maintenance/${suffix} branch; fallback path will fail if PyPI lacks gramps==${series}.*" + fi + echo "suffix=$suffix" >> "$GITHUB_OUTPUT" + echo "series=$series" >> "$GITHUB_OUTPUT" + echo "fallback_sha=$fallback_sha" >> "$GITHUB_OUTPUT" + + - name: Docker metadata + id: meta + uses: docker/metadata-action@v5 + with: + images: ${{ env.REGISTRY }}/${{ env.REPO }}/gramps-ci + tags: | + type=raw,value=${{ steps.params.outputs.suffix }} + type=sha,prefix=${{ steps.params.outputs.suffix }}- + + - name: Build and push gramps-ci + uses: docker/build-push-action@v6 + with: + context: .github/docker/gramps-ci + # Push everywhere EXCEPT pull_request, where this is a build-only + # validation that must not touch GHCR. + push: ${{ github.event_name != 'pull_request' }} + tags: ${{ steps.meta.outputs.tags }} + labels: ${{ steps.meta.outputs.labels }} + build-args: | + GRAMPS_SERIES=${{ steps.params.outputs.series }} + GRAMPS_FALLBACK_SHA=${{ steps.params.outputs.fallback_sha }} + # `no-cache` (workflow_dispatch input; empty/false on other events) + # forces a from-scratch rebuild — used by the weekly refresh. + no-cache: ${{ inputs.no-cache == true }} + cache-from: type=gha + cache-to: type=gha,mode=max + + weekly-rebuild: + name: Weekly no-cache refresh (fan-out) + # The scheduled event fires only on the default branch; dispatch a no-cache + # rebuild for every maintenance branch that carries this workflow. Inert + # until these workflows reach the default branch — by design, not a bug. + if: github.event_name == 'schedule' + runs-on: ubuntu-latest + permissions: + contents: read + actions: write # gh workflow run + steps: + - name: Dispatch a no-cache rebuild per maintenance branch + env: + GH_TOKEN: ${{ github.token }} + run: | + git ls-remote --heads "https://github.com/${GITHUB_REPOSITORY}.git" \ + 'refs/heads/maintenance/gramps*' \ + | awk -F'refs/heads/' '{print $2}' \ + | grep -E '^maintenance/gramps[0-9][0-9]$' \ + | while read -r branch; do + gh workflow run docker-build.yml --repo "$GITHUB_REPOSITORY" \ + --ref "$branch" -f no-cache=true \ + || echo "::warning::dispatch failed for $branch (workflow not on that branch yet?)" + done diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index dda23171b..96a0c7c77 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1094,6 +1094,18 @@ With the PR created, it's now a matter of working with the ```addons-source``` maintainers. Suggestions and corrections will be made and it may be mecessary to modify the original submission to get the addon accepted. +Your PR will also run automated CI checks (lint, compile, unit and +integration tests, plus an addon build); results appear on the PR page. +**Note:** on ```maintenance/grampsNN``` branches whose corresponding +Gramps release is not yet on PyPI (e.g. ```maintenance/gramps61``` while +6.1 is in development), the CI image is built from a SHA-pinned snapshot +of upstream ```gramps-project/gramps@maintenance/grampsNN``` rather than +a tagged release — a green check on such a PR means the addon works +against that *branch tip*, not against a released version. The exact +SHA used is logged as a ```::warning::``` line in the ```Build Docker +Images``` workflow output. + + The key thing is to monitor progress and comments. Your PR will have an ID number -- 1234, for example -- so you can always go to the github web page for it: diff --git a/tests/test_active_addons.py b/tests/test_active_addons.py new file mode 100644 index 000000000..62a2db705 --- /dev/null +++ b/tests/test_active_addons.py @@ -0,0 +1,201 @@ +# +# Gramps - a GTK+/GNOME based genealogy program +# +# Copyright (C) 2026 Eduard Ralph +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# + +"""The active-addon rule (include_in_listing) must be per-register and correct. + +``active_addons.py`` replaced a file-granular grep with an ast parse. These +tests pin the semantics that make it *more correct* than the grep (per-register, +comment-proof, tolerant defaults) AND assert it is behaviour-identical to the +old grep rule over the real addon tree today — so the change ships proven +equivalent, and the first gpr that exercises the difference trips the oracle +test (a human then confirms the intent and updates the oracle), rather than +silently changing which addons CI gates on. + +Pure stdlib; the bash integration test is skipped where bash is absent. +""" + +from __future__ import annotations + +import glob +import os +import re +import shutil +import subprocess +import sys +import tempfile +import unittest + +_HERE = os.path.dirname(os.path.abspath(__file__)) +_REPO_ROOT = os.path.dirname(_HERE) +_SCRIPTS = os.path.join(_REPO_ROOT, ".github", "scripts") +_HELPER_SH = os.path.join(_SCRIPTS, "active_addons.sh") + +sys.path.insert(0, _SCRIPTS) +import active_addons as aa # noqa: E402 + + +class ActiveAddonSemantics(unittest.TestCase): + """Per-register, comment-proof, tolerant classification.""" + + def setUp(self) -> None: + self.root = tempfile.mkdtemp(prefix="active_addons_") + + def tearDown(self) -> None: + shutil.rmtree(self.root, ignore_errors=True) + + def _addon(self, name: str, gpr_body: str) -> str: + d = os.path.join(self.root, name) + os.makedirs(d, exist_ok=True) + with open( + os.path.join(d, f"{name.lower()}.gpr.py"), "w", encoding="utf-8" + ) as fh: + fh.write(gpr_body) + return d + + def test_sibling_register_without_flag_is_active(self) -> None: + # One register sets False, a sibling omits it (make.py default True) → + # the addon IS built/released → active. (The old grep read this file as + # inactive — the correctness bug this change fixes.) + d = self._addon( + "Mixed", + 'register(TOOL, id="a", include_in_listing=False)\n' + 'register(GRAMPLET, id="b")\n', + ) + self.assertTrue(aa.addon_is_active(d)) + + def test_all_registers_false_is_inactive(self) -> None: + d = self._addon( + "AllFalse", + 'register(TOOL, id="a", include_in_listing=False)\n' + 'register(GRAMPLET, id="b", include_in_listing=False)\n', + ) + self.assertFalse(aa.addon_is_active(d)) + + def test_explicit_true_is_active(self) -> None: + d = self._addon("T", 'register(TOOL, id="a", include_in_listing=True)\n') + self.assertTrue(aa.addon_is_active(d)) + + def test_flag_only_in_comment_is_ignored(self) -> None: + # A False in a comment must not flip an otherwise-listed addon inactive. + d = self._addon( + "Commented", + "# include_in_listing=False (historical note)\n" + 'register(GRAMPLET, id="a")\n', + ) + self.assertTrue(aa.addon_is_active(d)) + + def test_no_register_is_active(self) -> None: + d = self._addon("NoReg", "PLUGINS = [] # descriptor with no register()\n") + self.assertTrue(aa.addon_is_active(d)) + + def test_unparsable_gpr_is_active(self) -> None: + d = self._addon("Broken", "register(TOOL, id= # truncated\n") + self.assertTrue(aa.addon_is_active(d)) + + def test_non_literal_flag_is_active(self) -> None: + # A value we cannot evaluate statically must not be assumed False. + d = self._addon( + "Dynamic", + 'LISTED = True\nregister(GRAMPLET, id="a", include_in_listing=LISTED)\n', + ) + self.assertTrue(aa.addon_is_active(d)) + + def test_dir_without_gpr_not_listed(self) -> None: + d = os.path.join(self.root, "NotAnAddon") + os.makedirs(d) + self.assertFalse(aa.addon_is_active(d)) + + +class BehaviourIdentityWithOldGrep(unittest.TestCase): + """The ast rule must match the old file-granular grep over the real tree.""" + + @staticmethod + def _old_grep_active(addon_dir: str) -> bool: + # The exact rule active_addons.sh used to inline: per FILE, an + # include_in_listing=True (anywhere) or the ABSENCE of any + # include_in_listing= makes the addon active; else inactive. + gprs = sorted(glob.glob(os.path.join(addon_dir, "*.gpr.py"))) + for gpr in gprs: + with open(gpr, encoding="utf-8") as fh: + text = fh.read() + if re.search(r"include_in_listing[ \t]*=[ \t]*True", text): + return True + if not re.search(r"include_in_listing[ \t]*=", text): + return True + return False + + def test_ast_matches_grep_over_whole_tree(self) -> None: + dirs = sorted( + { + os.path.dirname(g) + for g in glob.glob(os.path.join(_REPO_ROOT, "*", "*.gpr.py")) + } + ) + self.assertGreater(len(dirs), 100, "addon tree not found from test location") + diffs = [ + os.path.basename(d) + for d in dirs + if self._old_grep_active(d) != aa.addon_is_active(d) + ] + self.assertEqual( + diffs, + [], + "active_addons.py disagrees with the old grep rule on: " + f"{diffs}. This is the per-register/comment-proof semantic change " + "biting a real addon for the first time — confirm the new (correct) " + "classification is intended, then update this oracle to match.", + ) + + +@unittest.skipUnless(shutil.which("bash"), "bash not available") +class ShellHelperIntegration(unittest.TestCase): + """active_addons.sh's is_active() must agree with active_addons.py.""" + + def test_sourced_is_active_matches_check(self) -> None: + root = tempfile.mkdtemp(prefix="active_addons_sh_") + try: + for name, body in ( + ("Active", 'register(GRAMPLET, id="a")\n'), + ("Inactive", 'register(TOOL, id="a", include_in_listing=False)\n'), + ): + d = os.path.join(root, name) + os.makedirs(d) + with open(os.path.join(d, f"{name.lower()}.gpr.py"), "w") as fh: + fh.write(body) + script = ( + f"source {_HELPER_SH}\n" + "is_active Active && echo A:active || echo A:inactive\n" + "is_active Inactive && echo I:active || echo I:inactive\n" + ) + result = subprocess.run( + ["bash", "-c", script], + cwd=root, + capture_output=True, + text=True, + check=False, + ) + self.assertIn("A:active", result.stdout, result.stderr) + self.assertIn("I:inactive", result.stdout, result.stderr) + finally: + shutil.rmtree(root, ignore_errors=True) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_addon_python_deps.py b/tests/test_addon_python_deps.py new file mode 100644 index 000000000..b3f4c4f28 --- /dev/null +++ b/tests/test_addon_python_deps.py @@ -0,0 +1,348 @@ +# +# Gramps - a GTK+/GNOME based genealogy program +# +# Copyright (C) 2026 Eduard Ralph +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# + +"""The requires_mod machinery must track Gramps' own (gramps PR #2308). + +gramps 6.1 ships an install-time authority for addon Python deps: +``gramps/gen/utils/pypi.py`` with the import→distribution table +``_IMPORT_TO_PYPI``, and a ``Requirements.check_mod`` that really imports the +module after ``find_spec`` (merged as 7f94428b13; not backported to 6.0). +``addon_python_deps.py`` accounts for both at lookup time: + +* ``_distribution_map()`` prefers the installed gramps' ``_IMPORT_TO_PYPI`` + over the local ``_IMPORT_TO_DISTRIBUTION`` fallback mirror, so 6.1+ lanes + install exactly what Gramps' own installer would — including entries added + upstream after this file was written. +* ``_module_checker()`` delegates ``--check-resolves`` to the installed + gramps' ``Requirements().check_mod``, so the gate is find_spec-only on 6.0 + and find_spec-plus-real-import on 6.1+, per lane, automatically. +* the pip-installed-ness probe uses the *mapped distribution* name — ``pip + show PIL`` fails even with Pillow installed, so probing the raw import name + silently skipped exactly the declarations the mapping machinery exists for. + +These tests pin the two seams and the probe fix hermetically (fake gramps +module trees injected via ``mock.patch.dict(sys.modules, ...)``; no network, +no real gramps needed), plus one sync-guard that compares the fallback mirror +against the real authority table wherever gramps >= 6.1 is importable. + +NOTE on the sync-guard's reach: it is LATENT until these workflows land on +``maintenance/gramps61``. Today no CI lane imports a gramps that ships +``gramps.gen.utils.pypi`` — the gramps60 image predates it, and the conda +Windows lane pins 6.0.x — so every lane skips the guard; it self-activates +when the pipeline reaches the 6.1 branch. Keep it: the cost is one skipped +test until then, and it is the only place a mirror-vs-authority drift is +caught once 6.1 arrives. + +Import guards use ``except (Exception, SystemExit)`` throughout: a +half-installed gramps (raw source checkout on sys.path) raises ``SystemExit`` +from ResourcePath at import, not ImportError. +""" + +# ------------------------ +# Python modules +# ------------------------ +from __future__ import annotations + +import io +import os +import sys +import types +import unittest +from contextlib import redirect_stdout +from unittest import mock + +_HERE = os.path.dirname(os.path.abspath(__file__)) +_REPO_ROOT = os.path.dirname(_HERE) # repo root (tests/ lives directly under it) +_SCRIPTS = os.path.join(_REPO_ROOT, ".github", "scripts") + +sys.path.insert(0, _SCRIPTS) +import addon_python_deps as deps # noqa: E402 + + +def _fake_gramps_tree(**leaves: types.ModuleType) -> dict[str, types.ModuleType]: + """A sys.modules overlay for ``gramps.gen.utils`` plus the given leaves. + + ``leaves`` maps a leaf module name (``pypi``, ``requirements``) to a fake + module; the returned dict contains the full parent chain with attributes + linked, ready for ``mock.patch.dict(sys.modules, ...)``. + """ + gramps = types.ModuleType("gramps") + gen = types.ModuleType("gramps.gen") + utils = types.ModuleType("gramps.gen.utils") + gramps.gen = gen + gen.utils = utils + tree = {"gramps": gramps, "gramps.gen": gen, "gramps.gen.utils": utils} + for name, module in leaves.items(): + setattr(utils, name, module) + tree[f"gramps.gen.utils.{name}"] = module + return tree + + +def _no_gramps_overlay() -> dict[str, None]: + """A sys.modules overlay under which every ``import gramps...`` fails. + + A top-level ``{"gramps": None}`` alone is NOT enough: ``from + gramps.gen.utils.requirements import Requirements`` short-circuits on a + cached ``sys.modules['gramps.gen.utils.requirements']`` and never consults + the None-ed parent. And a cached submodule is the norm on CI — unittest + discovery imports ``test_plugin_registration``, which imports + ``gramps.gen.utils.requirements`` at module level, before these tests run. + So None-out the top-level name AND every already-cached ``gramps.*`` key; + that restores the ImportError the fallback branches must see. + """ + overlay: dict[str, None] = {"gramps": None} + for name in list(sys.modules): + if name == "gramps" or name.startswith("gramps."): + overlay[name] = None + return overlay + + +class GrampsTableSync(unittest.TestCase): + """The fallback mirror must equal Gramps' authority table (gramps >= 6.1).""" + + def test_local_mirror_matches_gramps_table(self) -> None: + try: + from gramps.gen.utils import pypi + except (Exception, SystemExit): + self.skipTest( + "gramps.gen.utils.pypi not importable — the sync-guard gates " + "on gramps >= 6.1 lanes; the mirror is inert here" + ) + # A loud failure on rename: at runtime getattr degrades safely to the + # mirror, so THIS assertion is the only place an upstream rename of + # _IMPORT_TO_PYPI surfaces. + self.assertTrue( + hasattr(pypi, "_IMPORT_TO_PYPI"), + "gramps.gen.utils.pypi no longer exposes _IMPORT_TO_PYPI — " + "update _distribution_map() and this mirror sync-guard", + ) + self.assertEqual( + dict(pypi._IMPORT_TO_PYPI), + deps._IMPORT_TO_DISTRIBUTION, + "_IMPORT_TO_DISTRIBUTION has drifted from gramps' " + "_IMPORT_TO_PYPI — re-sync the fallback mirror in " + ".github/scripts/addon_python_deps.py", + ) + + +class DistributionMapSeam(unittest.TestCase): + """_distribution_map(): authority when present, mirror otherwise.""" + + def test_prefers_gramps_table(self) -> None: + fake_pypi = types.ModuleType("gramps.gen.utils.pypi") + fake_pypi._IMPORT_TO_PYPI = {"PIL": "Pillow", "fake_mod": "fake-dist"} + with mock.patch.dict(sys.modules, _fake_gramps_tree(pypi=fake_pypi)): + table = deps._distribution_map() + self.assertEqual(table["fake_mod"], "fake-dist") + self.assertEqual(table["PIL"], "Pillow") + + def test_falls_back_without_gramps(self) -> None: + # Seed a cached gramps.gen.utils.pypi FIRST (as the integration lane's + # discovery would leave gramps submodules cached), then apply the + # overlay on top: the fallback must still fire, proving the overlay + # defeats a pre-cached submodule and not merely an absent gramps. + fake_pypi = types.ModuleType("gramps.gen.utils.pypi") + fake_pypi._IMPORT_TO_PYPI = {"PIL": "Pillow", "seeded": "seeded-dist"} + with mock.patch.dict(sys.modules, _fake_gramps_tree(pypi=fake_pypi)): + with mock.patch.dict(sys.modules, _no_gramps_overlay()): + table = deps._distribution_map() + self.assertEqual(table, deps._IMPORT_TO_DISTRIBUTION) + + def test_falls_back_when_table_attr_missing(self) -> None: + # Pins the safe-degradation path an upstream rename would take. + fake_pypi = types.ModuleType("gramps.gen.utils.pypi") + with mock.patch.dict(sys.modules, _fake_gramps_tree(pypi=fake_pypi)): + table = deps._distribution_map() + self.assertEqual(table, deps._IMPORT_TO_DISTRIBUTION) + + +class ModuleCheckerSeam(unittest.TestCase): + """_module_checker(): Gramps' own gate when present, find_spec otherwise.""" + + def test_delegates_to_gramps_check_mod(self) -> None: + calls: list[str] = [] + + class _FakeRequirements: + def check_mod(self, name: str) -> bool: + calls.append(name) + return name == "good_mod" + + fake_req = types.ModuleType("gramps.gen.utils.requirements") + fake_req.Requirements = _FakeRequirements + with mock.patch.dict(sys.modules, _fake_gramps_tree(requirements=fake_req)): + label, check = deps._module_checker() + self.assertIn("check_mod", label) + self.assertTrue(check("good_mod")) + self.assertFalse(check("bad_mod")) + self.assertEqual(calls, ["good_mod", "bad_mod"]) + + def test_stdlib_fallback(self) -> None: + # Seed a cached gramps.gen.utils.requirements FIRST (exactly what + # unittest discovery of test_plugin_registration leaves behind on the + # integration lane), then overlay: the checker must STILL fall back to + # find_spec. Without the full-tree overlay this false-red'd — the + # delegated path was taken because the submodule was already cached. + fake_req = types.ModuleType("gramps.gen.utils.requirements") + + class _FakeRequirements: + def check_mod(self, name: str) -> bool: # pragma: no cover + return True + + fake_req.Requirements = _FakeRequirements + with mock.patch.dict(sys.modules, _fake_gramps_tree(requirements=fake_req)): + with mock.patch.dict(sys.modules, _no_gramps_overlay()): + label, check = deps._module_checker() + self.assertIn("find_spec", label) + self.assertTrue(check("os")) + self.assertFalse(check("definitely_not_a_module_xyz")) + + +class CheckResolvesGate(unittest.TestCase): + """check_resolves(): probe by distribution name, judge by import name.""" + + def _run( + self, + *, + check, + pip_ok_for: set[str], + recorded: list[list[str]], + declared: set[str] | None = None, + dist_map: dict[str, str] | None = None, + ): + """Drive check_resolves hermetically. + + Defaults to one declared mod, ``PIL`` → ``Pillow``. Pass ``declared`` / + ``dist_map`` to exercise the wheel-only vs source-built branches against + the REAL classification sets (WHEEL_ONLY_MODS is not mocked). + """ + declared = {"PIL"} if declared is None else declared + dist_map = {"PIL": "Pillow"} if dist_map is None else dist_map + + def fake_run(argv, **kwargs): + recorded.append(list(argv)) + rc = 0 if argv[-1] in pip_ok_for else 1 + return types.SimpleNamespace(returncode=rc) + + out = io.StringIO() + with ( + mock.patch.object(deps, "declared_mods", return_value=declared), + mock.patch.object(deps, "_distribution_map", return_value=dist_map), + mock.patch.object( + deps, "_module_checker", return_value=("test gate", check) + ), + mock.patch("subprocess.run", side_effect=fake_run), + redirect_stdout(out), + ): + rc = deps.check_resolves(".") + return rc, out.getvalue() + + def test_probe_uses_distribution_name(self) -> None: + # Regression: `pip show PIL` fails even with Pillow installed, so the + # old raw-name probe skipped ("~") the one declaration the mapping + # machinery exists for — never validating it. + recorded: list[list[str]] = [] + checked: list[str] = [] + + def check(name: str) -> bool: + checked.append(name) + return True + + rc, out = self._run(check=check, pip_ok_for={"Pillow"}, recorded=recorded) + self.assertEqual(rc, 0) + pip_show = [argv for argv in recorded if "show" in argv] + self.assertTrue(pip_show and pip_show[0][-1] == "Pillow", pip_show) + self.assertNotIn("~", out) + self.assertIn("ok PIL", out) + # The gate must JUDGE the raw import name, never the distribution name — + # gramps' check_mod("Pillow") would wrongly fail a correct requires_mod + # =["PIL"]. (Kills the M1b mutant that passed `dist` to the checker.) + self.assertEqual(checked, ["PIL"]) + + def test_installed_but_unresolvable_fails(self) -> None: + recorded: list[list[str]] = [] + checked: list[str] = [] + + def check(name: str) -> bool: + checked.append(name) + return False + + rc, out = self._run(check=check, pip_ok_for={"Pillow"}, recorded=recorded) + self.assertEqual(rc, 1) + self.assertIn("x PIL", out) + self.assertIn("::error::", out) + self.assertEqual(checked, ["PIL"]) + + def test_wheel_only_never_installed_fails(self) -> None: + # A wheel-only dep (PIL is in the real WHEEL_ONLY_MODS) that pip never + # installed is a provisioning regression, not an environment gap: the + # gate must FAIL, and must not even consult the dep checker. + self.assertIn("PIL", deps.WHEEL_ONLY_MODS) # guard the fixture premise + recorded: list[list[str]] = [] + checked: list[str] = [] + + def check(name: str) -> bool: + checked.append(name) + return True + + rc, out = self._run( + check=check, pip_ok_for=set(), recorded=recorded + ) # nothing installs + self.assertEqual(rc, 1) + self.assertIn("wheel-only", out) + self.assertIn("::error::Wheel-only", out) + self.assertEqual( + checked, [], "a never-installed wheel must not be gate-checked" + ) + + def test_source_built_never_installed_stays_advisory(self) -> None: + # A source-built dep (pygraphviz is in the real MOD_BUILD_PACKAGES, not + # WHEEL_ONLY_MODS) may legitimately miss on an image/system gap: advisory + # skip, rc 0. + self.assertNotIn("pygraphviz", deps.WHEEL_ONLY_MODS) # guard the premise + recorded: list[list[str]] = [] + rc, out = self._run( + check=lambda name: True, + pip_ok_for=set(), + recorded=recorded, + declared={"pygraphviz"}, + dist_map={}, + ) + self.assertEqual(rc, 0) + self.assertIn("~ pygraphviz", out) + + +class NonStringRequiresMod(unittest.TestCase): + """A tuple-shaped requires_mod entry must be skipped, not crash the CLI.""" + + def test_tuple_entry_skipped_not_fatal(self) -> None: + import tempfile + + with tempfile.TemporaryDirectory() as root: + addon = os.path.join(root, "Addon") + os.makedirs(addon) + with open(os.path.join(addon, "x.gpr.py"), "w", encoding="utf-8") as fh: + fh.write('requires_mod = [("psycopg2", ">=2"), "svgwrite"]\n') + # Would previously TypeError in sorted() mixing tuple and str. + self.assertEqual(deps.declared_mods(root), {"svgwrite"}) + self.assertEqual(deps.install_list(root), ["svgwrite"]) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_addon_system_deps.py b/tests/test_addon_system_deps.py new file mode 100644 index 000000000..7e7f6c5ee --- /dev/null +++ b/tests/test_addon_system_deps.py @@ -0,0 +1,328 @@ +# +# Gramps - a GTK+/GNOME based genealogy program +# +# Copyright (C) 2026 Eduard Ralph +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# + +"""Source-built addon deps must be provisioned in CI — never a silent skip. + +Regression test for the build-toolchain coverage gap (addons-source PR #820): +the CI image purged the build toolchain (gcc/python3-dev/pkg-config) after the +Gramps install, and the system packages a source-built ``requires_mod`` needs +were not provisioned on either lane. So ``pip install pygraphviz`` / ``psycopg2`` +/ ``psycopg`` failed at CI runtime, the failure was swallowed by the install +step's ``|| echo … (continuing)``, and the affected addon's coverage degraded +while the job stayed green — silent coverage loss reported as success. + +The fix restores the invariant *declared addon dependencies are honestly +satisfied or honestly skipped* over the WHOLE category of source-built +``requires_mod``, on BOTH CI lanes: + +* apt — every source-built ``requires_mod`` has its ``-dev`` / libpq package in + the single-source map (``addon_system_deps.MOD_BUILD_PACKAGES``), and the + compiler toolchain stays in the CI image + (``.github/docker/gramps-ci/Dockerfile`` no longer purges it), so pip builds / + links the extension. +* conda — the same modules map to their prebuilt conda-forge package, so the + Windows lane installs them (``mamba install``) instead of silently skipping; + the conda side is NOT ``None``. + +It also closes the category over *future* additions: every declared +``requires_mod`` must be classified (wheel-only or source-built), and the +``--unmapped`` drift guard fails CI on any module that is neither — so a +newly-added source-built dep cannot quietly reopen the gap. + +Why this lives in ``tests/`` and not ``.github/scripts/tests/``: the C4 +red→green runner derives the unittest module name from the test path +(``path.replace('/', '.')``); a ``.github/``-rooted path yields a leading-dot +module name that ``python3 -m unittest`` rejects. ``tests/`` is the repo's real +test package (alongside ``test_plugin_load_gate.py``), so the module name is +importable. Pure stdlib / no ``gi`` / no ``gramps.gui`` imports — it runs under +the headless runner. It exercises the production derivation path (the same +``packages()`` / ``unmapped()`` / CLI that ci.yml calls), not a copy of it. +""" + +# ------------------------ +# Python modules +# ------------------------ +from __future__ import annotations + +import ast +import glob +import io +import os +import re +import sys +import unittest +from contextlib import redirect_stdout + +_HERE = os.path.dirname(os.path.abspath(__file__)) +_REPO_ROOT = os.path.dirname(_HERE) # repo root (tests/ lives directly under it) +_SCRIPTS = os.path.join(_REPO_ROOT, ".github", "scripts") +_DOCKERFILE = os.path.join(_REPO_ROOT, ".github", "docker", "gramps-ci", "Dockerfile") + +sys.path.insert(0, _SCRIPTS) +import addon_system_deps as deps # noqa: E402 + +# The requires_mod that are built from / linked against a system package in CI +# (no plain pip wheel that imports unaided on at least one lane). Kept here as the +# test's own statement of the category, independent of the production map, so a +# regression that drops an entry from MOD_BUILD_PACKAGES is caught rather than +# mirrored. ``psycopg`` (psycopg3, PostgreSQLEnhanced) is included alongside +# ``psycopg2``/``pygraphviz`` — its pure-Python build still needs libpq at import. +_SOURCE_BUILT_MODS = ("pygraphviz", "psycopg2", "psycopg") + +_REQUIRES_MOD_RE = re.compile(r"requires_mod\s*=\s*(\[[^\]]*\])") + + +def _declared_requires_mod() -> set[str]: + """Union of requires_mod across every .gpr.py in the repo (as ci.yml derives).""" + mods: set[str] = set() + for path in glob.glob(os.path.join(_REPO_ROOT, "*", "*.gpr.py")): + try: + with open(path, encoding="utf-8") as fh: + text = fh.read() + except OSError: + continue + for match in _REQUIRES_MOD_RE.finditer(text): + try: + mods.update(ast.literal_eval(match.group(1))) + except (ValueError, SyntaxError): + pass + return mods + + +class SourceBuiltModBuildDeps(unittest.TestCase): + """The install list must carry every source-built requires_mod, per lane.""" + + def test_known_source_built_mods_are_declared_by_some_addon(self): + # Grounds the rest of the suite in real declarations: if these addons ever + # drop the dep this test would otherwise pass vacuously. + declared = _declared_requires_mod() + for mod in _SOURCE_BUILT_MODS: + self.assertIn( + mod, + declared, + f"expected some addon's .gpr.py to declare requires_mod={mod!r}", + ) + + def test_packages_apt_includes_build_headers(self): + apt = deps.packages("apt") + for mod in _SOURCE_BUILT_MODS: + pkg = deps.MOD_BUILD_PACKAGES.get(mod, {}).get("apt") + self.assertIsNotNone( + pkg, + f"source-built requires_mod {mod!r} has no apt package in " + "MOD_BUILD_PACKAGES — its CI pip build/import will fail for a " + "missing header/library", + ) + self.assertIn( + pkg, + apt, + f"{pkg!r} (system package for {mod!r}) missing from packages('apt'); " + "ci.yml would not install it and the source build would be silently skipped", + ) + + def test_packages_conda_provisions_source_built_mods(self): + # The conda (Windows) lane provisions its own deps and must mirror apt: + # pygraphviz/psycopg2/psycopg ship prebuilt on conda-forge, so the conda + # side is the module's own conda-forge package — NEVER None (the + # silent-skip the invariant forbids over the whole category). + conda = deps.packages("conda") + for mod in _SOURCE_BUILT_MODS: + pkg = deps.MOD_BUILD_PACKAGES.get(mod, {}).get("conda") + self.assertIsNotNone( + pkg, + f"source-built requires_mod {mod!r} maps to conda=None — the conda " + "lane would not install it and the failed pip build would be " + "swallowed into a silently-degraded green. It is on conda-forge; map it.", + ) + self.assertIn( + pkg, + conda, + f"{pkg!r} (conda package for {mod!r}) missing from packages('conda'); " + "the Windows lane would skip the addon's suite instead of running it", + ) + + def test_cli_apt_emits_build_headers(self): + # Exercise the exact production entry point ci.yml calls: + # pkgs=$(python3 addon_system_deps.py --platform apt) + buf = io.StringIO() + with redirect_stdout(buf): + rc = deps.main(["--platform", "apt"]) + self.assertEqual(rc, 0) + emitted = buf.getvalue().split() + self.assertIn("libgraphviz-dev", emitted) + self.assertIn("libpq-dev", emitted) + + def test_cli_conda_emits_source_built_mods(self): + # The conda step calls: pkgs=$(python addon_system_deps.py --platform conda) + buf = io.StringIO() + with redirect_stdout(buf): + rc = deps.main(["--platform", "conda"]) + self.assertEqual(rc, 0) + emitted = buf.getvalue().split() + for mod in _SOURCE_BUILT_MODS: + self.assertIn( + mod, + emitted, + f"conda --platform output is missing {mod!r}; the Windows lane " + "would not install it", + ) + + def test_every_declared_source_built_mod_is_provisioned_on_both_lanes(self): + # Category guard: any source-built requires_mod an addon declares must be + # mapped AND surfaced on BOTH lanes, so a newly added one cannot silently + # lose coverage on apt or on conda. + apt = set(deps.packages("apt")) + conda = set(deps.packages("conda")) + for mod in _declared_requires_mod() & set(_SOURCE_BUILT_MODS): + entry = deps.MOD_BUILD_PACKAGES[mod] + for platform, available in (("apt", apt), ("conda", conda)): + pkg = entry.get(platform) + self.assertIsNotNone( + pkg, + f"{mod!r} maps to {platform}=None — silent skip on {platform}", + ) + self.assertIn(pkg, available) + + +class RequiresModCategoryIsComplete(unittest.TestCase): + """Every declared requires_mod must be classified — no silent new gap.""" + + def test_every_declared_requires_mod_is_classified(self): + # The heart of C5(b): each declared requires_mod is EITHER a known + # wheel-only module OR a mapped source-built one. An unclassified module + # is exactly how a future source-built dep would silently lose coverage. + classified = set(deps.WHEEL_ONLY_MODS) | set(deps.MOD_BUILD_PACKAGES) + for mod in _declared_requires_mod(): + self.assertIn( + mod, + classified, + f"requires_mod {mod!r} is in neither WHEEL_ONLY_MODS nor " + "MOD_BUILD_PACKAGES — classify it (a source-built one needs a " + "system-package mapping or its CI coverage silently degrades)", + ) + + def test_unmapped_reports_no_mod_drift(self): + # Drive the production drift guard ci.yml's "Validate addon system deps + # are mapped" step runs (python3 addon_system_deps.py --unmapped .). It + # must report no unmapped requires_mod for the current addon set. + _gi, _exe, mod = deps.unmapped(_REPO_ROOT) + self.assertEqual( + mod, + set(), + f"--unmapped reports unclassified requires_mod {sorted(mod)!r}; " + "ci.yml's mapping gate would fail. Classify each as wheel-only or " + "source-built.", + ) + + def test_unmapped_cli_exit_zero(self): + # The full CLI the validate step invokes must exit 0 (no drift) today. + buf = io.StringIO() + with redirect_stdout(buf): + rc = deps.main(["--unmapped", _REPO_ROOT]) + self.assertEqual( + rc, + 0, + "addon_system_deps.py --unmapped exits non-zero — an addon declares a " + f"GI/exe/mod dep with no mapping:\n{buf.getvalue()}", + ) + + +# The compiler toolchain a source-built requires_mod (pygraphviz, psycopg2, …) +# needs at CI runtime. All three must stay installed and unpurged in the image. +_TOOLCHAIN = ("gcc", "pkg-config", "python3-dev") +# `apt-get` followed by any flags/words and then a removal verb / an install. +_APT_PURGE_RE = re.compile(r"\bapt-get\b(?:\s+\S+)*?\s+(?:purge|remove|autoremove)\b") +_APT_INSTALL_RE = re.compile(r"\bapt-get\b(?:\s+\S+)*?\s+install\b") + + +def _mentions(text: str, tool: str): + """A whole-token match for a package name (so pkg-config != pkg-configurator).""" + return re.search(rf"(?=2"), "svgwrite"]\n') + # scan_modules / the --unmapped CLI would previously TypeError on the + # tuple entry (unhashable for a set, or sorted() mixing types). + self.assertEqual(deps.scan_modules(root), {"svgwrite"}) + self.assertEqual(deps.unmapped(root), (set(), set(), set())) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_requires_mod_dedup.py b/tests/test_requires_mod_dedup.py new file mode 100644 index 000000000..a468f3a69 --- /dev/null +++ b/tests/test_requires_mod_dedup.py @@ -0,0 +1,189 @@ +"""Regression test for the requires_mod / is_active CI de-duplication. + +ci.yml used to inline an identical ``requires_mod`` derivation heredoc in three +jobs and an identical ``is_active()`` bash helper in ~six job steps. The fix +moves the derivation into ``.github/scripts/addon_python_deps.py`` and the +helper into ``.github/scripts/active_addons.sh``, each consumed from one place. + +This test pins two things, so the duplication cannot silently come back and the +refactor is proven behaviour-preserving: + +1. Behaviour preservation — the single ``addon_python_deps`` module derives the + *same* install union and the *same* raw declared-name set the old inline + heredoc did (computed here by an independent oracle over the real tree). + +2. The DRY invariant, stated per-category — NO inline ``is_active()`` definition + and NO ``requires_mod`` heredoc survive in ci.yml, and EVERY job step that + *calls* ``is_active`` sources the shared helper (a missed step is caught, not + masked by an "at least one source" check). + +Pure stdlib / GUI-import-free on purpose: it imports the production module the +ci.yml jobs call (not a copy), runs headless, and needs no gi / gramps.gui. +""" + +from __future__ import annotations + +import ast +import glob +import os +import re +import sys +import unittest +from unittest import mock + +# Repo layout is fixed relative to this file: tests/ sits at the addons-source +# root, and the CI scripts live under .github/scripts/ — resolve both from +# __file__ so the test is cwd-independent (the C4 runner cd's into the repo, CI +# discover runs from the root). +_REPO_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +_SCRIPTS = os.path.join(_REPO_ROOT, ".github", "scripts") +_CI_YML = os.path.join(_REPO_ROOT, ".github", "workflows", "ci.yml") +_HELPER = os.path.join(_SCRIPTS, "active_addons.sh") + +if _SCRIPTS not in sys.path: + sys.path.insert(0, _SCRIPTS) + +# Imported at module load: the production module the three ci.yml jobs invoke. +# With the fix reverted (module file removed) this import fails and every test +# below errors — the red half of the red->green contract. +import addon_python_deps # noqa: E402 + +# --- independent oracle: the OLD inline heredoc algorithm, verbatim ---------- +_OLD_RE = re.compile(r"requires_mod\s*=\s*(\[[^\]]*\])") +# Install-name map the heredoc lacked; install-only (find_spec gate stays raw). +_INSTALL_MAP = {"PIL": "Pillow"} + + +def _read(path): + with open(path, encoding="utf-8") as fh: + return fh.read() + + +def _old_raw_union(root): + mods = set() + for fn in sorted(glob.glob(os.path.join(root, "*", "*.gpr.py"))): + try: + text = _read(fn) + except OSError: + continue + for m in _OLD_RE.finditer(text): + try: + mods.update(ast.literal_eval(m.group(1))) + except (ValueError, SyntaxError): + pass + return {m for m in mods if m} + + +def _ci_steps(): + """Yield each ci.yml step body as a string. A step starts at a 6-space + `- ` line and runs until the next one (no YAML dependency needed).""" + lines = _read(_CI_YML).splitlines(keepends=True) + steps, cur = [], None + for line in lines: + if re.match(r"^ - ", line): + if cur is not None: + steps.append("".join(cur)) + cur = [line] + elif cur is not None: + cur.append(line) + if cur is not None: + steps.append("".join(cur)) + return steps + + +class RequiresModDerivationDedup(unittest.TestCase): + """The derivation is single-sourced AND behaviour-preserving.""" + + def test_install_list_matches_old_heredoc(self): + old = _old_raw_union(_REPO_ROOT) + expected = sorted(_INSTALL_MAP.get(m, m) for m in old) + # Pin the import->distribution table to the LOCAL mirror for this + # comparison. The oracle (_INSTALL_MAP) states the old heredoc's + # behaviour plus the install-only map; production install_list() now + # ALSO consults gramps' authoritative _IMPORT_TO_PYPI when a gramps + # >= 6.1 is importable (the gramps61 lanes). Without this pin, an + # upstream table gaining a mapping for a *declared* mod would flip this + # oracle red for a change that is not a regression here. If that + # happens: the GrampsTableSync guard reds first (re-sync the mirror in + # addon_python_deps.py), then extend _INSTALL_MAP above to match. + with mock.patch.object( + addon_python_deps, + "_distribution_map", + return_value=dict(addon_python_deps._IMPORT_TO_DISTRIBUTION), + ): + self.assertEqual(addon_python_deps.install_list(_REPO_ROOT), expected) + + def test_declared_raw_names_match_old_heredoc(self): + # The find_spec gate consumes RAW import names — these must equal the + # old union exactly (the install map must NOT leak into them). + self.assertEqual( + addon_python_deps.declared_mods(_REPO_ROOT), _old_raw_union(_REPO_ROOT) + ) + + def test_install_map_is_install_only(self): + # PIL maps to Pillow on the install side, but the raw declared-name set + # never contains the distribution name (so Gramps' find_spec gate, which + # the module's --check-resolves mirrors, keeps checking the import name). + self.assertNotIn("Pillow", addon_python_deps.declared_mods(_REPO_ROOT)) + + def test_no_requires_mod_heredoc_remains(self): + # The previous assertion (re.findall(r"requires_mod\s*=\s*\(\[", ...)) + # was a tautology: the old inline heredoc's distinctive line was + # pat = re.compile(r"requires_mod\s*=\s*(\[[^\]]*\])") + # whose text contains the LITERAL characters `\s*`, which the guard's + # own `\s*` (matching whitespace) can never match — so it never bit, + # and pasting the heredoc back in stayed green. Match the heredoc's + # own literal fragments instead; both appear verbatim in the pre-dedup + # ci.yml and in none of the current file. + text = _read(_CI_YML) + for fragment in ('re.compile(r"requires_mod', r"requires_mod\s*"): + self.assertNotIn( + fragment, + text, + "a requires_mod derivation heredoc still lives inline in ci.yml " + f"(found {fragment!r})", + ) + + def test_three_jobs_consume_the_module(self): + text = _read(_CI_YML) + self.assertEqual( + len(re.findall(r"addon_python_deps\.py --install-list", text)), 3 + ) + self.assertEqual( + len(re.findall(r"addon_python_deps\.py --check-resolves", text)), 3 + ) + + +class IsActiveHelperDedup(unittest.TestCase): + """is_active() lives in one sourced helper, consumed by EVERY filtering step.""" + + def test_helper_file_defines_is_active(self): + self.assertTrue(os.path.isfile(_HELPER)) + self.assertIn("is_active()", _read(_HELPER)) + + def test_no_inline_is_active_definition_remains(self): + text = _read(_CI_YML) + self.assertEqual( + re.findall(r"is_active\(\)\s*\{", text), + [], + "an inline is_active() definition still lives in ci.yml", + ) + + def test_every_is_active_call_site_sources_the_helper(self): + # Per-category invariant: every job step that CALLS is_active must also + # source the shared helper. Asserting "sourced at least once" would miss + # a step that calls a now-undefined is_active; this checks each site. + calling = [s for s in _ci_steps() if re.search(r'is_active\s+"', s)] + self.assertGreaterEqual( + len(calling), 6, "expected >=6 active-addon filtering steps" + ) + for step in calling: + self.assertIn( + "source .github/scripts/active_addons.sh", + step, + "a step calls is_active without sourcing active_addons.sh:\n" + step, + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_run_addon_tests_paths.py b/tests/test_run_addon_tests_paths.py new file mode 100644 index 000000000..3b2666bb3 --- /dev/null +++ b/tests/test_run_addon_tests_paths.py @@ -0,0 +1,326 @@ +# +# Gramps - a GTK+/GNOME based genealogy program +# +# Copyright (C) 2026 Eduard Ralph +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# + +""" +Tests for ``run_addon_tests.py``'s per-addon import root (issue #52). + +The runner appends each addon's own directory to ``sys.path`` before loading +its tests, mirroring how Gramps' plugin loader puts the addon dir on the path. +This lets an addon's tests import the addon's top-level modules +(``from import …``) — including a nested-package addon whose test +modules live under ``/tests//`` — while the repo-root shared +``tests`` environment still wins (APPEND, not prepend). + +Driven via subprocess against synthetic addon trees in a temp dir, so no gramps +install is needed. +""" + +# ------------------------ +# Python modules +# ------------------------ +import os +import shutil +import subprocess +import sys +import tempfile +import unittest + + +ADDONS_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +RUN_ADDON_TESTS = os.path.join(ADDONS_ROOT, ".github", "scripts", "run_addon_tests.py") + + +@unittest.skipUnless( + os.path.isfile(RUN_ADDON_TESTS), "run_addon_tests.py not present on this branch" +) +class RunAddonTestsPathsTest(unittest.TestCase): + """The runner must load nested- and flat-package addon tests by adding the + addon dir to sys.path, without shadowing the shared repo-root tests env.""" + + def setUp(self) -> None: + self.root = tempfile.mkdtemp(prefix="run_addon_tests_paths_") + # Stand-in for the repo-root shared Gramps-emulation env (PR 950): + # a top-level `tests` package the addon tests may import. + self._write("tests/__init__.py", "") + self._write("tests/gramps_test_env.py", 'SENTINEL = "repo-root-shared-env"\n') + + def tearDown(self) -> None: + shutil.rmtree(self.root, ignore_errors=True) + + def _write(self, relpath: str, content: str) -> None: + path = os.path.join(self.root, relpath) + os.makedirs(os.path.dirname(path), exist_ok=True) + with open(path, "w", encoding="utf-8") as fp: + fp.write(content) + + def _run( + self, + modname: str, + platform: str = "apt", + extra_env: dict | None = None, + ) -> subprocess.CompletedProcess: + env = os.environ.copy() + env["PYTHONPATH"] = "." # repo root on sys.path, as ci.yml sets it + if extra_env: + env.update(extra_env) + return subprocess.run( + [ + sys.executable, + RUN_ADDON_TESTS, + "--platform", + platform, + "--root", + self.root, + modname, + ], + cwd=self.root, + env=env, + capture_output=True, + text=True, + check=False, + ) + + def test_nested_package_addon_loads(self) -> None: + """A nested-package addon: test module under tests//, importing + the addon's top-level lib AND the shared repo-root tests env.""" + # Namespace-package top (no SynthAddon/__init__.py); top-level lib module. + self._write("SynthAddon/synthlib.py", "VALUE = 42\n") + self._write("SynthAddon/tests/__init__.py", "") + self._write("SynthAddon/tests/sub/__init__.py", "") + self._write( + "SynthAddon/tests/sub/test_nested.py", + "import unittest\n" + "from synthlib import VALUE\n" # per-addon import root + "from tests.gramps_test_env import SENTINEL\n" # shared env not shadowed + "\n" + "class T(unittest.TestCase):\n" + " def test_addon_lib(self):\n" + " self.assertEqual(VALUE, 42)\n" + " def test_shared_env(self):\n" + ' self.assertEqual(SENTINEL, "repo-root-shared-env")\n', + ) + + result = self._run("SynthAddon.tests.sub.test_nested") + out = result.stdout + result.stderr + self.assertEqual(result.returncode, 0, "runner failed:\n%s" % out) + self.assertIn("ok SynthAddon.tests.sub.test_nested", result.stdout, out) + + def test_flat_model_b_addon_loads(self) -> None: + """A flat addon whose test imports a top-level addon module + (WebSearch-style) — currently broken without the per-addon import root.""" + self._write("FlatAddon/flatlib.py", "FLAT = 7\n") + self._write("FlatAddon/tests/__init__.py", "") + self._write( + "FlatAddon/tests/test_flat.py", + "import unittest\n" + "from flatlib import FLAT\n" + "\n" + "class T(unittest.TestCase):\n" + " def test_flat(self):\n" + " self.assertEqual(FLAT, 7)\n", + ) + + result = self._run("FlatAddon.tests.test_flat") + out = result.stdout + result.stderr + self.assertEqual(result.returncode, 0, "runner failed:\n%s" % out) + self.assertIn("ok FlatAddon.tests.test_flat", result.stdout, out) + + # ------------------------------------------------------------------ + # Outcome classification (load-failure taxonomy, zero-test, timeout, + # tolerant timeout env) — adversarial-review findings F1-F4/F10. + # GooCanvas is apt-provisioned but conda:None (addon_system_deps.GI_PACKAGES), + # so a `requires_gi=[("GooCanvas","2.0")]` addon is unsatisfiable on conda + # and satisfiable on apt — the lever these tests use. + # ------------------------------------------------------------------ + def _write_gi_addon(self, addon: str, test_body: str) -> None: + self._write( + f"{addon}/{addon.lower()}.gpr.py", + f'register(GRAMPLET, id="{addon}", requires_gi=[("GooCanvas", "2.0")])\n', + ) + self._write(f"{addon}/tests/__init__.py", "") + self._write(f"{addon}/tests/test_x.py", test_body) + + def test_syntax_error_fails_even_on_unsatisfiable_platform(self) -> None: + # A SyntaxError is a code bug, never a dependency shape — it must FAIL + # even where the addon's declared GI dep is unavailable (conda). + self._write_gi_addon("GiSyntax", "def broken(:\n pass\n") + result = self._run("GiSyntax.tests.test_x", platform="conda") + out = result.stdout + result.stderr + self.assertNotEqual(result.returncode, 0, out) + self.assertIn("not dependency-shaped", out) + + def test_dep_import_error_skips_on_unsatisfiable_platform(self) -> None: + # A dependency-shaped load failure (ImportError) where the addon's deps + # are unavailable is an expected platform skip. + self._write_gi_addon( + "GiDepSkip", "import definitely_not_installed_xyz # noqa\n" + ) + result = self._run("GiDepSkip.tests.test_x", platform="conda") + out = result.stdout + result.stderr + self.assertEqual(result.returncode, 0, out) + self.assertIn("skip", result.stdout) + + def test_dep_import_error_fails_on_satisfiable_platform(self) -> None: + # The same ImportError where the addon declares no unsatisfiable dep + # (satisfiable on apt) is a real failure, not a skip. + self._write( + "PlainAddon/plainaddon.gpr.py", 'register(GRAMPLET, id="PlainAddon")\n' + ) + self._write("PlainAddon/tests/__init__.py", "") + self._write( + "PlainAddon/tests/test_x.py", + "import definitely_not_installed_xyz # noqa\n", + ) + result = self._run("PlainAddon.tests.test_x", platform="apt") + out = result.stdout + result.stderr + self.assertNotEqual(result.returncode, 0, out) + + def test_zero_collected_tests_fails(self) -> None: + # A module that loads but collects no tests reads as green under plain + # unittest; the runner must fail it. + self._write( + "EmptyAddon/emptyaddon.gpr.py", 'register(GRAMPLET, id="EmptyAddon")\n' + ) + self._write("EmptyAddon/tests/__init__.py", "") + self._write( + "EmptyAddon/tests/test_x.py", + "class NotATestCase:\n def test_nope(self):\n pass\n", + ) + result = self._run("EmptyAddon.tests.test_x", platform="apt") + out = result.stdout + result.stderr + self.assertNotEqual(result.returncode, 0, out) + self.assertIn("zero tests", out) + + def test_non_integer_timeout_env_is_tolerated(self) -> None: + # A non-integer RUN_ADDON_TESTS_TIMEOUT must not crash the runner. + self._write("OkAddon/okaddon.gpr.py", 'register(GRAMPLET, id="OkAddon")\n') + self._write("OkAddon/tests/__init__.py", "") + self._write( + "OkAddon/tests/test_x.py", + "import unittest\n" + "class T(unittest.TestCase):\n" + " def test_ok(self):\n" + " self.assertTrue(True)\n", + ) + result = self._run( + "OkAddon.tests.test_x", + platform="apt", + extra_env={"RUN_ADDON_TESTS_TIMEOUT": "soon"}, + ) + out = result.stdout + result.stderr + self.assertEqual(result.returncode, 0, out) + self.assertIn("ignoring non-integer", result.stderr) + + def test_same_named_addon_module_does_not_shadow_package(self) -> None: + # Regression: many addons ship /.py. Once / is on + # sys.path (for the tests' bare sibling imports) that regular module wins + # the bare name over the namespace-package directory, and the dotted test + # name .tests.test_x died with "module '' has no attribute + # 'tests'" — 11 real addons failed this way on CI. + self._write("ShadowAddon/shadowaddon.gpr.py", 'register(GRAMPLET, id="s")\n') + self._write("ShadowAddon/ShadowAddon.py", "MAIN = 'addon main module'\n") + self._write("ShadowAddon/sibling.py", "SIB = 5\n") + self._write("ShadowAddon/tests/__init__.py", "") + self._write( + "ShadowAddon/tests/test_x.py", + "import unittest\n" + "from sibling import SIB\n" # bare sibling import needs addon dir + "\n" + "class T(unittest.TestCase):\n" + " def test_sibling(self):\n" + " self.assertEqual(SIB, 5)\n", + ) + result = self._run("ShadowAddon.tests.test_x", platform="apt") + out = result.stdout + result.stderr + self.assertEqual(result.returncode, 0, out) + self.assertIn("ok ShadowAddon.tests.test_x", result.stdout, out) + + def test_class_level_skip_is_not_reported_as_zero_tests(self) -> None: + # setUpClass raising SkipTest yields tests=0 with skips recorded. That is + # a skipped module, not an empty one — the zero-collected rule must not + # claim "collected zero tests" (it did, on RepositoriesReport in CI). + # The addon declares an unsatisfiable-on-conda dep so the all-skipped + # rule treats it as an expected platform skip. + self._write_gi_addon( + "GiClassSkip", + "import unittest\n" + "class T(unittest.TestCase):\n" + " @classmethod\n" + " def setUpClass(cls):\n" + " raise unittest.SkipTest('fixture not available')\n" + " def test_a(self):\n" + " pass\n", + ) + result = self._run("GiClassSkip.tests.test_x", platform="conda") + out = result.stdout + result.stderr + self.assertEqual(result.returncode, 0, out) + self.assertNotIn("zero tests", out) + self.assertIn("skip", result.stdout) + + def test_module_level_skiptest_is_honored(self) -> None: + # A module that raises SkipTest at import is explicitly opting out (the + # addon's own "needs a display / PyGObject" guard) — honour it as a skip + # on every platform, never a failure. + self._write("SkipAddon/skipaddon.gpr.py", 'register(GRAMPLET, id="s")\n') + self._write("SkipAddon/tests/__init__.py", "") + self._write( + "SkipAddon/tests/test_x.py", + "import unittest\nraise unittest.SkipTest('no display here')\n", + ) + result = self._run("SkipAddon.tests.test_x", platform="apt") + out = result.stdout + result.stderr + self.assertEqual(result.returncode, 0, out) + self.assertIn("opted out", out) + + @unittest.skipUnless(os.name == "posix", "process-group kill is POSIX-only") + def test_timeout_reaps_grandchild_holding_stdout(self) -> None: + # A test that spawns a long-lived child inheriting the worker's stdout + # must not defeat the timeout: the whole process group is reaped and the + # follow-up communicate() is bounded, so the runner returns promptly. + import time + + self._write( + "HangAddon/hangaddon.gpr.py", 'register(GRAMPLET, id="HangAddon")\n' + ) + self._write("HangAddon/tests/__init__.py", "") + self._write( + "HangAddon/tests/test_x.py", + "import subprocess, sys, time, unittest\n" + "class T(unittest.TestCase):\n" + " def test_hang(self):\n" + # child inherits stdout (the worker's pipe); then the test blocks + " subprocess.Popen([sys.executable, '-c', 'import time; time.sleep(120)'])\n" + " time.sleep(120)\n", + ) + start = time.monotonic() + result = self._run( + "HangAddon.tests.test_x", + platform="apt", + extra_env={"RUN_ADDON_TESTS_TIMEOUT": "3"}, + ) + elapsed = time.monotonic() - start + out = result.stdout + result.stderr + self.assertNotEqual(result.returncode, 0, out) + self.assertIn("timed out", out) + self.assertLess(elapsed, 45, f"timeout not bounded (took {elapsed:.1f}s)") + + +if __name__ == "__main__": + unittest.main()