Add CI/CD pipeline with container-based testing (Feature Request ID 9393 in Mantis)

.github/CI-MAINTAINER.md

@@ -0,0 +1,188 @@
+# CI Maintainer Notes
+
+Operational steps that a `gramps-project/addons-source` maintainer
+needs to handle once PR 820 (and the branch-neutral follow-up) lands.
+The pipeline is otherwise self-driving — this document is only about
+the rough edges.
+
+For day-to-day addon-release maintenance see [MAINTAINERS.md](../MAINTAINERS.md);
+for the contributor-facing summary of what a green CI check means on
+an unreleased branch see [CONTRIBUTING.md](../CONTRIBUTING.md#work-towards-a-merge).
+
+## Contents
+
+1. [One-time setup when the PR first merges](#one-time-setup-when-the-pr-first-merges)
+2. [Creating a new maintenance branch](#creating-a-new-maintenance-branch)
+3. [When a Gramps minor release lands on PyPI](#when-a-gramps-minor-release-lands-on-pypi)
+4. [Image lifecycle](#image-lifecycle)
+5. [Diagnostic log markers](#diagnostic-log-markers)
+6. [Optional future-proofing knobs](#optional-future-proofing-knobs)
+
+## One-time setup when the PR first merges
+
+### 1. Make the `gramps-ci` GHCR package public
+
+`docker-build.yml` pushes images to
+`ghcr.io/gramps-project/addons-source/gramps-ci:<suffix>` using the
+workflow's `GITHUB_TOKEN`. GHCR creates the package as **private** the
+first time. Same-repo CI keeps working (token covers own packages),
+but **fork PRs cannot pull the image** because their `GITHUB_TOKEN`
+has no read access to private packages in `gramps-project`. Fork-PR
+container jobs would fail at "Initialize containers" with an
+authentication error.
+
+Fix once, immediately after the first `Build Docker Images` run
+finishes:
+
+1. Go to <https://github.com/orgs/gramps-project/packages/container/addons-source%2Fgramps-ci/settings>
+2. Under "Danger Zone" → "Change visibility" → set to **Public**
+
+Every existing and future `gramps-ci:<suffix>` tag inherits public
+visibility from this single setting.
+
+### 2. Expect the first-push race on `maintenance/gramps60`
+
+The first push event after merge fires both workflows in parallel:
+
+- `Build Docker Images` builds and pushes `gramps-ci:gramps60`
+  (~5 min cold).
+- `CI` runs `setup` (~2 s), then its container jobs try to pull the
+  image.
+
+Because both start on the same push, the CI container jobs race the
+image push and may fail at "Initialize containers" the first time.
+This race only happens once per branch:
+
+1. Wait for `Build Docker Images` to complete.
+2. Open the failed CI run → click "Re-run failed jobs".
+3. Subsequent pushes find the image already in GHCR — no race.
+
+This is also explained in the header comment of `ci.yml`.
+
+## Creating a new maintenance branch
+
+When a new Gramps minor series goes into development and addons need
+a corresponding branch (e.g. `maintenance/gramps62` once 6.2 opens):
+
+```
+git branch maintenance/gramps62 maintenance/gramps61
+git push origin maintenance/gramps62
+```
+
+No *workflow* edits required — the workflows derive everything from
+`github.ref_name`. The first push fires the same race described
+above — re-run the failed CI jobs once `Build Docker Images`
+finishes.
+
+One file is NOT auto-derived and must be bumped by hand on the new
+branch: **`.github/environment.yml`**'s gramps pin (`gramps>=6.0,<6.1`).
+The conda Windows lane installs gramps from PyPI through that pin, so
+until you bump it to the new series (e.g. `>=6.2,<6.3`) — and the
+series is actually published on PyPI — the Windows lane keeps
+validating addons against the old series. `ci.yml`'s "Report
+gramps-vs-branch series" step prints a loud `::warning::` while the two
+diverge; it does not fail.
+
+The setup job's regex (`gramps[0-9][0-9]`) requires a two-digit
+suffix. When Gramps 10.0 opens this regex needs updating in two
+places (`ci.yml` setup job and `docker-build.yml` params step).
+
+## When a Gramps minor release lands on PyPI
+
+The hybrid Dockerfile auto-detects PyPI availability:
+
+- `pip install "gramps==X.Y.*"` succeeds → image installs the tagged
+  PyPI release (`::notice::` log line).
+- pip reports "No matching distribution found" → image falls back to a
+  SHA-pinned `git clone` of `gramps-project/gramps@maintenance/grampsNN`
+  at the SHA captured by `docker-build.yml`'s params step
+  (`::warning::` log line).
+
+When `gramps==6.1.0` is finally published to PyPI, the next image
+rebuild on `maintenance/gramps61` silently switches from "git tip" to
+"PyPI release" — no maintainer action needed.
+
+To **immediately** rebuild against the new PyPI release without
+waiting for the next push:
+
+1. Open the `Build Docker Images` workflow in the Actions tab.
+2. "Run workflow" → select `maintenance/grampsNN` → Run.
+
+The `::notice::` line in the build log confirms the switch.
+
+## Image lifecycle
+
+`docker-build.yml` owns the `gramps-ci:<suffix>` image. How it rebuilds:
+
+- **On push** to `maintenance/gramps**` — builds and pushes the image
+  for that branch. This is the normal path.
+- **On pull_request** touching `.github/docker/**` or
+  `docker-build.yml` — builds the image **without pushing** (the login
+  step is skipped and `push:` is false), so a Dockerfile change is
+  validated before merge. A fork PR never touches the push credential.
+  A green PR build proves "the image still builds"; it does **not**
+  update the tag CI pulls.
+- **workflow_dispatch** with **`no-cache: true`** — a from-scratch
+  rebuild that ignores the buildx layer cache. Use this to pick up a
+  new base-image (apt security) update or a newly published gramps
+  patch release that the cache key would otherwise skip.
+- **Weekly `schedule`** — the `weekly-rebuild` job fans out a
+  `no-cache` dispatch to every `maintenance/grampsNN` branch.
+  **Caveat:** GitHub runs scheduled workflows only from the repo's
+  **default branch**, so this is inert until these workflows exist
+  there (upstream's default is `maintenance/gramps61`). That is by
+  design — remove/adjust it if the default branch never carries the
+  pipeline.
+
+**Known one-push image lag (not fixed, be aware):** `ci.yml` and
+`docker-build.yml` both fire on the same push, and `ci.yml` pulls the
+moving `gramps-ci:<suffix>` tag while the rebuild is still running. So a
+push that changes the image (e.g. a Dockerfile edit) has its CI run
+against the **previous** image. After the `Build Docker Images` run for
+that push finishes, **re-run the affected CI jobs** to test against the
+new image. For the same reason, the `ruff` version pinned in the
+Dockerfile takes effect one push late.
+
+## Diagnostic log markers
+
+When investigating a CI failure, the install-step output in
+`Build Docker Images` carries these annotations:
+
+| Annotation | Meaning |
+| --- | --- |
+| `::notice::installed gramps==X.Y.* from PyPI` | Released-path install. CI is testing against a tagged release. |
+| `::warning::no gramps==X.Y.* on PyPI; installing from gramps-project/gramps@maintenance/grampsNN at <sha>` | Unreleased-branch fallback. CI is testing against the upstream branch tip at `<sha>`. Visible to contributors via the CONTRIBUTING.md note. |
+| `::error::no gramps==X.Y.* on PyPI and GRAMPS_FALLBACK_SHA is unset` | `git ls-remote` in `docker-build.yml`'s params step returned no SHA for `maintenance/grampsNN` on `gramps-project/gramps`. The matching upstream branch is missing, or the addons-source branch is misnamed. |
+| `::error::pip install gramps failed (non-version reason)` | pip failed for a network/registry reason, not because the version is missing. Captured stderr is dumped after the line. The build does **not** fall back to git in this case — by design, so a transient PyPI hiccup cannot silently flip a released branch into "git tip" mode. |
+
+Other useful entry points:
+
+- `ci.yml` setup job log shows the derived `branch_suffix` and
+  `ci_image`. A failure here means the branch name doesn't match
+  `maintenance/gramps[0-9][0-9]`.
+- `docker-build.yml` params step log shows the captured
+  `fallback_sha`. An empty value means upstream `gramps-project/gramps`
+  has no matching maintenance branch (warning issued; image build will
+  fail iff the fallback is needed).
+
+## Optional future-proofing knobs
+
+Not needed today; record here in case they ever come up.
+
+### Upstream gramps repo URL
+
+The Dockerfile hardcodes `https://github.com/gramps-project/gramps.git`
+as the fallback source. If the project ever reorgs or renames, this
+URL must change in one place (`.github/docker/gramps-ci/Dockerfile`).
+It could be parameterised as a build arg (`ARG GRAMPS_UPSTREAM_REPO`)
+with the current URL as default, but a hardcoded value keeps the
+Dockerfile simpler and a rename is a sufficiently large event that
+editing one string is not the bottleneck.
+
+### GHCR tag retention
+
+`docker-build.yml` pushes both a moving `gramps-ci:<suffix>` tag (e.g.
+`gramps60`) and a per-commit `gramps-ci:<suffix>-<short-sha>` tag.
+The moving tag is always overwritten; the SHA tags accumulate over
+time. Set a retention policy on the GHCR package settings page if
+the count becomes inconvenient — the moving tags are what CI consumes.

.github/docker/gramps-ci/Dockerfile

@@ -0,0 +1,143 @@
+# .github/docker/gramps-ci/Dockerfile
+#
+# Gramps CI image. The Gramps minor series (6.0, 6.1, …) is picked at
+# build time via the GRAMPS_SERIES build arg, so the same Dockerfile
+# produces gramps-ci:gramps60, gramps-ci:gramps61, … without per-branch
+# edits. Includes everything jobs need:
+#   - Python + pip-installed Gramps, PyGObject, pycairo
+#   - GTK typelibs (so addon modules that `from gi.repository import Gtk`
+#     at module load time are importable — widgets still need xvfb to render)
+#   - intltool/gettext/git for make.py builds
+#   - ruff, dbf for lint/test tooling (tests use stdlib unittest per AGENTS.md)
+#   - xvfb + xauth for tests that actually render (wrap with `xvfb-run`)
+#
+# No display server runs by default — the image is headless unless a command
+# explicitly invokes `xvfb-run`. When running with docker locally, pass
+# `--init` (or use a container runtime that injects tini) because xvfb-run
+# hangs if it inherits PID 1.
+#
+# Gramps install path. PyPI is tried first; when no gramps==${SERIES}.*
+# release exists (e.g. while 6.1 is still in development) the build
+# falls back to a SHA-pinned git clone of
+# gramps-project/gramps@maintenance/gramps${SERIES_NODOT}. Other pip
+# failures (network, etc.) are NOT silently retried so a transient
+# PyPI outage cannot flip a normally-released branch to "test against
+# moving tip" mode. docker-build.yml captures the upstream SHA via
+# git ls-remote and passes it in as GRAMPS_FALLBACK_SHA; the SHA
+# becomes part of the buildx cache key so a moved upstream tip
+# actually re-runs the install layer.
+#
+# Local build (released series):
+#   docker build --build-arg GRAMPS_SERIES=6.0 .github/docker/gramps-ci
+#
+# Local build (unreleased series, e.g. 6.1):
+#   sha=$(git ls-remote https://github.com/gramps-project/gramps.git \
+#         refs/heads/maintenance/gramps61 | awk '{print $1}')
+#   docker build --build-arg GRAMPS_SERIES=6.1 \
+#                --build-arg GRAMPS_FALLBACK_SHA=$sha \
+#                .github/docker/gramps-ci
+#
+ARG PYTHON_VERSION=3.12
+FROM python:${PYTHON_VERSION}-slim
+
+# Gramps minor series to install (e.g. 6.0, 6.1). No default — must be
+# passed explicitly so a wrong default cannot silently produce an image
+# for the wrong Gramps series. docker-build.yml derives this from the
+# branch ref.
+ARG GRAMPS_SERIES
+# Commit SHA on gramps-project/gramps@maintenance/grampsNN. Only used
+# when no gramps==${GRAMPS_SERIES}.* release exists on PyPI. Ignored
+# otherwise. Empty default is intentional — on released branches the
+# fallback never fires.
+ARG GRAMPS_FALLBACK_SHA=""
+RUN [ -n "$GRAMPS_SERIES" ] || { echo "GRAMPS_SERIES is required (e.g. 6.0)"; exit 1; }
+
+LABEL org.opencontainers.image.source="https://github.com/gramps-project/addons-source"
+LABEL org.opencontainers.image.description="Gramps ${GRAMPS_SERIES} CI image (Python, Gramps, GTK typelibs, xvfb)"
+
+RUN apt-get update && apt-get install -y --no-install-recommends \
+        libgirepository-2.0-dev \
+        gir1.2-glib-2.0 \
+        gir1.2-gtk-3.0 \
+        gir1.2-pango-1.0 \
+        gir1.2-gdkpixbuf-2.0 \
+        gir1.2-atk-1.0 \
+        gir1.2-gexiv2-0.10 \
+        gcc \
+        g++ \
+        pkg-config \
+        python3-dev \
+        libcairo2-dev \
+        libicu-dev \
+        intltool \
+        gettext \
+        git \
+        xvfb \
+        xauth \
+    && rm -rf /var/lib/apt/lists/*
+
+# Addon runtime deps (dbf, networkx, lxml, svgwrite, boto3, etc.) are
+# NOT baked in here — ci.yml's "Install addon runtime deps (derived from
+# requires_mod)" step pip-installs them at CI runtime from every
+# .gpr.py's requires_mod list, matching what Gramps' Addon Manager does
+# for an end user. Keeps .gpr.py the single source of truth.
+# ruff is PINNED: an unpinned ruff would change the lint verdict on rebuild (a
+# new rule in the E9/F63/F7/F82 selection flips a lane red — or silently stops
+# flagging — with no repo change). Bump deliberately, not by rebuild drift.
+#
+# PyICU is REQUIRED, not optional. Gramps degrades localization without it
+# ("ICU not loaded ... Localization will be impaired"), and worse, gramps 6.0's
+# gramps/plugins/webreport/common.py cannot even be IMPORTED without it: when
+# neither `icu` nor `PyICU` resolves, its outer `except ImportError: pass`
+# leaves `localAlphabeticIndex` unbound and the module-level
+# `AlphabeticIndex = localAlphabeticIndex` raises NameError. Every addon that
+# imports that module (e.g. TimePedigreeHTML) then fails to load, so CI without
+# PyICU reports an addon defect that does not exist for real users. Installing
+# it also makes the image match a normal Gramps environment. (Built from source
+# against libicu-dev above.)
+RUN pip install --no-cache-dir PyGObject pycairo orjson PyICU ruff==0.15.22
+
+# Install gramps: PyPI first, SHA-pinned git clone as fallback.
+RUN /bin/bash -eo pipefail <<'BASH'
+suffix_nodot="${GRAMPS_SERIES//./}"
+if pip install --no-cache-dir "gramps==${GRAMPS_SERIES}.*" 2>/tmp/pip.err; then
+    echo "::notice::installed gramps==${GRAMPS_SERIES}.* from PyPI"
+elif grep -q "No matching distribution found for gramps==" /tmp/pip.err; then
+    if [ -z "${GRAMPS_FALLBACK_SHA}" ]; then
+        echo "::error::no gramps==${GRAMPS_SERIES}.* on PyPI and GRAMPS_FALLBACK_SHA is unset"
+        exit 1
+    fi
+    echo "::warning::no gramps==${GRAMPS_SERIES}.* on PyPI; installing from gramps-project/gramps@maintenance/gramps${suffix_nodot} at ${GRAMPS_FALLBACK_SHA}"
+    mkdir -p /tmp/gramps
+    cd /tmp/gramps
+    git init -q
+    git remote add origin https://github.com/gramps-project/gramps.git
+    git fetch --depth 1 origin "${GRAMPS_FALLBACK_SHA}"
+    git checkout FETCH_HEAD
+    pip install --no-cache-dir .
+    cd /
+    rm -rf /tmp/gramps
+else
+    echo "::error::pip install gramps failed (non-version reason); aborting rather than silently switching to git tip"
+    cat /tmp/pip.err
+    exit 1
+fi
+BASH
+
+# NOTE: the build toolchain (gcc, python3-dev, pkg-config) is deliberately KEPT
+# in the image — it is not purged here. Addon requires_mod are pip-installed at CI
+# runtime (ci.yml "Install addon runtime deps"), and the source-built ones with no
+# wheel on the apt lane (pygraphviz, psycopg2) compile a C extension during that
+# step. Purging gcc/pkg-config left those builds with no compiler, so they failed
+# and were swallowed by the install step's "|| echo … (continuing)" — silently
+# degrading the affected addon's coverage while the job stayed green. The per-
+# package system headers those builds link against (libgraphviz-dev, libpq-dev on
+# apt) are derived from the addons' .gpr.py and installed at CI runtime by the same
+# single-source map as requires_gi/requires_exe — see .github/scripts/
+# addon_system_deps.py (MOD_BUILD_PACKAGES) and ci.yml's "Install addon system
+# deps" step — so .gpr.py stays the single source of truth.
+
+RUN python -c "from gramps.gen.const import VERSION; print('Gramps', VERSION)" \
+ && python -c "import gi; gi.require_version('Gtk', '3.0'); from gi.repository import Gtk; print('GTK OK')"
+
+WORKDIR /workspace

.github/environment.yml

@@ -0,0 +1,23 @@
+name: addons-ci
+channels:
+  - conda-forge
+dependencies:
+  - python=3.12
+  - pygobject
+  - gtk3
+  - pip
+  # Addon runtime deps (dbf, networkx, lxml, svgwrite, boto3, etc.) are
+  # installed at CI runtime by ci.yml's auto-derive step from .gpr.py
+  # requires_mod — single source of truth. Keep only the stable base
+  # here (Gramps + orjson for plugin registration).
+  #
+  # NOTE: gramps is installed from PyPI via the `pip:` block below (conda-forge
+  # is NOT the source), and the pin is a deliberate per-series bound. `<6.1`
+  # can never resolve 6.1, so on a maintenance/gramps61 (or later) branch this
+  # file MUST be bumped (e.g. to ">=6.1,<6.2") — the workflows derive the series
+  # automatically, but this pin does not and will otherwise validate addons
+  # against 6.0.x forever (see .github/CI-MAINTAINER.md and ci.yml's "Report
+  # gramps-vs-branch series" step, which warns loudly when the two diverge).
+  - pip:
+      - "gramps>=6.0,<6.1"
+      - orjson