SEP-1864: Remediate HIGH/CRITICAL CVEs in the side-car image

[yyyyyyyan]

Summary

Takes the published SEP side-car image from 101 HIGH/CRITICAL Trivy findings to 17, and dispositions each of the remaining 17 as a documented, purl-scoped exception. Three files change; no application code is touched.

sidecar/Containerfile.sidecar — three edits to the final stage

Edit	Effect
Base pin python:3.11.14-slim → python:3.11.16-slim	Debian 13.3 → 13.6, bringing fixed openssl (3.5.6-1~deb13u2), libcap2, and wheel 0.46.3
Delete the first apt-get block entirely	Removes g++ (which pulled 43 transitive packages including linux-libc-dev) and the WeasyPrint stack (libpango-1.0-0, libpangoft2-1.0-0, fontconfig, fonts-dejavu, shared-mime-info, which brought libglib2.0-0t64 and libxml2). netcat-openbsd is installed separately further down and stays
Add apt-get upgrade -y to the surviving layer	The pin bump alone leaves util-linux 2.41-5; Debian published 2.41.5-0+deb13u1 after the tag was cut. update → upgrade → install → clean stay in one RUN so the upgrade never runs against a stale index

RUN pip install --no-cache-dir wheel is also dropped. It was a no-op: python:3.11.14-slim already shipped wheel 0.45.1, so the line resolved as already-satisfied and installed nothing. The bumped base ships 0.46.3, past the 0.46.2 fix, so the base bump is what clears that finding.

.trivyignore.yaml (new, repository root) — 12 entries covering the 17 residual findings, each carrying id, purls, statement and expired_at: 2027-02-19. Every statement gives a reachability argument plus the condition under which the entry should be removed, not merely "no upstream fix".

changelog.d/SEP-1864.security.md (new) — security fragment.

Two things a reviewer should know

1. The exception file is inert unless the scan passes --ignorefile. Trivy's DefaultIgnoreFile is .trivyignore, so a bare trivy image never loads the YAML form — it parses the full schema only for a path handed to it explicitly. PMM's own scan (percona/percona-docker/.github/workflows/trivy_scan_pmm.yml) passes no such flag. The requirement is stated in the file's own header comment so the follow-up CI gate inherits it rather than rediscovering it.

2. purls on every entry is load-bearing, not decoration. In Trivy's pkg/result/ignore.go, matchPURL() returns true when the list is empty, so an unscoped entry suppresses its CVE for every package — including one where the reachability argument does not hold. Every justification here is package-specific ("nothing executes perl", "SEP serves no QUIC", "the archive paths link zlib rather than invoking the gzip binary"), so each entry is scoped to exactly the packages it assessed. The constraints carry no version, so a Debian point release does not silently un-except a finding.

setuptools: why no upgrade line

An earlier iteration added RUN pip install --no-cache-dir --upgrade "setuptools>=81,<82" to clear the jaraco.context 5.3.0 finding while preserving pkg_resources. It was measured to be inert and removed:

• requirements.txt pins setuptools==83.0.0. The builder stage bakes setuptools-83.0.0-py3-none-any.whl into /wheels, and the next recipe line — RUN pip install --no-cache-dir /wheels/*, untouched here — reinstalls it. Any bound placed above that line is overwritten.
• pkg_resources is consequently already absent from the pre-change image, through that same unchanged line. Verified directly: installing setuptools 83.0.0 on the old base python:3.11.14-slim removes it. Not a regression introduced here.
• The security goal is met regardless: setuptools 83.0.0 vendors jaraco_context-6.1.0, and the built image contains no stale 5.3.0 copy anywhere under /usr/local/lib/python3.11.

Restoring a bound would mean downgrading against the lockfile. Constraining setuptools repo-wide is a separate decision.

Verification performed

Live evidence against the rebuilt image. Trivy pinned to v0.69.3, the version PMM installs, run as docker.io/aquasec/trivy:0.69.3 against a podman save --format docker-archive export of sep:HEAD.

Check	Result
Image builds with no compiler present	make builder and RELEASE_VER=HEAD make image both exit 0
g++ and the WeasyPrint stack gone	g++ absent; linux-libc-dev, libglib2.0-0t64 and all five rendering libs not-installed; libxml2 reports unknown ok not-installed
Base layer refreshed and upgraded	/etc/debian_version 13.6; util-linux 2.41.5-0+deb13u1; openssl 3.5.6-1~deb13u2; libcap2 1:2.75-10+deb13u1+b1
Python packaging state	setuptools 83.0.0, wheel 0.46.3; only jaraco_context-6.1.0.dist-info present
Container healthy	healthcheck.sh printed healthy, exit 0. All five non-one-shot supervisord programs RUNNING; all three migrations complete with their sentinels
App set unchanged	verified restricted: 5 app packages (atw, framework, inventory, mysql_backups, shared), exit 0
Bare rescan	17 HIGH/CRITICAL, all in the Debian layer; 0 python-package findings
Gated rescan	--ignorefile .trivyignore.yaml --exit-code 1 → 0 findings, exit 0
Reconciliation	17 scan (cve, pkg) pairs vs 17 exception pairs — empty set difference in both directions
Every entry scoped	12/12 carry purls, id, statement, expired_at

The two rescans are deliberately separate runs. A single run with --ignorefile cannot distinguish "correctly suppressed" from "wrote a file Trivy ignored".

The rescan matched the projected residual set exactly — 4 CRITICAL and 13 HIGH across 12 distinct CVEs, no finding added or dropped — so the ticket's residual table needed no correction.

Residual set (all excepted)

Package(s)	CVE	Sev	Why unreachable
perl-base	CVE-2026-13221, CVE-2026-42496, CVE-2026-57433, CVE-2026-8376	CRITICAL ×4	Nothing in the image executes perl
perl-base	CVE-2026-42497, CVE-2026-48962, CVE-2026-57432, CVE-2026-9538	HIGH ×4	Same
libssl3t64, openssl, openssl-provider-legacy	CVE-2026-14456	HIGH ×3	QUIC server DoS; the three API processes speak HTTP/1.1 over loopback behind PMM's nginx
libncursesw6, libtinfo6, ncurses-base, ncurses-bin	CVE-2025-69720	HIGH ×4	Local vector needing user interaction with a crafted terminfo entry inside the container
gzip	CVE-2026-41992	HIGH	In-container archive paths use Python's gzip/tarfile stdlib, which link zlib. All four /usr/bin/tar -czf call sites run on a managed target host, not in this image
libacl1	CVE-2026-54369	HIGH	Local privesc needing pre-existing privileges; the image runs as the unprivileged sep user

Four CRITICALs ship as documented exceptions, so the Percona CVE Process § 7 sign-off (engineering leadership, Product, and security) is requested on the ticket and cross-posted to PMM-15297.

Backwards compatibility

No public contract changes — no API response models, DB schemas, enums, config keys, error responses, seed data, or task payloads.

One behavioural change, confined to a build mode that ships nothing: an unrestricted (SEP_RESTRICT_APPS=0) build of this recipe would ship the report app without its system libraries, and app/sep/apps/report/service.py:37 imports weasyprint at module scope, so it would fail at load. This is not a regression to any shipped artifact — the app-restricted image is the only published one, and both make image and .github/workflows/ci.yml pass SEP_RESTRICT_APPS=1. The comment added beside the surviving apt-get layer names the five libraries and the condition under which they must return, which is what makes a future unrestricted build recoverable rather than a puzzle. weasyprint stays in pyproject.toml and in the wheel set; it contributes no findings, only its system libraries did.

Known limitations

• Nothing in this repository consumes .trivyignore.yaml yet. No tracked workflow or pipeline invokes Trivy at all, so until a scan gate ships, this file is validated only by the gated rescan above, on the implementer's machine — a later edit could merge with nothing parsing it. Adding a .trivyignore.yaml row to .github/workflows/ci.yml's changes filter would not close this: the job that row triggers is build, which does static image inspection and runs no scanner. The recurring gate needs its own ticket — a workflow, a Trivy version pin, a failure threshold, the path filters, and the --ignorefile .trivyignore.yaml flag PMM's own scan does not carry — plus a revisit before expired_at: 2027-02-19 starts failing it.

Out of scope

Removing weasyprint from the Python dependency set; rebuilding or republishing released image tags; force-removing Debian-Essential packages (measured at 12 fewer findings and four fewer CRITICALs, and rejected on certainty grounds — --force-remove-essential exists precisely because dpkg cannot verify the outcome, and debconf is itself a perl program, so any later apt/dpkg operation in the image would break).

Tested

The side-car does not run standalone: SECRET_KEY has no default, and the three migration one-shots block on nc -z $SEP_DB_HOST $SEP_DB_PORT, whose host defaults to pmm-server. The scenarios below therefore need one PostgreSQL container plus the image on a shared network.

• Build both stages from a clean cache: make builder, then RELEASE_VER=HEAD make image. Confirm both exit 0 — the runtime stage now has no compiler, so a wheel needing one at install time would fail here.
• Inspect the rebuilt image and confirm g++ is absent, that linux-libc-dev, libglib2.0-0t64 and the five WeasyPrint libraries report not-installed, and that /etc/debian_version is 13.6 with util-linux 2.41.5-0+deb13u1.
• Start postgres:16-alpine and sep:HEAD on a shared container network with SECRET_KEY, SEP_DB_HOST and SEP_DB_PASSWORD set, then exec the image's bundled healthcheck (sidecar/healthcheck.sh, installed at $APP_HOME/healthcheck.sh) inside the running container and confirm it prints healthy and exits 0. This asserts every non-one-shot supervisord program RUNNING, all three migrations complete, all three /health endpoints answering, and the broker responding to PING.
• Confirm the shipped app set is unchanged: CONTAINER_RUNTIME=podman sidecar/verify_image_apps.sh sep:HEAD restricted.
• Scan the rebuilt image with Trivy v0.69.3 at --severity HIGH,CRITICAL with no ignorefile, and confirm 17 findings, all in the Debian layer, with zero python-package findings.
• Re-scan the same image adding --ignorefile .trivyignore.yaml --exit-code 1 and confirm it exits 0. Running both is what distinguishes a correctly suppressed set from a file Trivy never read.

Checklist

• New/modified functions have type hints and rST docstrings (N/A for this change — no Python touched)
• New tests added for new features or bug fixes (N/A for this change — a container-build recipe plus two non-Python files; verification is live evidence against the rebuilt image)
• Database migrations generated if models changed (N/A for this change — no models changed)
• User-facing changes documented (README, inline help, UI text) (N/A for this change)
• Configuration changes documented with examples (N/A for this change)
• Changelog fragment added under changelog.d/

[Copilot]

Pull request overview

Updates the side-car image to remediate container vulnerabilities and document residual exceptions. The gzip exception’s call-site inventory is incomplete and needs correction.

Changes:

• Refreshes the Python/Debian base and upgrades system packages.
• Removes unused compiler and PDF-rendering dependencies.
• Adds scoped Trivy exceptions and a security changelog fragment.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
sidecar/Containerfile.sidecar	Hardens and reduces the runtime image.
.trivyignore.yaml	Documents residual vulnerability exceptions.
changelog.d/SEP-1864.security.md	Records the security remediation.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[yyyyyyyan]

Tracked the follow-up from the PR description as SEP-1869: Nothing in this repository consumes .trivyignore.yaml yet.