Skip to content

build(docker): Switch to Docker Hardened Images (DHI)#216

Merged
oioki merged 15 commits into
masterfrom
build/switch-to-dhi-base-images
Mar 4, 2026
Merged

build(docker): Switch to Docker Hardened Images (DHI)#216
oioki merged 15 commits into
masterfrom
build/switch-to-dhi-base-images

Conversation

@oioki
Copy link
Copy Markdown
Member

@oioki oioki commented Mar 3, 2026

Attempt 3 of #212

Previous attempts failed due to:

  • failing acceptance test (fixed in healthcheck)
  • missing fonts (caused inc-2061)

Switch both builder and runtime stages to Docker Hardened Images (DHI).

Before: builder and runtime both used node:24.14.0/node:24.14.0-slim (Debian 12), with canvas system libraries installed via apt-get in the runtime. Trivy reported 1,457 vulnerabilities (3 CRITICAL, 148 HIGH) across 288 OS packages, compressed image size ~502 MB.

After: builder uses us-docker.pkg.dev/sentryio/dhi/node:24-debian13-dev, runtime uses the minimal us-docker.pkg.dev/sentryio/dhi/node:24-debian13 (Debian 13, distroless — no shell, no apt). Trivy reports 10 vulnerabilities (0 CRITICAL, 1 HIGH) across 12 OS packages, compressed image size ~83 MB.

  ┌────────────────┬────────────────────────────────────────┬─────────────────────────────┬────────┐
  │     Metric     │       Before (node:24.14.0-slim)       │         After (DHI          │ Change │
  │                │                                        │      node:24-debian13)      │        │
  ├────────────────┼────────────────────────────────────────┼─────────────────────────────┼────────┤
  │ Compressed     │ ~502 MB                                │ ~83 MB                      │ −83%   │
  │ size           │                                        │                             │        │
  ├────────────────┼────────────────────────────────────────┼─────────────────────────────┼────────┤
  │ OS             │ Debian 12.13 (bookworm)                │ Debian 13.3 (trixie)        │        │
  ├────────────────┼────────────────────────────────────────┼─────────────────────────────┼────────┤
  │ OS packages    │ 288                                    │ 12                          │ −96%   │
  ├────────────────┼────────────────────────────────────────┼─────────────────────────────┼────────┤
  │ OS vulns       │ 1455 (C:3 H:148 M:659 L:644 U:1)       │ 10 (C:0 H:1 M:2 L:7)        │ −99%   │
  ├────────────────┼────────────────────────────────────────┼─────────────────────────────┼────────┤
  │ npm vulns      │ 2 HIGH (minimatch                      │ 0                           │ −100%  │
  │                │ CVE-2026-27903/27904)                  │                             │        │
  ├────────────────┼────────────────────────────────────────┼─────────────────────────────┼────────┤
  │ Total vulns    │ 1457                                   │ 10                          │ −99%   │
  └────────────────┴────────────────────────────────────────┴─────────────────────────────┴────────┘

How the builder conflict was resolved

DHI's pre-installed libexpat1 2.7.4 (arch:all) conflicts with canvas's build dependencies (libcairo2-dev → libfontconfig-dev → libexpat1-dev requires exactly libexpat1 = 2.7.1-2). This turned out not to matter: canvas 3.x uses node-pre-gyp to download pre-built binaries at install time, so no system canvas libraries are needed in the builder at all.

How the minimal runtime was made to work

canvas 3.x bundles all its graphics libs (libcairo, libpango, librsvg, etc.) in node_modules/canvas/build/Release/, so no system canvas libraries are needed at runtime either. However, the bundled libs still need some basic system libs absent from the distroless image. Rather than maintaining a hand-picked list that could silently go stale when canvas or another native module is updated, the builder uses ldd to auto-detect all transitive system dependencies of every .node file and collects them into /canvas-sys-libs/. These are then copied into the runtime with COPY --from=builder /canvas-sys-libs/usr/lib/ /usr/lib/.

canvas also bundles libfontconfig.so.1 but needs /etc/fonts/fonts.conf to initialise — without it all chart text renders as box glyphs. fontconfig is installed in the builder (which also pulls in fonts-dejavu-core as a dependency), and /etc/fonts/ and /usr/share/fonts/ are copied into the runtime as plain files. They do not appear as installed packages in the runtime image, hence 12 OS packages rather than the 24 in the previous build.

Build-time smoke test

The runtime stage runs smoketest.js as a RUN step (exec form, since the minimal image has no shell). It exercises the full rendering pipeline — canvas init, font loading, echarts render, PNG output — and compares the result byte-for-byte against a pinned SHA-256 hash of the known-good output.

This catches two classes of failure at build time rather than in production:

  • Hard failures (crash/exception): missing .so file, fontconfig not initialised, etc.
  • Silent corruption: a technically valid PNG that renders box glyphs or wrong pixels — the kind that would pass a magic-bytes check but was exactly the failure mode that caused inc-2061

The render is deterministic: identical input spec → identical bytes across repeated runs and separate container invocations (verified). When intentionally changing the chart spec, the file includes instructions to regenerate the expected hash.

oioki and others added 14 commits February 27, 2026 13:39
Replace node:24.14.0 and node:24.14.0-slim base images with Sentry's
DHI equivalents (us-docker.pkg.dev/sentryio/dhi/node:24-debian13-dev
and node:24-debian13).

Move canvas native compilation (yarn install + build deps) entirely into
the builder stage so the runtime image no longer needs build-essential
or -dev headers. The runtime stage only installs the minimal shared
libraries that canvas needs at runtime (libcairo2, libpango-1.0-0,
libjpeg62-turbo, libgif7, librsvg2-2), eliminating ~1,400 Trivy findings
that were rooted in the Debian system package footprint.

Co-Authored-By: Claude <noreply@anthropic.com>
The DHI manifest entries for amd64 have an unexpected variant field
("v8"), which is normally an ARM designator. Docker BuildKit fails to
match linux/amd64 against these entries.

Add --platform=linux/amd64 to both FROM lines so BuildKit selects by
architecture directly rather than relying on variant matching.

Co-Authored-By: Claude <noreply@anthropic.com>
The DHI manifest non-standardly labels the amd64 image with variant
"v8" (an ARM designator), making its platform string linux/amd64/v8
rather than the expected linux/amd64. Cloud Build's BuildKit fails to
match linux/amd64 against linux/amd64/v8.

Specify the exact platform string from the manifest so BuildKit resolves
the correct image layer.

Co-Authored-By: Claude <noreply@anthropic.com>
The DHI dev image has libexpat1=2.7.4 (arch:all) pre-installed as a
security patch, which conflicts with the entire canvas build dep chain
(libcairo2-dev → libfontconfig-dev → libexpat1-dev requires libexpat1
= 2.7.1-2 arch-specific). Both APT solver 3.0 and the classic solver
fail to resolve this.

Use standard node:24.14.0 for the builder stage (compiles canvas native
module and TypeScript without package conflicts). The runtime stage uses
DHI node:24-debian13-dev since the fully minimal node:24-debian13 image
has no shell or package manager — canvas needs runtime shared libraries
(libcairo2, libpango-1.0-0, libjpeg62-turbo, libgif7, librsvg2-2) which
apt-get install correctly on Debian 13 in the -dev image.

The --platform=linux/amd64/v8 flag is scoped to only the DHI FROM line
because the DHI manifest incorrectly labels the amd64 entry with variant
"v8" (an ARM designator). Standard node:24.14.0 uses linux/amd64 without
a variant, so the platform flag must not apply to that stage.

Compiled node_modules (including canvas.node native binary) is copied
from the builder so the runtime stage never needs build tools.

Co-Authored-By: Claude <noreply@anthropic.com>
canvas 3.x bundles its own copies of libcairo, libpango, libjpeg,
libgif, librsvg, harfbuzz, glib, etc. inside
node_modules/canvas/build/Release/. No system-level canvas libraries
need to be installed in the runtime image.

The -dev variant is still required over the minimal node:24-debian13
image because canvas's bundled dependencies (librsvg, glib) need base
system libs (libz, libexpat, libuuid, liblzma) that are present in the
dev image but absent from the stripped-down minimal image.

Co-Authored-By: Claude <noreply@anthropic.com>
canvas 3.x downloads pre-built binaries via node-pre-gyp, so the
build stage no longer needs system canvas libraries (libcairo2-dev,
libpango1.0-dev, etc.). This removes the conflict with DHI's
pre-installed libexpat1 2.7.4 package that previously blocked using
the hardened image in the builder stage.

Also removes the --platform=linux/amd64/v8 workaround now that the
DHI image manifests have correct platform variant labels.

Co-Authored-By: Claude <noreply@anthropic.com>
Switch the runtime stage from node:24-debian13-dev to node:24-debian13
(the minimal/distroless image) to reduce the attack surface.

canvas 3.x bundles its graphics libs (libcairo, libpango, etc.) so no
system canvas libraries are needed. However, canvas's bundled librsvg
and glib still require four basic system libs absent from the minimal
image (libz, libexpat, libuuid, liblzma). These are collected in the
builder stage and copied into the runtime, rather than pulling in the
full -dev image.

The smoke-test RUN uses exec form since the minimal image has no shell.

Co-Authored-By: Claude <noreply@anthropic.com>
Add a separate deps stage that runs yarn install --production so that
jest, typescript, eslint, ts-jest, supertest, and their transitive
deps are not copied into the runtime image. The builder stage keeps
the full install for TypeScript compilation.

node_modules layer in the runtime image shrinks from ~94 MB to ~46 MB.

Co-Authored-By: Claude <noreply@anthropic.com>
Merge the separate deps and builder stages into a single builder stage.
Install all dependencies, compile TypeScript, then prune to
production-only deps in place before copying into the runtime image.
Runtime image contents remain identical.

Co-Authored-By: Claude <noreply@anthropic.com>
The DHI minimal runtime image has no fontconfig config or font files.
canvas bundles libfontconfig.so.1 but without /etc/fonts/fonts.conf it
cannot initialise, causing all chart text to render as box glyphs.

Install fontconfig in the builder stage (which also pulls in
fonts-dejavu-core as a dependency), then copy /etc/fonts/ and
/usr/share/fonts/ into the runtime image. No extra packages are
added to the runtime image itself.

Co-Authored-By: Claude <noreply@anthropic.com>
Replace the hand-picked list of system libraries (libz, libexpat,
libuuid, liblzma) with ldd-based auto-detection. Scanning all .node
files ensures any new transitive system dependency added by a future
canvas or other native module upgrade is automatically included,
rather than silently breaking the distroless runtime image.

Also replace the `--help` build-time check with a full render smoke
test (smoketest.js) that exercises the canvas init, font loading,
and echarts render pipeline end-to-end, catching missing .so files
or broken fonts at build time rather than at runtime.

Co-Authored-By: Claude <noreply@anthropic.com>
Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Comment thread smoketest.js Outdated
Replace the PNG magic-bytes check with a SHA-256 hash comparison
against the known-good render output. This catches subtle regressions
(corrupted pixels, font changes, echarts behaviour changes) that still
produce a technically valid PNG. Instructions to update the hash when
intentionally changing the chart spec are included in the file.

Co-Authored-By: Claude <noreply@anthropic.com>
@oioki oioki merged commit 3acff56 into master Mar 4, 2026
12 checks passed
@oioki oioki deleted the build/switch-to-dhi-base-images branch March 4, 2026 20:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants