fix(docker): install harper from the packed tarball's own shrinkwrap

[kriszyp]

Offered for @heskew since #1960 is assigned to them — Kris asked for this to be dispatched anyway. Please take/edit/close as you see fit.

What

The Docker image installs Harper from a locally-built tarball with npm install --ignore-scripts --global harper-*.tgz. npm only honors a package's bundled npm-shrinkwrap.json when the registry packument says the package has one (_hasShrinkwrap) — metadata set at publish time. A local tarball has no packument, so npm never learns the shrinkwrap exists and resolves the whole dependency tree fresh against package.json's ranges instead of the pinned tree. Registry installs (npm install harper) are unaffected; only the image build was.

This extracts the tarball into a normal npm project directory and runs npm install --omit=dev --ignore-scripts there instead, so npm reads npm-shrinkwrap.json straight off disk the same way it would for any checked-out project. Since that's a local (non-global) install, it doesn't create the usual global bin symlink, so the change replicates npm's own global-install layout by hand: extract into $NPM_CONFIG_PREFIX/lib/node_modules/harper, then symlink dist/bin/harper.js into $NPM_CONFIG_PREFIX/bin/harper. docker-entrypoint.sh's which harper / bun "$(which harper)" paths only depend on PATH, so this is transparent to it. Also added npm cache clean --force after install — neither Dockerfile variant cleaned the npm download cache before, and left uncleaned it made the fixed image bigger than the current one (see Verification).

Also strips devDependencies from the Dockerfile's own extracted copy of package.json before installing — not the published tarball, registry consumers never see this. Pre-push review (round 4) found that npm install --omit=dev still resolves dev edges to compute the ideal tree even though it won't install them, which turned out to be a real, not hypothetical, correctness gap: a production package that a devDependency also happens to want (verified with harper's own tree — mqtt is a devDependency whose ws/mqtt-packet ranges also cover the pinned production copies of those same packages) could silently resolve above its shrinkwrap pin the next time either range moves — the exact class of bug this whole PR exists to close, just reachable through a side door. It also forced an unnecessary network fetch of every dev package's manifest during the image build, including one (uWebSockets.js) pinned to a raw GitHub tarball URL that never ends up installed. Verified both are fixed (see Verification).

Also extends .github/workflows/docker-smoke.yml, which is the CI guard for exactly this class of change, with:

• a which harper check,
• a second boot pass under HARPER_RUNTIME=bun (runs even if the default-runtime leg fails, so one doesn't mask the other),
• a step that actually proves the shrinkwrap took effect — build-tools/check-shrinkwrap-pins.mjs compares two installed dependency versions against the shrinkwrap's pins inside the built image. The comparison is against npm-shrinkwrap.packed.json, a copy the Dockerfile freezes before npm install runs — not the live npm-shrinkwrap.json, because npm rewrites that file in place to match whatever it actually installs, which would make a post-install comparison vacuous (confirmed this by testing it — round 1 of pre-push review shipped exactly that bug, round 2 caught and fixed it; see Verification below).
• curl --max-time on both boot-check loops and a test -f npm-shrinkwrap.json fail-fast guard, so a hang or a packaging regression fails loud instead of silent.

The existing smoke test only exercised the default runtime and never actually checked that a version was pinned.

Also fixes .dockerignore's /harperfast-harper-*.tgz pattern (stale from a scoped-name era; this package is harper, so it never matched anything) and updates DESIGN.md's packaging note, which still described the pre-fix behavior.

Why now

rocksdb-js 2.6.0/2.6.1 (published 2026-07-31) redefined the compression store option in a way that throws on the value Harper passes, breaking every table open. harper main pins ^2.5.0, which is satisfied by 2.6.1. The lockfile/shrinkwrap pin the working 2.5.0, so CI and registry installs are protected — but the image's fresh-resolve-at-build-time install would have shipped 2.6.1: a green build producing a completely broken container. See #2036 / #2037 for that thread. This PR doesn't touch that fix directly — it closes the reproducibility gap that turned a caret-range regression into a shipped image regression. I reproduced this concretely (see Verification): the current Dockerfile resolves rocksdb-js to 2.6.1 today; this PR's Dockerfile resolves it to the pinned 2.5.0.

Open questions from #1960

1. Extract-and-install vs. accepting an unpinned image. Went with extract-and-install. Verified locally with a minimal test package that npm install --global <tarball> re-resolves fresh (reproducing the bug), and that npm install --global <local-directory> is actually a link install — it symlinks the package in without installing any of its dependencies at all, which is worse. Extracting and running a plain npm install inside the extracted directory was the only one of the three that honored the shrinkwrap for version pinning.
2. npm ci viability (stripping devDependencies from the packed package.json). Not doing this for the react-native exclusion — see "Risks & open questions" below and Docker image still ships the react-native-fs subtree unpinned (residual from #1960/#2042) #2043, the follow-up issue. (This PR does strip devDependencies from the Docker image's own ephemeral extracted copy, for an unrelated and unconditionally-safe reason — see "What" above.)
3. Bin/symlink placement. Manual symlink into $NPM_CONFIG_PREFIX/bin, matching npm's own global-install convention — see "What" above.

Verification

Built both the current main Dockerfile ("before") and this PR's Dockerfile ("after") locally (no registry push) and compared:

	before (current main)	after (this PR)
@harperfast/rocksdb-js resolved	2.6.1 (latest, satisfies ^2.5.0, breaks table opens)	2.5.0 (shrinkwrap pin)
fastify resolved	5.11.0 (latest)	5.10.0 (shrinkwrap pin)
image size	2182.8 MB	1869.2 MB (−313.6 MB, −14.4%)
which harper	resolves	resolves, same target convention
boots (default runtime)	—	HTTP 200 from the Operations API
boots (HARPER_RUNTIME=bun)	—	HTTP 200 from the Operations API

docker isn't available as a CLI in the sandbox this ran in, but the daemon socket was reachable; I downloaded static docker/buildx binaries (no root) to drive real builds against it rather than asserting the result.

Independent review: codex + gemini + grok (experimental) + harper-domain, run 7 times as I fixed what each round found (round 5 was a full re-review after a major behavioral fix; the rest were delta rounds against the growing diff). Two rounds caught defects in my own review-driven fixes before they shipped: round 2 caught that my round-1 CI fix compared npm's post-install output to itself (vacuous); round 6 caught that my round-5 self-verifying canary compared against the wrong reference value (absolute latest instead of range-max). Both fixed and reverified against a real build before the next round.

Risks & open questions

• The react-native tree is not excluded by this fix, contrary to what Docker image dependency tree is resolved fresh at build time, not from the published shrinkwrap #1960 assumed — tracked as a follow-up, Docker image still ships the react-native-fs subtree unpinned (residual from #1960/#2042) #2043, per Kris's call. fix(build): strip the react-native tree from the published shrinkwrap #1959 prunes npm-shrinkwrap.json of the alasql → react-native-fs → react-native subtree (~200 packages, react-native/metro/hermes/etc.), but the packed package.json is untouched — alasql's own manifest inside the tarball still declares react-native-fs as an optionalDependency. Plain npm install (unlike a registry install of harper as someone else's dependency, where the shrinkwrap is fully authoritative and package.json isn't consulted) reconciles the local project's lockfile against its own package.json, so it silently re-adds the pruned subtree to satisfy that edge. I confirmed this three ways:
  • npm ci (after also stripping devDependencies to pass its root sync check) refuses outright: Missing: react-native-fs@2.20.0 from lock file plus ~30 more entries — i.e. ci is strict enough to not re-add it, but strict enough that it won't proceed either.
  • npm install --omit=optional (tree-wide) does drop the react-native subtree, but it also drops @harperfast/rocksdb-js's per-platform native-binary optionalDependencies — confirmed the linux-x64-glibc binding package is simply absent — which would silently break every table open. Rejected.
  • Only a real fix survives: npm ci against a package.json where alasql's own packed manifest also has the optional edge removed. That's a materially bigger change to the published tarball (patching a third-party dependency's manifest inside our own package) than the devDependencies-strip Docker image dependency tree is resolved fresh at build time, not from the published shrinkwrap #1960 anticipated, so I raised it with Kris instead of deciding unilaterally — the call was to ship this PR now and track the fuller fix separately (Docker image still ships the react-native-fs subtree unpinned (residual from #1960/#2042) #2043), rather than fold a published-artifact change into an already-large PR.
  • Net effect: this PR still fixes the urgent, concrete problem (unpinned versions — the rocksdb-js scenario above) and still nets a real size decrease, but the size drop is smaller than Docker image dependency tree is resolved fresh at build time, not from the published shrinkwrap #1960's speculative 444M→268M, because that number assumed the react-native tree would also be excluded.
  • Residual supply-chain note from review: that re-added subtree now has no pinned version and no lockfile integrity hash — the one part of the image this PR doesn't make reproducible. It sits behind an isReactNative guard unreachable under Node either way, but it's unverified code in a shipped image.
  • Sharper than it first looked: this isn't confined to dead react-native code. @endo/static-module-record — a real production dependency used by the SES sandbox (security/jsLoader.ts) — needs @babel/parser/@babel/traverse/@babel/types, the same packages the react-native chain re-adds unpinned. When npm reconciles the two, the shared copy can drift above the shrinkwrap's pin. Confirmed the mechanism, not confirmed it's live today (npm would normally keep the locked entry since ranges are satisfied) — but it means the residual gap isn't just image bloat, it's version drift risk in a parser the sandbox runs on application code. Written up in full in Docker image still ships the react-native-fs subtree unpinned (residual from #1960/#2042) #2043.
• This PR does not change what registry consumers receive — npm install harper from the registry is untouched. Docker image still ships the react-native-fs subtree unpinned (residual from #1960/#2042) #2043's fuller fix would.
• build-tools/check-shrinkwrap-pins.mjs checks two named dependencies, not the whole tree — deliberately, after testing the alternative, and it now verifies its own canaries still work. A reviewer correctly pointed out a 2-canary check can decay: it only proves anything while those two pins lag the registry. I tried the more thorough fix (compare every dependency in the packed shrinkwrap) and reverted it after testing against a real build: it's permanently red today, because the still-open react-native gap drags shared production packages to newer versions than the shrinkwrap pins for them — and this isn't harmless: @endo/static-module-record (a real production dependency used by the SES sandbox, security/jsLoader.ts) needs @babel/parser/@babel/traverse/@babel/types, the same packages the react-native chain re-adds unpinned, so this is a second, independent, security-relevant reason to close that gap, not just noise in a check. The check now also verifies its own discrimination: it calls npm view <dep>@<range> version (the max version satisfying the dependency's declared range — not the bare latest dist-tag, which could be a newer major the range excludes and a broken install could never reach either) and fails loudly if both canaries have caught up to what their range allows, instead of silently becoming a no-op. Verified both directions against a real build, including rigging both canaries to their range-max to confirm the guard fires.
• publish-docker.yaml (the workflow that builds and pushes the image consumers actually pull) doesn't run any of these checks. All the guards added here — which harper, the shrinkwrap-pin comparison, both boot legs — live only in docker-smoke.yml, which never runs on a release tag. A reviewer flagged this; I'm not folding a change to the release pipeline into an already-large PR without it getting its own attention, but it means a packaging regression that docker-smoke.yml's path filters miss (e.g. a build-tools/build.sh change not covered by this PR's widened filters) would only be caught by the test -f npm-shrinkwrap.json fail-closed guard at release-build time, after the npm package has already published. Worth a follow-up — reusing docker-smoke.yml as a workflow_call job in the publish pipeline, or running check-shrinkwrap-pins.mjs against the built image before the push step, would close it.
• Pinning the image changes the incident-remediation path (now documented in DESIGN.md): before this PR, rebuilding the image picked up any newer in-range dependency automatically, which is how the rocksdb-js 2.4.0→2.5.0 fix could ship as "just rebuild." After this PR, that path needs a lock bump + re-release; a rebuild alone reproduces the same pinned tree, bug included.

Test coverage

• docker-smoke.yml's existing "container boots" step now also runs under HARPER_RUNTIME=bun (scoped to only run if the image actually built), and there are new steps asserting which harper resolves and that two representative dependencies match their shrinkwrap pins — all previously unguarded.
• Every check in this PR was run against a real local build (see Verification), not asserted — including two negative-case sanity checks (deliberately corrupting the shrinkwrap/a version to confirm the new CI guards actually fail closed).
• No unit tests: this is a Dockerfile/CI/build-tools-script change, no application code changed. check-shrinkwrap-pins.mjs doesn't have its own unit test — it's a 50-line CI-only script exercised end-to-end by the workflow itself.

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request updates the Dockerfile to extract the local tarball and run npm install directly in the project directory, ensuring that npm-shrinkwrap.json is honored for version pinning. It also strips devDependencies before installation to prevent unexpected dependency hoisting. Additionally, a new verification script check-shrinkwrap-pins.mjs is introduced to assert that the installed dependencies match the packed shrinkwrap pins. The reviewer suggested strictly validating that the packed shrinkwrap uses lockfileVersion: 3 in the verification script to prevent silent failures if the lockfile format changes.

[claude]

Reviewed; no blockers found.