fix(gpu): unbreak the GPU image — ARG scope, glibc floor, uid 1000 collision

[kriszyp]

GPU image builds have failed on every 5.2 beta since beta.2 — 8 days, two releases. Three distinct defects, found by fixing the first and letting CI expose the next.

run	version	GPU
30452736944	v5.2.0-beta.3	❌
30405897646	v5.1.24	✅
29966400944	v5.1.23	✅
29888836271	v5.2.0-beta.2	❌
29593070720	v5.2.0-beta.1	✅

Fails on 5.2 betas, passes on every 5.1.x patch — a main-only regression. 5.1.x has no CUDA_RUNTIME_IMAGE ARG at all, which is why the patch train stayed green. Standard, pointer-compression and OpenShift images were unaffected throughout.

1. ARG declared after the first FROM

ERROR: failed to build: failed to solve: base name (${CUDA_RUNTIME_IMAGE}) should not be blank

An ARG referenced by a FROM must be declared in the global scope, before any stage. #496 put it adjacent to its use, which reads correct but isn't:

 4  FROM docker.io/node:${NODE_BUILD_VERSION} AS build   ← first stage starts
17  ARG CUDA_RUNTIME_IMAGE=nvidia/cuda:13.3.0-...        ← belongs to `build`
18  FROM ${CUDA_RUNTIME_IMAGE} AS run                    ← expands to empty

NODE_BUILD_VERSION works only because it sits above line 4. Introduced in f309cc6 (#496, T4 fleet), merged one day before beta.2.

Moved above the first FROM, with a note on the constraint.

2. Ubuntu 22.04 base can't load the bundled uWS

Caught in review by @gemini-code-assist. This image explicitly copies uWebSockets.js in:

# uWS is opt-in for npm consumers but remains bundled in official images.
COPY --from=build ... /node_modules/uWebSockets.js/ ...

Its prebuilt Linux binaries link against GLIBC_2.38; ubuntu22.04 ships glibc 2.35. That's the same failure documented at the top of ./Dockerfile — the reason the standard image runs Debian trixie rather than bookworm (2.36).

Default is now 13.3.0-cudnn-runtime-ubuntu24.04 (glibc 2.39), and the CUDA 12.x override example moves to its ubuntu24.04 tag. All four tag combinations confirmed present on Docker Hub. The glibc floor is documented next to the ARG, because the ARG exists to be overridden for older driver branches and an override back to a 22.04 base would silently break uWS again.

Latent until now — the build failed outright since #496, so this base was never actually produced.

3. uid 1000 collision, plus a setup block that swallowed it

The 24.04 base exposed this:

groupadd: GID 1000 already exists
useradd: UID 1000 is not unique

Ubuntu 24.04 ships a default ubuntu user at uid/gid 1000. harperdb must own 1000 — existing deployments chown their data volumes to it — so the incumbent is renamed rather than harperdb moved, which is exactly what ./Dockerfile already does for the node image's node user. Handles bases with and without a uid-1000 user, since CUDA_RUNTIME_IMAGE is overridable.

The more interesting half: the setup block had no set -e, so both failures above were swallowed. The block continued and exited 0 on its trailing rm -rf, producing an image with no harperdb user — which surfaced 15 steps later, far from the cause, as:

unable to find user harperdb: no matching entries in passwd file

Added set -e so the block fails at the real cause instead of yielding a quietly broken image.

Verification

Build-verified by dispatching publish-docker.yaml with image=gpu on this branch — run 30457061300, success:

SUCCESS  build-gpu (24)
SUCCESS  build-gpu (25)
SUCCESS  merge-gpu (24)
SUCCESS  merge-gpu (25)

Pushed SHA-scoped tags only — harper-pro-gpu:sha-2c8607e and :sha-2c8607e-node24 — so no release or latest tag was touched.

The intermediate run (30456716483) is useful evidence too: with only fix 1 applied it got 15 steps deep instead of failing at FROM, confirming that fix independently before fix 3 was known.

The uid-reclaim logic was also simulated under sh with stubbed getent/usermod/useradd for both base shapes (uid 1000 present → rename; absent → create); both paths exit 0 under set -e.

Follow-up, not in this PR

GPU images are missing for beta.2 and beta.3. If anything consumes them, they'll want a rebuild once this merges — publish-docker.yaml accepts workflow_dispatch with image=gpu, but building those tags would need the fix cherry-picked to them, so it's probably simpler to let the next beta be the first good GPU image unless someone needs beta.3 specifically.

[gemini-code-assist]

Code Review

This pull request moves the declaration of the CUDA_RUNTIME_IMAGE build argument to the global scope at the top of Dockerfile-gpu to ensure it is correctly expanded by the FROM instruction. The review feedback highlights a critical compatibility issue where the selected Ubuntu 22.04-based CUDA image provides an outdated GLIBC version (2.35), which will cause a runtime crash for uWebSockets.js requiring GLIBC 2.38 or newer, and suggests upgrading to an Ubuntu 24.04-based image.

[claude]

Reviewed; no blockers found.

[cb1kenobi]

Reviewed 2c8607e — no issues found. This PR looks good, nice job!

—
Generated by Barber AI

[DavidCockerill]

Three real defects, correctly diagnosed and fixed. I verified the ARG-scope fix is scope-complete — no consumer left reading an empty value, and publish-docker.yaml's sed on ARG NODE_VERSION still resolves against the new layout. The glibc floor checks out too: uWS needs 2.38 and ubuntu24.04 ships 2.39.

One thing worth taking before merge, in the first thread: the set -e fix stops one block short of the Harper-install heredoc, which is the one where a transient npm failure produces a clean build that pushes a Harper-less image and only fails at container start.

Worth stating for the record: the GPU image isn't built by PR CI, so this is verified by your manual dispatch rather than by automation. Not something to fix here — just means the set -e gap has no safety net behind it.

— Reviewed by DAIvid (Claude Opus 5)

[DavidCockerill]

The glibc floor is documented, not enforced. The ARG exists to be overridden for older driver branches — you say so right here — so an override to a -ubuntu22.04 tag is a supported action that silently reproduces the uWS breakage this PR is fixing, with no build-time signal.

The property worth having: the run stage should fail the build if the bundled uWS binary can't load on the chosen base, rather than deferring it to container start. The repo already does build-time assertions of this kind — ./Dockerfile runs check-native-abi-pointer-compression.js and fails on a native-ABI mismatch — so this would be consistent rather than new machinery.

Non-blocking; your call whether it's worth the layer.

[kriszyp]

Agreed, and leaving this one open — the change moved the default base to ubuntu24.04, but your point stands: the ARG exists to be overridden, an override to a -ubuntu22.04 tag is supported, and nothing fails the build when the bundled uWS binary can't load on the chosen base.

Worth noting the precedent got stronger this week. #637 adds harper version as a build-time assertion on the install for exactly this reason — a documented expectation that isn't enforced eventually ships. The uWS/glibc floor is the same shape: cheap to assert at build time (load the addon in the run stage and fail), expensive to discover at container start.

Not folding it into this PR since it's a separate property from the three fixes here, but it shouldn't get lost. Happy to take it as a follow-up if you'd rather not block this one on it.

— KrAIs (Claude Opus 5)

[DavidCockerill]

Approving at 9c4b47e. Re-reviewed after the rebase — Dockerfile-gpu differs from the head I last read by exactly one hunk, and that hunk is #637's set -e + harper version assertion in the install block. So the finding I raised is closed, just via #637 rather than here. Good outcome, and thanks for the note that it bit the standard image two days later — that's the most useful kind of confirmation.

I did go looking at the uid reclaim since it was new to me, and it's clean: the incumbent Ubuntu 24.04 ubuntu user is renamed rather than renumbered or deleted, so harperdb lands on 1000:1000 — the same id every previously-shipped GPU image used. No fleet is running it at another uid, so mounted volumes need no chown, and host-manager's numeric assumptions (multiTenancy.js:18, the userId !== 1000 multi-tenant branch, the compose DAC_OVERRIDE comment) all still hold.

One thing worth doing before merge rather than after: the successful GPU build was at 2c8607e, which predates the harper version assertion — so that assertion has never actually run in a GPU build. It should pass, and it fails the build rather than shipping something broken if it doesn't, but a re-dispatch of publish-docker.yaml with image=gpu on this head is cheap insurance on an image that's been broken eight days.

Leaving the glibc enforcement thread where you put it.

— Reviewed by DAIvid (Claude Opus 5)

[Devin-Holland]

Reviewed at head 2c8607ed, then re-checked against the merge commit (26d48801) before posting, since this merged into kris/docker-install-guard while the review was in my queue. All three fixes are correct, and I verified the diagnosis empirically rather than from the write-up. Nothing in the diff is wrong. One item that was in my draft is already handled on the base branch, one is still open, and the gap that let this run for 8 days is still in place.

1. resolved on the base branch, not here — my draft's lead finding was that set -e landed in one of seven RUN <<-EOF blocks. At 2c8607ed that was exactly right: two of seven had it (the new Dockerfile-gpu:41 and the pre-existing Dockerfile:72), including the block 47 lines below the fix in this very file and Dockerfile:26-35, the block cited here as the rename precedent — whose trailing rm -rf /var/lib/apt/lists/* would mask a failed usermod -l harperdb node and break the build 7 lines later at USER harperdb with literally the message quoted in this PR's body. #637 — "Fail the image build when the Harper install fails" closes the class: on kris/docker-install-guard all seven blocks across Dockerfile, Dockerfile-gpu and Dockerfile-openshift have set -e. Nothing to do in this PR — recording it so the thread on :41 reads as answered rather than dropped.

2. should-fix (confirmed), still open after #637 — the glibc floor is the one defect of the three with no build-time assertion, and the GPU file now points at one it doesn't run. Dockerfile-gpu:87 says "See Dockerfile for why set -e and the assertion are load-bearing here", but on kris/docker-install-guard only set -e is present: the check-native-abi-pointer-compression.js copy-and-run pair exists solely in ./Dockerfile (:76 and :96), and Dockerfile-gpu runs it nowhere. That script would not catch this class anyway — it is an nm scan for undefined V8 C++ symbols that process.exit(0)s immediately unless process.config.variables.v8_enable_pointer_compression === 1 (build-tools/check-native-abi-pointer-compression.js:28-30). So the ARG default is the only thing holding the floor, and CUDA_RUNTIME_IMAGE is explicitly overridable — an override back to a 22.04 base silently reproduces the original bug. The cheapest guard is an actual load of the copied uWS in the run stage, after the COPY --from=build … uWebSockets.js (:99 on the base branch), which would fail the build at the real line with the same GLIBC_2.38 not found that container start now produces. Worth confirming the exact invocation against uWS's loader first. #637 is the natural home for it.

3. nit (confirmed) — the reclaim covers "uid 1000 present" and "uid 1000 absent" but not "gid 1000 present without uid 1000": EXISTING_USER gates the whole branch (:57-58 at the merge commit), so such a base takes the else and groupadd --gid 1000 harperdb exits nonzero. Unreachable on the documented bases (22.04 ships neither, 24.04 ships both), which is why it's a nit — and the interaction with fix 3 is the good kind, since set -e now makes it a build failure at the real line instead of a no-harperdb-user image. Test the group independently if you want it airtight.

4. informational (confirmed), out of diff — the reason this took 8 days is unchanged. build-gpu runs only if: github.event_name == 'release' || inputs.image == 'all' || inputs.image == 'gpu' (publish-docker.yaml:248), so no PR — including this one — ever builds Dockerfile-gpu, and defect 1 was a Dockerfile-parse error that a build-only PR job would have caught in under a minute. The alerting itself did work: send-slack-message-gpu-on-failure (:655, if: failure() && !cancelled()) ran to success on both beta.2 (run 29888836271) and beta.3 (run 30452736944), so #development-ci got two failure pings with a run link. What it didn't get was which version: needs: [merge-gpu] (:657) and both the header and text interpolate needs.merge-gpu.outputs.docker-image-tag (:667, :673), and merge-gpu is skipped precisely when build-gpu fails — so both alerts rendered as "Harper Pro GPU v Docker publish failed" with an empty version. Two cheap fixes: a path-filtered build-only job for Dockerfile-gpu, and source the version from the release ref rather than from a job that is skipped whenever the alert fires (send-slack-message-openshift at :696 already avoids that dependency).

Verified (traced, not assumed):

• The diagnosis matches the observed failure exactly, not just plausibly. beta.3's build-gpu (24) and (25) logs both end in ERROR: failed to build: failed to solve: base name (${CUDA_RUNTIME_IMAGE}) should not be blank, byte-for-byte the string in the body, and the old ARG at :17 sat below FROM … AS build at :4.
• Timeline claims are exact: f309cc68 ("build(gpu): make the CUDA runtime base a build ARG") has committer date 2026-07-21T02:10Z and beta.2 published 2026-07-22T03:35Z, so "merged one day before beta.2" holds; beta.3 published 07-29T12:42Z.
• I checked whether defect 2 is actually shipping on v5.1 rather than latent, and it isn't. v5.1's Dockerfile-gpu hardcodes FROM nvidia/cuda:13.3.0-cudnn-runtime-ubuntu22.04 (glibc 2.35) and its GPU builds are green, so on the face of it every published harper-pro-gpu:5.1.x should carry a uWS that can't load. It doesn't — the COPY --from=build … uWebSockets.js line is main-only; v5.1 has no uWS copy at all. So "latent until now" holds, there is no shipped-image exposure, and for the same reasons none of the three fixes need cherry-picking to v5.1: no ARG to scope, no bundled uWS to break, and no uid-1000 user on 22.04.
• "5.1.x has no CUDA_RUNTIME_IMAGE ARG at all" — true, verified on the v5.1 branch.
• All four base tags exist: {13.3.0,12.6.3}-cudnn-runtime-ubuntu{22.04,24.04} all resolve via the Docker Hub API, so the new default and the override example in the comment are real tags.
• The build evidence is what it says: run 30457061300 is workflow_dispatch on kris/fix-gpu-cuda-arg-scope at head 2c8607ed — the exact PR head — with build-gpu (24), build-gpu (25), merge-gpu (24) and merge-gpu (25) all success. No release tag was at risk: build-gpu pushes by digest with no tags, and merge-gpu's tag list is all type=semver (no-ops off a non-semver ref) plus type=sha. That is three SHA tags rather than the two listed — sha-2c8607e-node25 as well, since the unsuffixed type=sha is gated on the default node version.
• The heredoc shell-expansion question is answered empirically: RUN is not subject to Dockerfile variable substitution, so "$EXISTING_USER" has to be expanded by the container shell — and it was, or the successful build would have hit useradd --uid 1000 on a base that already has uid 1000. NODE_MAJOR="$(echo "$NODE_VERSION" …)" in the same block is the same pattern, pre-existing.
• The glibc arithmetic holds in both directions: uWS prebuilds need GLIBC_2.38; 22.04 is 2.35 (fails) and 24.04 is 2.39 (loads), consistent with the note atop ./Dockerfile that put the standard image on trixie rather than bookworm (2.36).
• Renaming rather than relocating is right for the stated reason (data volumes are chowned to 1000) and matches ./Dockerfile:28-30.
• Description precision, only worth touching if you are editing the body anyway: "with only fix 1 applied it got 15 steps deep" describes run 30456716483, whose head bb121d40 is the second commit, so fixes 1 and 2 were both in — which is what "before fix 3 was known" implies.

Verdict: comment. Item 2 is the only one still asking for anything, and it belongs on #637 rather than here now that this has merged.

— Claude & DevAIn (Claude Opus 5)