fix(agent): bundle busybox nsenter — host binary fails to load in distroless - #145
Merged
Conversation
…roless PR #144 made stat() of /proc/1/root/usr/bin/nsenter succeed but execve still failed with ENOENT, observed live on mjolnir: pty: start failed: fork/exec /proc/1/root/usr/bin/nsenter: no such file or directory Root cause: the host's nsenter is dynamically linked. Its PT_INTERP (e.g. /lib64/ld-linux-x86-64.so.2) is resolved by the kernel against the CALLER's mount namespace, not against PID 1's. The distroless container has no /lib64 and no glibc, so the ELF loader can't find the interpreter and execve returns ENOENT (the misleading error message refers to the missing interpreter, not the binary itself). Fix: bundle a statically-linked nsenter inside the agent image. busybox-musl ships a single static binary that dispatches applets by argv[0]; copying /bin/busybox to /usr/bin/nsenter inside the agent image gives us a working nsenter that exec's cleanly in any mount namespace. ~1MB image-size cost; agents not using host-exec pay it once but never run it. Resolver: reorder candidate list so container-local paths (/usr/bin/nsenter etc.) come first; /proc/1/root paths remain as last-ditch fallback for non-distroless agent flavors that happen to share enough libc layout with the host. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Follow-up to #144 fixing the distroless agent's host-exec console: the host's dynamically-linked nsenter could be stat()ed through /proc/1/root but failed to execve because its ELF interpreter is invisible in the container's mount namespace. This PR bundles a statically-linked busybox as /usr/bin/nsenter inside the agent image and reorders the resolver to prefer container-local paths.
Changes:
- Add a
COPY --from=busybox:1.37.0-musl /bin/busybox /usr/bin/nsenterlayer to the agent Dockerfile. - Reorder
nsenterCandidatesso container-local/usr/bin,/bin,/usr/sbin,/sbinprecede/proc/1/root/...fallbacks. - Trim
/proc/1/root/usr/sbin/nsenterand/proc/1/root/sbin/nsenterfrom the fallback list and remove the now-redundant in-resolver/usr/bin/nsenterlast-ditch check.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| gearbox-agent/internal/api/console/pty/nsenter_linux.go | Reworked candidate list, comments, and resolver to prefer bundled in-container nsenter. |
| gearbox-agent/Dockerfile | Copies a statically-linked busybox binary into the distroless image as /usr/bin/nsenter. |
- Pin busybox by image-index digest (sha256:19b646668802469d968a05342a601e78da4322a414a7c09b1c9ee25165042138) so the build is reproducible across upstream tag rebuilds. buildx multi-arch resolver picks the right platform manifest from the index automatically. Verified busybox 1.37.0's nsenter applet supports -t/--target, -m/--mount, -u/--uts, -i/--ipc, -n/--net, -p/--pid — the exact flags SpawnNsenter emits. - Restore /proc/1/root/usr/sbin/nsenter and /proc/1/root/sbin/nsenter to the candidate list so the host-fallback search mirrors the container-local list (which already includes /usr/sbin and /sbin). - "binary not found" error now reports the actual list searched (built from the candidates slice) instead of the stale "/proc/1/root and /usr/bin" string. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #144. The resolver change there let stat() find the host's nsenter via
/proc/1/root/usr/bin/nsenter, but execve still failed:The host's
nsenteris dynamically linked. ItsPT_INTERP(e.g./lib64/ld-linux-x86-64.so.2) is resolved by the kernel against the caller's mount namespace, not against PID 1's. Distroless containers have no/lib64and no glibc, so the ELF loader can't find the interpreter and execve returns ENOENT. The error message is misleading — it refers to the missing interpreter, not the binary itself.Fix
Bundle a statically-linked
nsenterinside the agent image.busybox-muslships a single static binary that dispatches applets byargv[0]; copying/bin/busyboxto/usr/bin/nsenterinside the agent image gives a working nsenter that exec's cleanly in any mount namespace. ~1MB image-size cost; agents not using host-exec mode never run it.Resolver: container-local paths (
/usr/bin/nsenter, …) come first now;/proc/1/root/...paths remain as last-ditch fallback for non-distroless agent flavors that happen to share enough libc layout with the host.Test plan
go test ./...green on agent modulehostnamereturns the TrueNAS hostname;pwdshows host directory)🤖 Generated with Claude Code