Skip to content

fix(agent): bundle busybox nsenter — host binary fails to load in distroless - #145

Merged
sarg3nt merged 2 commits into
mainfrom
fix/nsenter-bundle-busybox
May 19, 2026
Merged

fix(agent): bundle busybox nsenter — host binary fails to load in distroless#145
sarg3nt merged 2 commits into
mainfrom
fix/nsenter-bundle-busybox

Conversation

@sarg3nt

@sarg3nt sarg3nt commented May 19, 2026

Copy link
Copy Markdown
Owner

Summary

Follow-up to #144. The resolver change there let stat() find the host's nsenter via /proc/1/root/usr/bin/nsenter, but execve still failed:

pty: start failed: fork/exec /proc/1/root/usr/bin/nsenter:
  no such file or directory

The host's nsenter is dynamically linked. Its PT_INTERP (e.g. /lib64/ld-linux-x86-64.so.2) is resolved by the kernel against the caller's mount namespace, not against PID 1's. Distroless containers have no /lib64 and no glibc, so the ELF loader can't find the interpreter and execve returns ENOENT. The error message is misleading — it refers to the missing interpreter, not the binary itself.

Fix

Bundle a statically-linked nsenter inside the agent image. busybox-musl ships a single static binary that dispatches applets by argv[0]; copying /bin/busybox to /usr/bin/nsenter inside the agent image gives a working nsenter that exec's cleanly in any mount namespace. ~1MB image-size cost; agents not using host-exec mode never run it.

Resolver: container-local paths (/usr/bin/nsenter, …) come first now; /proc/1/root/... paths remain as last-ditch fallback for non-distroless agent flavors that happen to share enough libc layout with the host.

Test plan

  • Linux build green
  • go test ./... green on agent module
  • Redeploy agent on mjolnir; click Mjolnir shell icon — expect session to open and land on TrueNAS host (hostname returns the TrueNAS hostname; pwd shows host directory)
  • Container size diff < 2MB

🤖 Generated with Claude Code

…roless

PR #144 made stat() of /proc/1/root/usr/bin/nsenter succeed but execve
still failed with ENOENT, observed live on mjolnir:

  pty: start failed: fork/exec /proc/1/root/usr/bin/nsenter:
    no such file or directory

Root cause: the host's nsenter is dynamically linked. Its PT_INTERP
(e.g. /lib64/ld-linux-x86-64.so.2) is resolved by the kernel against
the CALLER's mount namespace, not against PID 1's. The distroless
container has no /lib64 and no glibc, so the ELF loader can't find
the interpreter and execve returns ENOENT (the misleading error
message refers to the missing interpreter, not the binary itself).

Fix: bundle a statically-linked nsenter inside the agent image.
busybox-musl ships a single static binary that dispatches applets
by argv[0]; copying /bin/busybox to /usr/bin/nsenter inside the
agent image gives us a working nsenter that exec's cleanly in any
mount namespace. ~1MB image-size cost; agents not using host-exec
pay it once but never run it.

Resolver: reorder candidate list so container-local paths
(/usr/bin/nsenter etc.) come first; /proc/1/root paths remain as
last-ditch fallback for non-distroless agent flavors that happen
to share enough libc layout with the host.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings May 19, 2026 00:08
@github-actions github-actions Bot added the agent label May 19, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Follow-up to #144 fixing the distroless agent's host-exec console: the host's dynamically-linked nsenter could be stat()ed through /proc/1/root but failed to execve because its ELF interpreter is invisible in the container's mount namespace. This PR bundles a statically-linked busybox as /usr/bin/nsenter inside the agent image and reorders the resolver to prefer container-local paths.

Changes:

  • Add a COPY --from=busybox:1.37.0-musl /bin/busybox /usr/bin/nsenter layer to the agent Dockerfile.
  • Reorder nsenterCandidates so container-local /usr/bin, /bin, /usr/sbin, /sbin precede /proc/1/root/... fallbacks.
  • Trim /proc/1/root/usr/sbin/nsenter and /proc/1/root/sbin/nsenter from the fallback list and remove the now-redundant in-resolver /usr/bin/nsenter last-ditch check.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File Description
gearbox-agent/internal/api/console/pty/nsenter_linux.go Reworked candidate list, comments, and resolver to prefer bundled in-container nsenter.
gearbox-agent/Dockerfile Copies a statically-linked busybox binary into the distroless image as /usr/bin/nsenter.

Comment thread gearbox-agent/internal/api/console/pty/nsenter_linux.go
Comment thread gearbox-agent/internal/api/console/pty/nsenter_linux.go
Comment thread gearbox-agent/Dockerfile Outdated
- Pin busybox by image-index digest
  (sha256:19b646668802469d968a05342a601e78da4322a414a7c09b1c9ee25165042138)
  so the build is reproducible across upstream tag rebuilds. buildx
  multi-arch resolver picks the right platform manifest from the
  index automatically. Verified busybox 1.37.0's nsenter applet
  supports -t/--target, -m/--mount, -u/--uts, -i/--ipc, -n/--net,
  -p/--pid — the exact flags SpawnNsenter emits.
- Restore /proc/1/root/usr/sbin/nsenter and /proc/1/root/sbin/nsenter
  to the candidate list so the host-fallback search mirrors the
  container-local list (which already includes /usr/sbin and /sbin).
- "binary not found" error now reports the actual list searched
  (built from the candidates slice) instead of the stale
  "/proc/1/root and /usr/bin" string.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@sarg3nt
sarg3nt merged commit ac34c8f into main May 19, 2026
22 checks passed
@sarg3nt
sarg3nt deleted the fix/nsenter-bundle-busybox branch May 28, 2026 16:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants