Skip to content

fix(spawn): map full id range for nested/in-pod builds so image-owned files copy up (#432)#433

Merged
skeptomai merged 1 commit into
mainfrom
fix/nested-build-idmap-range-432
Jul 4, 2026
Merged

fix(spawn): map full id range for nested/in-pod builds so image-owned files copy up (#432)#433
skeptomai merged 1 commit into
mainfrom
fix/nested-build-idmap-range-432

Conversation

@skeptomai

Copy link
Copy Markdown
Collaborator

Summary

Fixes #432. A nested pelagos build RUN step (uid 0 without CAP_SYS_ADMIN — a k8s pod, the #426 path) could not create files inside image directories owned by a non-root uid/gid. cargo build was the visible casualty:

error: failed to acquire package cache lock
  failed to open: /usr/local/cargo/.package-cache
  Value too large for defined data type (os error 75)   # EOVERFLOW

Root cause (isolated end-to-end on an ext4 node)

The nested user namespace mapped only 0→0. Container image layers own files as arbitrary ids — rust's /usr/local/cargo is gid 983. overlayfs copy-up must preserve that ownership onto the new upper file, but an id unmapped in the namespace cannot be represented, so any create beneath such a directory fails with EOVERFLOW.

Everything else was a red herring, including the issue's own framing:

  • ❌ "package cache lock" — fails at openat, before any flock
  • userxattr — the real mount doesn't even use it
  • ❌ inode overflow / xino / cross-device — inodes ~10M, same device
  • ❌ Rust / O_LARGEFILE — plain touch (glibc) fails identically
  • unmapped uid/gid in the nested userns → copy-up EOVERFLOW

Fix

In the nested uid-0 case, identity-map the whole id space instead of 0→0. We still hold CAP_SETUID/CAP_SETGID there (only CAP_SYS_ADMIN was dropped), so the parent writes a multi-count map — the same mechanism already used for root-created user namespaces. Keeps in-pod builds fully rootless on native overlay (no privileged pod, no fuse-overlayfs perf hit). True rootless (uid ≠ 0) is unchanged (keeps subuid-range mapping). Both spawn and spawn_interactive map-setup sites updated.

Validation

  • Gold-standard end-to-end: the real nested rust:1-bookworm + cargo build RUN step that failed with EOVERFLOW now completes and produces an image, rootless on ext4 (ipc4).
  • Isolation proof: 0→0 map → CREATE-FAIL (EOVERFLOW); identity range map → CREATE-OK.
  • New integration test test_nested_build_overlay_copyup_unmapped_gid (+ doc); cargo test --lib 392 passed; canonical cargo clippy -- -D warnings clean.
  • Reproduction / root-cause / fix-proof scripts in scripts/repro-432-*.sh.

🤖 Generated with Claude Code

… files copy up (#432)

A nested `pelagos build` RUN step (uid 0 without CAP_SYS_ADMIN — a k8s pod, the
#426 path) could not create files inside image directories owned by a non-root
uid/gid. cargo build was the visible casualty:

    error: failed to acquire package cache lock
      failed to open: /usr/local/cargo/.package-cache
      Value too large for defined data type (os error 75)   # EOVERFLOW

Root cause (isolated end-to-end on an ext4 node): the nested user namespace
mapped only `0→0`. Container image layers own files as arbitrary ids — rust's
`/usr/local/cargo` is gid 983. overlayfs copy-up must preserve that ownership
onto the new upper file, but an id that is unmapped in the namespace cannot be
represented, so *any* create beneath such a directory fails with EOVERFLOW. It
is not cargo-specific (plain `touch` fails identically), not the flock, not the
file size, and not `userxattr` — the real pelagos mount does not even use
userxattr. cargo is simply the first thing a real build does that writes under
a non-root-owned image directory (its package-cache lock).

Fix: in the nested uid-0 case, identity-map the whole id space instead of just
`0→0`. We still hold CAP_SETUID/CAP_SETGID there (only CAP_SYS_ADMIN was
dropped), so the parent writes a multi-count map — the same mechanism already
used for root-created user namespaces. This keeps in-pod builds fully rootless
on native overlay (no privileged pod, no fuse-overlayfs perf hit). True rootless
(uid != 0) is unchanged: it keeps subuid-range mapping, falling back to the
single-id map when no subuid delegation exists. Both the `spawn` and
`spawn_interactive` map-setup sites are updated; a new `has_effective_cap`
helper generalises the existing CAP_SYS_ADMIN probe.

Validated end-to-end: the real nested `rust:1-bookworm` + `cargo build` RUN step
that failed with EOVERFLOW now completes and produces an image, rootless on
ext4. Reproduction and root-cause/fix isolation scripts land in scripts/.

Adds `test_nested_build_overlay_copyup_unmapped_gid` (drops CAP_SYS_ADMIN, builds
an overlay whose lower has a gid-983 directory, creates a file inside it) and its
entry in docs/INTEGRATION_TESTS.md.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@skeptomai skeptomai merged commit 13b6371 into main Jul 4, 2026
6 checks passed
@skeptomai skeptomai deleted the fix/nested-build-idmap-range-432 branch July 4, 2026 15:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Nested rootless pelagos build: cargo (flock) fails with EOVERFLOW on the userxattr overlay

1 participant