Skip to content

feat(box): add privileged plumbing for DinD - #646

Open
G4614 wants to merge 31 commits into
boxlite-ai:mainfrom
G4614:feat/dind-privileged-plumbing
Open

feat(box): add privileged plumbing for DinD#646
G4614 wants to merge 31 commits into
boxlite-ai:mainfrom
G4614:feat/dind-privileged-plumbing

Conversation

@G4614

@G4614 G4614 commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

Add Docker-style advanced.privileged across the Rust runtime, REST API, runner, CLI, and SDKs. privileged=true is normalized to capabilities.add=["ALL"] with an empty drop list and removes only /proc/sys from the guest readonly paths; capability add/drop remains a separate shape.

Test plan:

  • cargo fmt --all -- --check
  • cargo check -p boxlite-guest --target aarch64-unknown-linux-musl --tests
  • API Jest tests for REST validation, normalization, and mapping (30 tests)
  • API development build and TypeScript check
  • Python E2E case added for REST privileged vs cap_add=ALL behavior
  • Run apps/e2e/cases/test_privileged_options.py on the labeled Linux E2E runner

The Linux E2E workflow remains label-gated because it requires nested KVM. Privileged mode does not add a cgroup mount or clear masked paths.

@cla-assistant

cla-assistant Bot commented Jun 6, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@G4614
G4614 force-pushed the feat/dind-privileged-plumbing branch from b928ed3 to c581579 Compare July 14, 2026 09:01
@coderabbitai

coderabbitai Bot commented Jul 14, 2026

Copy link
Copy Markdown

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

CapabilitySet now identifies privileged containers by full guest-kernel capability coverage. OCI Linux specs use that state to conditionally configure readonly paths while preserving masked paths, with tests covering both privilege modes.

Changes

Privileged container specification

Layer / File(s) Summary
Capability privilege detection
src/guest/src/container/capabilities.rs
CapabilitySet::is_privileged() now requires all capabilities supported by the guest kernel, with coverage for narrow and full capability requests.
Privilege-aware OCI spec construction
src/guest/src/container/spec.rs
OCI spec creation passes privilege state to Linux configuration; privileged specs filter /proc/sys from readonly paths, while unprivileged specs leave defaults applied and preserve masking.
Privilege-dependent behavior tests
src/guest/src/container/spec.rs
Tests update helper calls and verify readonly paths, /proc/sysrq-trigger, and /proc/kcore behavior across privilege states.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant CapabilitySet
  participant create_oci_spec
  participant build_linux_spec
  CapabilitySet->>create_oci_spec: is_privileged()
  create_oci_spec->>build_linux_spec: privileged flag
  build_linux_spec-->>create_oci_spec: readonly and masked paths
Loading
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Description check ⚠️ Warning The description includes a summary and verification steps but omits the required Call graph and Changes sections. Add the required Call graph and Changes sections, and include the before-and-after execution path and notable implementation changes.
✅ Passed checks (4 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly identifies the main change: adding privileged-container plumbing for DinD.
✨ Finishing Touches 💡 1
🛠️ Fix failing CI checks 💡
  • Fix failing CI checks
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@G4614
G4614 force-pushed the feat/dind-privileged-plumbing branch from c581579 to 8712206 Compare July 14, 2026 10:25
@G4614
G4614 force-pushed the feat/dind-privileged-plumbing branch 2 times, most recently from ae4a493 to 542c814 Compare July 29, 2026 04:57
@G4614
G4614 marked this pull request as ready for review July 29, 2026 05:16
@G4614
G4614 requested a review from a team as a code owner July 29, 2026 05:16
@boxlite-agent

boxlite-agent Bot commented Jul 29, 2026

Copy link
Copy Markdown

📦 BoxLite review — couldn't complete

claude exited 1

stdout:
{"is_error":true,"duration_api_ms":0,"num_turns":1,"stop_reason":"stop_sequence","session_id":"fb413cef-022f-49f7-925c-086b2effb0a4","total_cost_usd":0,"usage":{"output_tokens_details":{"thinking_tokens":0},"input_tokens":0,"cache_creation_input_tokens":0,"cache_read_input_tokens":0,"output_tokens":0,"server_tool_use":{"web_search_requests":0,"web_fetch_requests":0},"service_tier":"standard","cache_creation":{"ephemeral_1h_input_tokens":0,"ephemeral_5m_input_tokens":0},"inference_geo":"","iterations":[],"speed":"standard"},"modelUsage":{},"permission_denials":[],"terminal_reason":"api_error","fast_mode_state":"off","fast_mode_disabled_reason":"sdk_opt_in_required","subtype":"success","api_error_status":403,"result":"Your organization has disabled Claude subscription access for Claude Code · Use an Anthropic API key instead, or ask your admin to enable access","type":"result","duration_ms":290,"uuid":"e41f4eb3-a12a-41f3-a9c7-7b0a38f56c56"}

stderr:
<empty>

powered by BoxLite

@G4614
G4614 force-pushed the feat/dind-privileged-plumbing branch from 542c814 to 75a0811 Compare July 29, 2026 05:23

@boxlite-agent boxlite-agent Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📦 BoxLite review — 2 issues

Comment thread src/guest/src/container/spec.rs Outdated
Comment thread src/guest/src/container/capabilities.rs Outdated
@G4614
G4614 force-pushed the feat/dind-privileged-plumbing branch 2 times, most recently from 49babff to c8c4a10 Compare July 29, 2026 07:53

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/guest/src/container/spec.rs`:
- Around line 170-173: Update the container spec flow around `privileged` and
`build_linux_spec` to pass privilege state into mount generation, and when
enabled create an isolated cgroup namespace/hierarchy before mounting it
read-write; keep the guest-wide hierarchy inaccessible. Also revise the adjacent
comment to describe `is_privileged()` accurately rather than implying
CAP_SYS_ADMIN alone is sufficient.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 005eff7d-cd06-4cd6-af93-7bbe65fc3ba8

📥 Commits

Reviewing files that changed from the base of the PR and between 49babff and c8c4a10.

📒 Files selected for processing (2)
  • src/guest/src/container/capabilities.rs
  • src/guest/src/container/spec.rs

Comment thread src/guest/src/container/spec.rs Outdated
@G4614
G4614 force-pushed the feat/dind-privileged-plumbing branch 2 times, most recently from c4b5ed3 to 38402cb Compare July 29, 2026 10:49

@DorianZheng DorianZheng left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the wiring is small, but is_privileged() turns an existing capability flag into a sandbox switch, and a few things outside the diff now hang off it.

guest version gate not bumpedMIN_CAPABILITY_GUEST_VERSION (src/boxlite/src/litebox/init/tasks/guest_init.rs:23) is still (0, 9, 8), but add=["ALL"] now means two different things depending on which guest you land on. guest rootfs images are cached per version and reused — the constant's own doc says exactly that — so a host built after this change can boot an old cached 0.9.8 guest, pass the version check, get 41 caps with /proc/sys still read-only, and dockerd dies on ip_forward with nothing pointing at the cause.

no privileged flag anywhereContainerCapabilities is add/drop only at every layer (cli.rs:473, advanced_options.rs:574, the proto). so you can't ask for a writable /proc/sys without also taking all 41 caps, and you can't take all 41 without the wider sandbox. docs/reference/cli/README.md:625 still describes --cap-add as capabilities only. and config.json is rebuilt on every start (lifecycle.rs:180 → start.rs:126), so a box created with --cap-add ALL before this change gets a writable /proc/sys on its next restart after upgrade — no flag touched, no code changed on their side.

no opt-in gate — the repo already has one for this shape of change: ExperimentalFeature::{CustomKernel, NestedVirtualization} behind BOXLITE_EXPERIMENTAL (experimental.rs:16-24). nested virt only hands the workload a /dev/kvm node inside its own VM and is gated. this opens the guest's sysctl tree and isn't.

the capability table is now load-bearingCAPABILITIES_BY_NUMBER is [Capability; 41], and supported_capabilities() does .take(cap_last_cap + 1) with no bound check (capabilities.rs:232-236). on a future kernel with cap_last_cap > 40 it silently truncates: 41 ⊆ 41, so the box is labelled fully privileged while genuinely missing the newest caps — and ensure_supported rejects those same caps with "is not supported by the guest kernel". it used to be an input-validation aid; it now decides a sandbox.

design doc — docs/architecture/container-capabilities.md:49 (status: accepted) says downstream code "cannot reinterpret the policy", which is what create_oci_spec now does. its user-facing advice to prefer drop=["ALL"] plus the minimum additions is also now the recipe that turns DinD off.

description is stale — it says --cap-add SYS_ADMIN counts as privileged (the code deliberately rejects that), that a writable cgroup2 mount at /sys/fs/cgroup is added (still commented out at spec.rs:644), and that readonlyPaths/maskedPaths are cleared (one entry is dropped, masked paths are untouched). the commit message gets all three right. it also still says "remains draft" but the PR isn't.

unrelated hunk — src/deps/libkrun-sys/build.rs:476 is formatting only.

Comment thread src/guest/src/container/capabilities.rs Outdated
Comment thread src/guest/src/container/capabilities.rs Outdated
Comment thread src/guest/src/container/capabilities.rs Outdated
Comment thread src/guest/src/container/spec.rs Outdated
Comment thread src/guest/src/container/spec.rs Outdated
Comment thread src/guest/src/container/spec.rs Outdated
Comment thread src/guest/src/container/spec.rs Outdated
Comment thread src/guest/src/container/spec.rs
Comment thread src/guest/src/container/spec.rs
@G4614
G4614 force-pushed the feat/dind-privileged-plumbing branch from 38402cb to d49e642 Compare July 29, 2026 12:39
@G4614

G4614 commented Jul 29, 2026

Copy link
Copy Markdown
Contributor Author

All nine addressed. The shape changed as a result, so it is easier to describe what the PR does now than to answer point by point.

Privileged now clears both lists, matching docker run --privileged. Your #8 is the reason. I checked it rather than argued it: in a --cap-add ALL box, /proc/bus starts ro and one mount -o remount,rw /proc/bus turns it rw. So the subset I was keeping stopped nothing, and the comment claiming otherwise was the worst part of it. A default container has no CAP_SYS_ADMIN, and there the same lists do hold — mount: permission denied — so the hardened path is unchanged.

I tried to make them enforceable instead. A user namespace is the mechanism that would do it, and the uid/gid mappings are already built here — only the LinuxNamespaceType::User line is commented out. Adding it fails in the guest:

build failed: received unexpected message: OtherError("cgroup error: systemd error:
systemd cgroup feature is required, but was not enabled during compile time"),
expected: WriteMapping

libcontainer routes a user namespace through the systemd cgroup manager, and src/guest/Cargo.toml builds it with default-features = false, features = ["v2", "libseccomp"]. That is a guest dependency question, not something this PR can settle, so it is out of scope here and worth its own issue.

#9 stands and I could not fix it. It is the whole /proc/sys tree, and it is the guest's, not the container's. I tried a network namespace for the privileged branch — it is created, and /proc/sys/net/* does become per-container — but the namespace comes up empty, with no veth and no route, so dockerd cannot reach a registry (failed to resolve reference). Wiring it into gvproxy is its own piece of work. Reverted; the commit message now says the guest-wide tree opens, without claiming otherwise.

#1, #2is_privileged returns BoxliteResult<bool> and propagates the ceiling-read failure, since resolve treats it as fatal at line 107. On the keyword: I kept the resolved-set comparison and fixed the doc instead. Naming 27 capabilities and typing ALL leave the container equally powerful, and the sandbox should follow what it can do, not how the request was spelled — but you are right that the doc claimed something else.

#3, #4 — both local lists are gone. Nothing consults them now, and as you noted the copy had already drifted from what Linux::default() actually ships.

#5, #6, #7 — the tautological assert is gone. The gap you named in #6 is the one that mattered: capability_set_decides_which_shape_the_spec_gets now drives create_oci_spec from a real CapabilitySet and asserts on the spec it produces, so inverting the predicate at the call site fails the suite instead of passing it.

Verified end to end after the rework: Docker 29.6.2 in a box on this build pulls images and runs both single containers and a two-service compose project with working service DNS.

@G4614
G4614 force-pushed the feat/dind-privileged-plumbing branch from d49e642 to b5d605a Compare July 29, 2026 12:46
@G4614

G4614 commented Jul 29, 2026

Copy link
Copy Markdown
Contributor Author

Correction to my previous comment: I wrote that nothing filters mount(2), and implied this repository has no seccomp. The second half was wrong.

There is a seccomp filter — jailer::seccomp::apply_vmm_filter, built from resources/seccomp/*.json and applied in src/shim/src/main.rs — but it wraps the VMM process on a Linux host and guards the host from the VMM. It says nothing about what a container may call inside the guest.

The container spec is the part with no seccomp: create_oci_spec never sets linux.seccomp, so mount(2) reaches the guest kernel unfiltered and the remount I demonstrated succeeds. (The libseccomp feature on libcontainer compiles the capability in; nothing in this tree emits a profile that would use it.)

So the conclusion holds, but the reasoning I gave for it was sloppy about which layer is protected. The commit message now states it precisely.

@G4614
G4614 force-pushed the feat/dind-privileged-plumbing branch from b5d605a to d1f9396 Compare July 29, 2026 13:04
@G4614
G4614 requested a review from DorianZheng July 30, 2026 06:39
@G4614
G4614 force-pushed the feat/dind-privileged-plumbing branch 2 times, most recently from a6509dd to 85847d4 Compare July 31, 2026 04:32

@boxlite-agent boxlite-agent Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📦 BoxLite review — 1 issue

Comment thread src/boxlite/src/runtime/advanced_options.rs Outdated
@G4614
G4614 force-pushed the feat/dind-privileged-plumbing branch from 85847d4 to d06bd67 Compare July 31, 2026 04:42
@G4614
G4614 requested review from DorianZheng and removed request for DorianZheng August 3, 2026 04:18
@G4614
G4614 force-pushed the feat/dind-privileged-plumbing branch from cbce52d to c4d0e63 Compare August 3, 2026 05:00
G4614 and others added 28 commits August 12, 2026 19:59
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
apps/api, apps/runner, apps/libs (generated clients), and sdks/** now
ship in boxlite-ai#1156 instead. This PR is left with the
guest + core runtime + CLI + REST contract — the actual privileged/
DinD feature — so review stays scoped to the security-relevant parts
of the change instead of being split across 84 files of mixed risk.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ResolvedSecurityPolicy::from_resolved no longer validates a canonical
privileged shape, so a test asserting that mismatched atomic options
and capabilities still resolve is testing a trivial pass-through, not
guest behavior. resolved_policy_consumes_atomic_security_options and
all_capabilities_without_privileged_keep_proc_sys_readonly already
cover the same constructor with both all-true and all-false inputs.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
cgroup_namespace and allow_all_devices are unnecessary for DinD: the
guest never enforced a restrictive device-cgroup default, and dockerd
tolerates running without a private cgroup namespace view. Only
unconfined_paths (masked/readonly path relief, for network sysctls)
and writable_sysfs (/sys stays writable, for dockerd's own cgroup
writes) are load-bearing.

ResolvedContainerSecurityConfig / ContainerAdvancedConfig /
ResolvedSecurityPolicy drop from 4 fields to 2. service.proto
renumbers ContainerAdvancedOptions to match (internal wire message,
no external consumer pinned to the old field numbers).

Before
  resolve_container_security     (Core · advanced_options.rs:789)
    -> ResolvedContainerSecurityConfig{cgroup_namespace, writable_sysfs, allow_all_devices, unconfined_paths}
         -> create_oci_spec       (Guest · spec.rs:153)
              |- build_default_namespaces(cgroup_namespace)   — adds a cgroup namespace
              |- build_linux_spec(..., allow_all_devices, ...) — builds an allow-all device-cgroup rule
              `- build_standard_mounts(..., writable_sysfs)

After
  resolve_container_security     (Core · advanced_options.rs:789)
    -> ResolvedContainerSecurityConfig{unconfined_paths, writable_sysfs}
         -> create_oci_spec       (Guest · spec.rs:153)
              |- build_default_namespaces()                    — fixed list, no cgroup namespace
              |- build_linux_spec(..., unconfined_paths)        — no device-cgroup rule
              `- build_standard_mounts(..., writable_sysfs)

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
architecture/README.md, container-capabilities.md, cli/README.md, and
rust/README.md described privileged mode with a dead link and with the
cgroup namespace / allow-all device rule this PR's own last commit
removed. Reverting to main here rather than patching them in place:
the accurate design record already lives outside the repo, and this
PR's diff should stay code, not docs that immediately went stale.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
create_inner's own normalize_privileged() call always ran on data
sanitize_local_options had already normalized moments earlier in the
same call: RuntimeImpl is never re-exported outside the crate, and
core.rs's only construction site wraps it in LocalRuntime immediately,
so every reachable caller goes through sanitize_local_options first.
Replaced the redundant call with a debug_assert documenting that
invariant.

get_or_create_rejects_privileged_upgrade called RuntimeImpl directly
with a hand-built AdvancedBoxOptions{privileged: true, ..} literal,
bypassing that pipeline the way no real caller does; updated it to use
set_privileged, which is what actually produces the normalized shape.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ies verbatim

Host resolve_container_security() now sends literal masked_paths,
readonly_paths, and /sys mount options instead of two booleans; the guest
assigns them as-is with no reinterpretation. Matches how Docker, Podman, and
Kata Containers hand the enforcing side a finished OCI shape rather than a
flag to re-derive. capabilities stays as add/drop deltas — the guest is the
only side that knows its own kernel's capability ceiling.

Before: host sends {capabilities, unconfined_paths: bool, writable_sysfs: bool}
  → guest capabilities.rs::from_resolved(bool, bool)
  → spec.rs::build_linux_spec(.., unconfined_paths: bool) branches on it
  → spec.rs::build_standard_mounts(.., writable_sysfs: bool) branches on it

After: host resolve_container_security() resolves literal values
  → ContainerAdvancedOptions{masked_paths, readonly_paths, sys_mount_options}
  → guest capabilities.rs::from_resolved(Vec<String>, Vec<String>, Vec<String>)
  → spec.rs::build_linux_spec(.., masked_paths, readonly_paths) assigns verbatim
  → spec.rs::build_standard_mounts(.., sys_mount_options) assigns verbatim

Moved the privileged-vs-hardened test coverage to where the decision is now
made (advanced_options.rs); guest-side tests now assert pass-through fidelity.
…ollow-up PR

Keeps boxlite-ai#646 to the mechanism itself — proto, host resolve, guest apply — same
split rationale boxlite-ai#1156 already used to separate control-plane/runner/SDK
consumers from boxlite-ai#646. The Rust API (AdvancedBoxOptions.privileged,
set_privileged) stays; only the CLI --privileged flag and the self-hosted
REST DTO/OpenAPI exposure move out, since neither adds mechanism, just a
caller.

Before: CLI/REST call AdvancedBoxOptions.privileged directly, bundled with
  the guest/core mechanism in one PR.
After: CLI/REST removed here, follow-up PR re-adds them on top of this
  mechanism-only branch (rust API entry point unchanged, so the follow-up is
  pure plumbing with nothing left to test beyond wiring).
advanced_options.rs hand-copied oci-spec's default masked/readonly path
lists rather than calling the crate's own public functions for them, even
though the host crate already depends on oci-spec (runtime feature, on by
default). Two independent copies of the same 10+5-entry list, one of which
could silently drift on an oci-spec bump.

Before: default_masked_paths() -> literal 10-entry Vec<String>
  default_readonly_paths() -> literal 5-entry Vec<String>
After: default_masked_paths() -> oci_spec::runtime::get_default_maskedpaths()
  default_readonly_paths() -> oci_spec::runtime::get_default_readonly_paths()
@G4614
G4614 force-pushed the feat/dind-privileged-plumbing branch from 4d27ef9 to 1eb8b2a Compare August 12, 2026 12:02
G4614 added a commit to G4614/boxlite that referenced this pull request Aug 12, 2026
…ed REST

Adds the CLI --privileged flag and the self-hosted REST/OpenAPI surface for
the AdvancedBoxOptions.privileged mechanism feat/dind-privileged-plumbing
(boxlite-ai#646) adds. Pure consumer: no new mechanism, just wiring a caller onto the
Rust API's set_privileged/AdvancedBoxOptions.privileged that already exists
on this branch.

Before: --privileged / the REST body field don't exist; only the Rust API
  (AdvancedBoxOptions.privileged, set_privileged) can request privileged mode.
After: cli.rs/commands::create::run parse --privileged and call
  set_privileged(true) -> AdvancedBoxOptions.privileged
  rest/types.rs deserializes the same field from the REST request body
  openapi/box.openapi.yaml documents both the request field and the
  capabilities.privileged_enabled response flag
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants