Skip to content

Commit 16ed6e6

Browse files
committed
feat(driver-native): add containerd-backed native compute driver
Adds openshell-driver-native, an OCI/runc-based compute driver that builds sandboxes from Linux namespaces and cgroups v2. A system-provided containerd is used only for image pull, unpack, and snapshot management: this driver, not containerd, drives the configured low-level OCI runtime (runc/crun) directly, and containerd never creates a Container or Task for these sandboxes. Also extracts the VM driver's nftables ruleset generator into a shared openshell-nft-ruleset crate, and wires ComputeDriverKind::Native through openshell-core and every openshell-server compute-driver selection path. Hardening from review: sandbox names are validated against path traversal, the runtime's --root is scoped under the driver's own state directory instead of the shared global default, the sandbox endpoint no longer defaults to an unreachable loopback address, per-sandbox subnet allocation detects and avoids collisions instead of a stateless hash, cleanup after a failed create no longer leaves a mounted rootfs and bundle directory behind, and sandbox token files are written with owner-only permissions. Verified end to end against a real containerd 2.x + runc/crun install (see tests/containerd_integration.rs, #[ignore]d in CI since CI has no containerd). Known gaps (rootless mode, image-based supervisor injection, full CDI GPU support, polling-based watch) are documented in the driver README and docs/reference/sandbox-compute-drivers.mdx. Related: #2255 Signed-off-by: Eric Curtin <eric.curtin@docker.com>
1 parent d556748 commit 16ed6e6

30 files changed

Lines changed: 4524 additions & 65 deletions

File tree

Cargo.lock

Lines changed: 68 additions & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

architecture/build.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -29,8 +29,8 @@ collect telemetry: `openshell-server` (gateway), `openshell-sandbox`
2929
(supervisor), and `openshell-driver-vm`. Every crate depends on
3030
`openshell-core` with `default-features = false`, so the binary crate's feature
3131
is the single switch that enables `openshell-core/telemetry` for its build
32-
graph. In-process drivers (`docker`, `kubernetes`, `podman`) inherit the
33-
gateway's setting through feature unification and carry no passthrough.
32+
graph. In-process drivers (`docker`, `kubernetes`, `podman`, `native`) inherit
33+
the gateway's setting through feature unification and carry no passthrough.
3434

3535
Building a binary with `--no-default-features` compiles out telemetry entirely:
3636
no endpoint, no telemetry HTTP client, and no emission code. With telemetry

architecture/compute-runtimes.md

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -38,12 +38,14 @@ of re-querying drivers on each request.
3838
| Podman | Rootless or single-machine deployments. | Container plus nested sandbox namespace. | Uses the Podman REST API, OCI image volumes, and CDI GPU devices when available. |
3939
| Kubernetes | Cluster deployment through Helm. | Pod plus nested sandbox namespace. | Uses Kubernetes API objects, service accounts, secrets, PVC-backed workspace storage, and GPU resources. |
4040
| VM | Experimental microVM isolation. | Per-sandbox libkrun VM. | Managed endpoint-backed driver. The gateway spawns `openshell-driver-vm`, waits for its Unix socket, and then consumes it through the same remote `compute_driver.proto` path used by unmanaged endpoint drivers. The VM driver boots a cached bootstrap `rootfs.ext4`, prepares requested OCI images inside a bootstrap VM with `umoci`, attaches the prepared image disk read-only, and gives each sandbox a writable `overlay.ext4` for merged-root changes and runtime material. The driver persists each accepted launch request beside the overlay and restarts those VMs on driver startup without recreating the overlay. |
41-
| Extension | Out-of-tree drivers operated alongside the gateway. | Whatever boundary the driver implements. | Selected by a non-reserved custom `compute_drivers = ["<name>"]` entry with `[openshell.drivers.<name>].socket_path`, or at launch time by pairing `--drivers <name>` with `--compute-driver-socket=<path>`. Reserved built-in names such as `vm`, `docker`, `podman`, and `kubernetes` cannot be used as unmanaged socket endpoints. The gateway connects to a UDS the operator already provisioned, runs `GetCapabilities`, logs the advertised `driver_name`, and dispatches all sandbox lifecycle calls through `compute_driver.proto`. The driver process and socket lifecycle are operator-owned; the gateway does not spawn, supervise, or remove unmanaged extension drivers. The trust boundary is the socket's filesystem permissions: the operator must ensure only the gateway uid can read/write it. |
41+
| Native | Early/opt-in kernel-primitive isolation against an existing system `containerd`. | Container built from Linux namespaces + cgroups v2. | In-process driver, like Docker/Podman/Kubernetes. The driver itself — not containerd — spawns the configured `runtime_binary` (`runc` by default, `crun`, or another OCI-runtime-spec-compatible binary) directly through its `create`/`start`/`state`/`delete` CLI contract; containerd is used only for image pull/unpack and snapshot management (protected from containerd's GC via a lease, since no containerd `Container`/`Task` is ever created) and never bundled. Sandboxes are tracked by scanning the driver's own state directory rather than querying containerd or the runtime for a global list. Per-sandbox network namespace + veth pair + nftables ruleset (shared `openshell-nft-ruleset` crate, also used by the VM driver). GPU support is kernel-device-node passthrough only. Rootless mode is implemented but not yet functional (defaults off) — see `crates/openshell-driver-native/README.md`. |
42+
| Extension | Out-of-tree drivers operated alongside the gateway. | Whatever boundary the driver implements. | Selected by a non-reserved custom `compute_drivers = ["<name>"]` entry with `[openshell.drivers.<name>].socket_path`, or at launch time by pairing `--drivers <name>` with `--compute-driver-socket=<path>`. Reserved built-in names such as `vm`, `docker`, `podman`, `kubernetes`, and `native` cannot be used as unmanaged socket endpoints. The gateway connects to a UDS the operator already provisioned, runs `GetCapabilities`, logs the advertised `driver_name`, and dispatches all sandbox lifecycle calls through `compute_driver.proto`. The driver process and socket lifecycle are operator-owned; the gateway does not spawn, supervise, or remove unmanaged extension drivers. The trust boundary is the socket's filesystem permissions: the operator must ensure only the gateway uid can read/write it. |
4243

4344
Per-sandbox CPU and memory values currently enter the driver layer through
4445
template resource limits. Docker and Podman apply them as runtime limits.
4546
Kubernetes mirrors each limit into the matching request. VM accepts the fields
46-
but currently ignores them.
47+
but currently ignores them. Native applies them as cgroup v2 CPU quota/period
48+
and memory limits in the generated OCI spec.
4749

4850
Docker and Podman also accept per-sandbox driver-config mounts for existing
4951
runtime-managed named volumes and tmpfs mounts. Podman additionally accepts
@@ -76,6 +78,7 @@ Runtime-specific implementation notes belong in the driver crate README:
7678
- `crates/openshell-driver-podman/README.md`
7779
- `crates/openshell-driver-kubernetes/README.md`
7880
- `crates/openshell-driver-vm/README.md`
81+
- `crates/openshell-driver-native/README.md`
7982

8083
The combined VM topology runs `openshell-sandbox` as guest PID 1. libkrun
8184
executes the driver-owned guest bootstrap as PID 1, and the bootstrap preserves
@@ -91,6 +94,7 @@ The supervisor must be available inside each sandbox workload:
9194
| Podman | Read-only OCI image volume containing the supervisor binary. |
9295
| Kubernetes | Supervisor image side-loaded into the sandbox pod by image volume or init container. |
9396
| VM | Embedded in the guest rootfs bundle. |
97+
| Native | Bind-mounted from a host path (`supervisor_binary_path`), not sourced from an image like the other in-tree drivers — see the driver README's Known Gaps. |
9498
| Extension | Defined by the out-of-tree driver. |
9599

96100
Driver-controlled environment variables must override sandbox image or template

crates/openshell-core/src/config.rs

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -87,6 +87,7 @@ pub enum ComputeDriverKind {
8787
Vm,
8888
Docker,
8989
Podman,
90+
Native,
9091
}
9192

9293
impl ComputeDriverKind {
@@ -97,6 +98,7 @@ impl ComputeDriverKind {
9798
Self::Vm => "vm",
9899
Self::Docker => "docker",
99100
Self::Podman => "podman",
101+
Self::Native => "native",
100102
}
101103
}
102104
}
@@ -137,8 +139,9 @@ impl FromStr for ComputeDriverKind {
137139
"vm" => Ok(Self::Vm),
138140
"docker" => Ok(Self::Docker),
139141
"podman" => Ok(Self::Podman),
142+
"native" => Ok(Self::Native),
140143
other => Err(format!(
141-
"unsupported compute driver '{other}'. expected one of: kubernetes, vm, docker, podman"
144+
"unsupported compute driver '{other}'. expected one of: kubernetes, vm, docker, podman, native"
142145
)),
143146
}
144147
}
@@ -1001,6 +1004,10 @@ mod tests {
10011004
"docker".parse::<ComputeDriverKind>().unwrap(),
10021005
ComputeDriverKind::Docker
10031006
);
1007+
assert_eq!(
1008+
"native".parse::<ComputeDriverKind>().unwrap(),
1009+
ComputeDriverKind::Native
1010+
);
10041011
}
10051012

10061013
#[test]

crates/openshell-core/src/telemetry.rs

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -161,6 +161,7 @@ pub enum TelemetryComputeDriver {
161161
Kubernetes,
162162
Podman,
163163
Vm,
164+
Native,
164165
Unknown,
165166
}
166167

@@ -172,6 +173,7 @@ impl TelemetryComputeDriver {
172173
Self::Kubernetes => "kubernetes",
173174
Self::Podman => "podman",
174175
Self::Vm => "vm",
176+
Self::Native => "native",
175177
Self::Unknown => "unknown",
176178
}
177179
}
@@ -183,6 +185,7 @@ impl TelemetryComputeDriver {
183185
"k8s" | "kubernetes" => Self::Kubernetes,
184186
"podman" => Self::Podman,
185187
"vm" => Self::Vm,
188+
"native" => Self::Native,
186189
_ => Self::Unknown,
187190
}
188191
}
@@ -194,6 +197,7 @@ impl TelemetryComputeDriver {
194197
Some(crate::ComputeDriverKind::Kubernetes) => Self::Kubernetes,
195198
Some(crate::ComputeDriverKind::Podman) => Self::Podman,
196199
Some(crate::ComputeDriverKind::Vm) => Self::Vm,
200+
Some(crate::ComputeDriverKind::Native) => Self::Native,
197201
None => Self::Unknown,
198202
}
199203
}
Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
[package]
5+
name = "openshell-driver-native"
6+
description = "Linux-native compute driver for OpenShell, backed by a system-provided containerd"
7+
version.workspace = true
8+
edition.workspace = true
9+
rust-version.workspace = true
10+
license.workspace = true
11+
repository.workspace = true
12+
13+
[lib]
14+
name = "openshell_driver_native"
15+
path = "src/lib.rs"
16+
17+
[[bin]]
18+
name = "openshell-driver-native"
19+
path = "src/main.rs"
20+
21+
[dependencies]
22+
openshell-core = { path = "../openshell-core", default-features = false }
23+
openshell-nft-ruleset = { path = "../openshell-nft-ruleset" }
24+
openshell-policy = { path = "../openshell-policy" }
25+
26+
# containerd's own generated gRPC client. This is the "system-provided
27+
# containerd" integration point, used only for image pull/unpack and
28+
# snapshot management (see src/image.rs) -- never bundled, installed, or
29+
# managed by this driver. The driver itself, not containerd, spawns the
30+
# configured low-level OCI runtime (runc/crun/etc.) directly (see
31+
# src/runtime.rs); containerd's container/task services and shim are
32+
# never used.
33+
containerd-client = "0.9"
34+
# OCI runtime-spec and image-spec types (namespaces, cgroups v2 resources,
35+
# seccomp, image manifest/config parsing for chain-ID computation).
36+
oci-spec = { version = "0.10", default-features = false, features = [
37+
"runtime",
38+
"image",
39+
] }
40+
41+
tokio = { workspace = true }
42+
tokio-stream = { workspace = true }
43+
tonic = { workspace = true, features = ["transport"] }
44+
prost = { workspace = true }
45+
prost-types = { workspace = true }
46+
futures = { workspace = true }
47+
clap = { workspace = true }
48+
tracing = { workspace = true }
49+
tracing-subscriber = { workspace = true }
50+
miette = { workspace = true }
51+
serde = { workspace = true }
52+
serde_json = { workspace = true }
53+
nix = { workspace = true }
54+
rand = { workspace = true }
55+
thiserror = { workspace = true }
56+
sha2 = "0.10"
57+
58+
[dev-dependencies]
59+
tempfile = "3"
60+
61+
[lints]
62+
workspace = true

0 commit comments

Comments
 (0)