fix(driver-native): spawn the low-level runtime directly instead of via containerd

Corrects an architectural mistake from the previous commit: containerd's
Containers/Tasks services (and therefore containerd's own
io.containerd.runc.v2 shim) were spawning runc/crun, not OpenShell. This
put containerd in the process-supervision path for the sandbox's root
process, which is not what "OpenShell drives the runtime, containerd is
just used elsewhere" was supposed to mean.

This driver -- not containerd -- now spawns the configured low-level OCI
runtime (runc by default, crun, or any other OCI-runtime-spec-compatible
binary) directly:

- New runtime.rs builds a standard OCI bundle (config.json + a mounted
  rootfs/) from the snapshot containerd prepared, and drives it through
  the runtime's own create/start/state/kill/delete CLI contract via
  std::process::Command -- the same integration pattern containerd's
  shim, CRI-O, and Podman use, just invoked by this driver's process
  instead of containerd's.
- Removed runc_options.rs entirely: it existed only to wrap
  runtime_binary as a containerd.runc.v1.Options message for containerd's
  shim to read. With the shim no longer in the picture, selecting runc vs.
  crun is now simply which binary this driver execs.
- containerd is now used only for image pull/unpack (Transfer/Images/
  Content) and snapshot management (Snapshots.Prepare) -- image.rs no
  longer creates a containerd Container or Task at all.
- Because no containerd Container/Task references the prepared snapshot
  anymore, it has nothing protecting it from containerd's background GC.
  Verified this is a real problem (an otherwise-identical run without
  protection failed at teardown with "snapshot ... does not exist"
  because GC had already reaped it) and fixed it with containerd's Leases
  service: image.rs now creates a lease per sandbox and attaches the
  snapshot to it, deleting the lease at teardown.
- Sandbox tracking (get/list) no longer queries containerd (there is
  nothing left to query); it scans this driver's own state directory for
  bundle directories, the same pattern the VM driver uses to rediscover
  sandboxes on restart, and calls `runtime state` directly for status.

Re-verified end to end against a real containerd 2.x + runc/crun install:
pull -> chain-ID resolve -> lease-protected snapshot prepare -> bundle
mount -> create/start/state/delete directly against both runc and crun
(new second integration test) -> stop -> delete, plus confirming a
snapshot without a lease is reaped by GC while one with a lease survives.
The full openshell-gateway binary was also re-run with --drivers native
against live containerd to confirm the wiring still works.

Docs, the driver README, and inline comments are updated to describe the
corrected division of labor between this driver and containerd.

Signed-off-by: Eric Curtin <eric.curtin@docker.com>

Author: ericcurtin

architecture/compute-runtimes.md

@@ -34,7 +34,7 @@ when a sandbox create request asks for GPU resources.
 | Podman | Rootless or single-machine deployments. | Container plus nested sandbox namespace. | Uses the Podman REST API, OCI image volumes, and CDI GPU devices when available. |
 | Kubernetes | Cluster deployment through Helm. | Pod plus nested sandbox namespace. | Uses Kubernetes API objects, service accounts, secrets, PVC-backed workspace storage, and GPU resources. |
 | VM | Experimental microVM isolation. | Per-sandbox libkrun VM. | Managed endpoint-backed driver. The gateway spawns `openshell-driver-vm`, waits for its Unix socket, and then consumes it through the same remote `compute_driver.proto` path used by unmanaged endpoint drivers. The VM driver boots a cached bootstrap `rootfs.ext4`, prepares requested OCI images inside a bootstrap VM with `umoci`, attaches the prepared image disk read-only, and gives each sandbox a writable `overlay.ext4` for merged-root changes and runtime material. The driver persists each accepted launch request beside the overlay and restarts those VMs on driver startup without recreating the overlay. |
-| Native | Early/opt-in kernel-primitive isolation against an existing system `containerd`. | Container built from Linux namespaces + cgroups v2. | In-process driver, like Docker/Podman/Kubernetes. Drives a system-provided `containerd` over its own gRPC API — never bundles containerd or a low-level OCI runtime; containerd invokes the configured `runtime_binary` (`runc` by default, `crun`, or another OCI-runtime-spec-compatible binary) through its own shim. Per-sandbox network namespace + veth pair + nftables ruleset (shared `openshell-nft-ruleset` crate, also used by the VM driver). GPU support is kernel-device-node passthrough only. Rootless mode is implemented but not yet functional (defaults off) — see `crates/openshell-driver-native/README.md`. |
+| Native | Early/opt-in kernel-primitive isolation against an existing system `containerd`. | Container built from Linux namespaces + cgroups v2. | In-process driver, like Docker/Podman/Kubernetes. The driver itself — not containerd — spawns the configured `runtime_binary` (`runc` by default, `crun`, or another OCI-runtime-spec-compatible binary) directly through its `create`/`start`/`state`/`delete` CLI contract; containerd is used only for image pull/unpack and snapshot management (protected from containerd's GC via a lease, since no containerd `Container`/`Task` is ever created) and never bundled. Sandboxes are tracked by scanning the driver's own state directory rather than querying containerd or the runtime for a global list. Per-sandbox network namespace + veth pair + nftables ruleset (shared `openshell-nft-ruleset` crate, also used by the VM driver). GPU support is kernel-device-node passthrough only. Rootless mode is implemented but not yet functional (defaults off) — see `crates/openshell-driver-native/README.md`. |
 | Extension | Out-of-tree drivers operated alongside the gateway. | Whatever boundary the driver implements. | Selected by a non-reserved custom `compute_drivers = ["<name>"]` entry with `[openshell.drivers.<name>].socket_path`, or at launch time by pairing `--drivers <name>` with `--compute-driver-socket=<path>`. Reserved built-in names such as `vm`, `docker`, `podman`, `kubernetes`, and `native` cannot be used as unmanaged socket endpoints. The gateway connects to a UDS the operator already provisioned, runs `GetCapabilities`, logs the advertised `driver_name`, and dispatches all sandbox lifecycle calls through `compute_driver.proto`. The driver process and socket lifecycle are operator-owned; the gateway does not spawn, supervise, or remove unmanaged extension drivers. The trust boundary is the socket's filesystem permissions: the operator must ensure only the gateway uid can read/write it. |
 
 Per-sandbox CPU and memory values currently enter the driver layer through

crates/openshell-driver-native/Cargo.toml

@@ -24,10 +24,12 @@ openshell-nft-ruleset = { path = "../openshell-nft-ruleset" }
 openshell-policy = { path = "../openshell-policy" }
 
 # containerd's own generated gRPC client. This is the "system-provided
-# containerd" integration point: the driver never bundles or drives a
-# low-level OCI runtime (runc/crun/etc.) itself. It talks to a
-# system-installed containerd over its UDS gRPC API, and containerd invokes
-# the configured low-level runtime via its own shim.
+# containerd" integration point, used only for image pull/unpack and
+# snapshot management (see src/image.rs) -- never bundled, installed, or
+# managed by this driver. The driver itself, not containerd, spawns the
+# configured low-level OCI runtime (runc/crun/etc.) directly (see
+# src/runtime.rs); containerd's container/task services and shim are
+# never used.
 containerd-client = "0.9"
 # OCI runtime-spec and image-spec types (namespaces, cgroups v2 resources,
 # seccomp, image manifest/config parsing for chain-ID computation).

crates/openshell-driver-native/README.md

@@ -2,14 +2,21 @@
 
 The native compute driver builds sandbox containers directly from Linux kernel
 primitives (namespaces, cgroups v2) by driving a **system-provided
-`containerd`** rather than a container engine daemon (Docker/Podman) or a
-hypervisor. It runs in-process within the gateway server, the same way the
-Docker, Podman, and Kubernetes drivers do.
-
-This driver deliberately does not bundle, install, or manage `containerd`
-itself, and it never shells out to a low-level OCI runtime (`runc`, `crun`,
-...) directly — containerd does that, through its own shim, using whichever
-`runtime_binary` is configured. See
+`containerd`** for image management and directly invoking a **configurable
+low-level OCI runtime** (`runc` by default, `crun`, or any other
+OCI-runtime-spec-compatible binary already installed on the host) itself. It
+runs in-process within the gateway server, the same way the Docker, Podman,
+and Kubernetes drivers do.
+
+**This driver — not containerd — spawns the sandbox's process.** containerd
+is used only for image pull/unpack and snapshot management; it never creates
+a containerd `Container` or `Task` object, and its shim
+(`io.containerd.runc.v2`) is never involved. This driver builds a standard
+OCI bundle (`config.json` + a mounted `rootfs/`) from the snapshot containerd
+prepared and drives the configured runtime directly through its own
+`create`/`start`/`state`/`kill`/`delete` CLI contract. It never bundles,
+installs, or manages `containerd` itself, and it never bundles the low-level
+runtime either — both must already be present on the host. See
 [`docs/reference/sandbox-compute-drivers.mdx`](../../docs/reference/sandbox-compute-drivers.mdx#native-driver)
 for the user-facing reference and known gaps.
 
@@ -19,8 +26,8 @@ for the user-facing reference and known gaps.
 graph TB
     CLI["openshell CLI"] -->|gRPC| GW["Gateway Server<br/>(openshell-server)"]
     GW -->|in-process| ND["NativeComputeDriver"]
-    ND -->|gRPC<br/>Unix socket| CD["containerd<br/>(system-provided)"]
-    CD -->|shim: io.containerd.runc.v2<br/>binary_name = runtime_binary| RT["Low-level OCI runtime<br/>runc / crun / ..."]
+    ND -->|gRPC<br/>Unix socket, image/snapshot/lease only| CD["containerd<br/>(system-provided)"]
+    ND -->|exec: create/start/state/delete| RT["Low-level OCI runtime<br/>runc / crun / ..."]
     RT -->|creates| C["Sandbox Container"]
     C -->|bind mount<br/>read-only| SV["Supervisor Binary<br/>/opt/openshell/bin/openshell-sandbox"]
     ND -->|ip netns / veth / nft| NET["Per-sandbox network namespace"]
@@ -34,10 +41,13 @@ graph TB
 |---|---|
 | Registry pull, layer download, unpack | containerd's `Transfer` + `Images` + `Content` services (`image.rs`) |
 | Writable per-sandbox rootfs layer | containerd's `Snapshots` service (`Prepare`) |
+| Protecting that snapshot from containerd's background GC | containerd's `Leases` service (`image.rs`) — necessary precisely because this driver never registers a `Container`/`Task` that would otherwise reference it |
 | OCI runtime spec (namespaces, cgroups v2, capabilities, seccomp defaults) | This crate (`spec.rs`), generated from OpenShell policy |
-| Namespace/cgroup/seccomp *enforcement* | The configured low-level runtime (`runc`/`crun`), invoked by containerd's shim — never this crate directly |
+| Bundle assembly (mounting the snapshot, writing `config.json`) | This crate (`runtime.rs`) |
+| Namespace/cgroup/seccomp *enforcement*, and the actual `create`/`start`/`state`/`kill`/`delete` process invocations | This crate (`runtime.rs`), execing the configured low-level runtime directly — containerd is never involved |
 | Network namespace, veth pair, nftables ruleset | This crate (`network.rs`), reusing `openshell-nft-ruleset` |
 | GPU device nodes | This crate (`gpu.rs`), kernel device-node passthrough only (see below) |
+| Tracking which sandboxes exist | This crate's own state directory (`state_dir/bundles/<sandbox>/`), scanned directly — containerd has no record of these containers at all |
 
 ## Isolation Model
 
@@ -94,17 +104,40 @@ resolved the same way as the Docker/Podman drivers.
   subscribing to containerd's own event stream. Functionally correct,
   higher-latency than a push subscription would be.
 - **CDI GPU injection is device-node-only** (see above).
+- **Sandbox discovery is a state-directory scan**, not a containerd query
+  (there is nothing for containerd to query — see the table above). A
+  `state_dir` shared or manually edited by something other than this
+  driver process could produce inconsistent `list_sandboxes` results;
+  this matches the trust model the VM driver already uses for its own
+  state directory.
+
+## containerd Leases (why they exist here)
+
+Every other in-tree driver that talks to a container engine registers a
+persistent object (a Docker/Podman container, a containerd `Container`) that
+the engine itself uses to know a resource is still wanted. This driver
+deliberately never creates that containerd-side object — see "What this
+crate owns vs. what containerd owns" above — which means the writable
+snapshot it prepares has **nothing** protecting it from containerd's
+background garbage collector by default. This was not a theoretical
+concern: during development, an otherwise-identical flow without a lease
+failed at teardown with `snapshot ... does not exist` because GC had
+already reaped it. `image.rs`'s `create_lease`/`protect_snapshot_with_lease`/
+`delete_lease` functions close that gap by attaching the snapshot to a
+lease for the sandbox's lifetime.
 
 ## Testing
 
 Unit tests (`cargo test -p openshell-driver-native --lib`) cover OCI spec
 generation, chain-ID computation, nftables ruleset generation, GPU device
-resolution, and config validation — all pure/deterministic, no containerd
-required.
+resolution, bundle/state-directory bookkeeping, and config validation — all
+pure/deterministic, no containerd required.
 
 `tests/containerd_integration.rs` is a real end-to-end test against a live
-containerd (pull → chain-ID resolve → snapshot prepare → container/task
-create → start → stop → delete, plus network namespace isolation). It is
-`#[ignore]`d by default (requires a reachable containerd socket, `runc`,
-`CAP_NET_ADMIN`, and network access) — see the module doc comment in that
-file for how to run it.
+containerd (pull → chain-ID resolve → lease-protected snapshot prepare →
+bundle mount → `create`/`start`/`state`/`delete` directly against the
+low-level runtime → stop → delete, plus network namespace isolation). It
+runs the full round trip against both `runc` and `crun` to prove the
+configurable-runtime design point end to end. It is `#[ignore]`d by default
+(requires a reachable containerd socket, the configured runtime, `CAP_NET_ADMIN`,
+and network access) — see the module doc comment in that file for how to run it.

crates/openshell-driver-native/src/config.rs

@@ -14,9 +14,10 @@ pub const DEFAULT_CONTAINERD_SOCKET_PATH: &str = "/run/containerd/containerd.soc
 pub const DEFAULT_CONTAINERD_NAMESPACE: &str = "openshell";
 /// Default low-level OCI runtime.
 ///
-/// Configurable (e.g. `crun`) — see [`crate::runc_options::RuncOptions`].
-/// Never bundled: this must already be installed and resolvable through
-/// containerd's shim on the host.
+/// Configurable (e.g. `crun`, or any other OCI-runtime-spec-compatible
+/// binary). Never bundled: this driver execs it directly (see
+/// [`crate::runtime`]), so it must already be installed and resolvable on
+/// the gateway host. containerd is never involved in invoking it.
 pub const DEFAULT_RUNTIME_BINARY: &str = "runc";
 /// Default containerd snapshotter.
 pub const DEFAULT_SNAPSHOTTER: &str = "overlayfs";
@@ -43,10 +44,11 @@ pub struct NativeComputeConfig {
     pub containerd_socket_path: PathBuf,
     /// containerd namespace this driver operates in.
     pub containerd_namespace: String,
-    /// Low-level OCI runtime binary name (or absolute path) containerd's
-    /// shim should exec. Default `runc`; set to `crun` or another
-    /// OCI-runtime-spec-compatible binary already installed on the host.
-    /// This driver never bundles or invokes this binary itself.
+    /// Low-level OCI runtime binary name (or absolute path) this driver
+    /// execs directly (see [`crate::runtime`]). Default `runc`; set to
+    /// `crun` or another OCI-runtime-spec-compatible binary already
+    /// installed on the host. Never bundled: containerd is never involved
+    /// in invoking it.
     pub runtime_binary: String,
     /// containerd snapshotter used for image unpack and per-sandbox
     /// writable layers.
@@ -85,8 +87,8 @@ pub struct NativeComputeConfig {
     /// (UID/GID 0) to an unprivileged host UID/GID range. This is the
     /// mechanism behind the issue's "rootless by default" goal: the
     /// sandboxed process would run as root only from its own point of
-    /// view, never as host root, even though the host containerd/runc
-    /// invocation itself is not rootless.
+    /// view, never as host root, even though the runtime invocation
+    /// this driver performs itself is not rootless.
     ///
     /// **Known gap, defaults to `false` until fixed:** verified against a
     /// real containerd + runc during development, enabling this currently