Skip to content

Commit bb6d5cb

Browse files
committed
feat(docker): supply CDI context for GPU sandboxes
Signed-off-by: Evan Lezar <elezar@nvidia.com>
1 parent 0c7cd4a commit bb6d5cb

8 files changed

Lines changed: 470 additions & 127 deletions

File tree

architecture/compute-runtimes.md

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -169,6 +169,17 @@ Resource requirements enter the driver layer through `SandboxSpec.resource_requi
169169
can request a specific number of GPUs or the driver-specific default behaviour.
170170
For all in-tree drivers, this is equivalent to selecting a single GPU.
171171

172+
For Docker GPU sandboxes, the driver treats CDI specs as runtime metadata for
173+
both outer injection and inner sandbox policy. It selects opaque CDI device IDs,
174+
passes them to Docker, mounts daemon-reported CDI spec directories into
175+
supervisor-only paths, and uploads a versioned CDI context before starting the
176+
container. The supervisor resolves that context inside the sandbox and derives
177+
Landlock paths and supplemental groups from CDI `containerEdits`. Host-side CDI
178+
spec paths are diagnostic only and are never treated as sandbox policy paths.
179+
Kubernetes must not infer CDI device IDs from the `nvidia.com/gpu` resource
180+
request; it needs a node-local selected-device handoff before using the same
181+
supervisor resolver.
182+
172183
VM runtime state paths are derived only from driver-validated sandbox IDs
173184
matching `[A-Za-z0-9._-]{1,128}`. The gateway-owned VM driver socket uses a
174185
private `run/` directory plus Unix peer UID/PID checks. Standalone

crates/openshell-core/src/cdi.rs

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,12 +9,21 @@ use serde::{Deserialize, Serialize};
99

1010
pub const CDI_CONTEXT_VERSION: u32 = 1;
1111

12+
/// File name used for the serialized CDI context.
13+
pub const CDI_CONTEXT_FILE_NAME: &str = "cdi-context.json";
14+
1215
/// Absolute supervisor path for the CDI context file mounted by a compute driver.
1316
pub const CDI_CONTEXT_PATH: &str = "/run/openshell/supervisor/cdi-context.json";
1417

1518
/// Base supervisor path under which compute drivers mount CDI specification directories.
1619
pub const CDI_SPEC_DIR_BASE: &str = "/run/openshell/supervisor/cdi-specs";
1720

21+
/// Return the supervisor path used for a CDI specification directory.
22+
#[must_use]
23+
pub fn cdi_spec_mount_path(index: usize) -> String {
24+
format!("{CDI_SPEC_DIR_BASE}/{index}")
25+
}
26+
1827
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
1928
pub struct CdiContext {
2029
pub version: u32,

crates/openshell-driver-docker/Cargo.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,10 +24,10 @@ serde_json = { workspace = true }
2424
prost-types = { workspace = true }
2525
bollard = { version = "0.20" }
2626
url = { workspace = true }
27+
tar = "0.4"
2728

2829
[dev-dependencies]
2930
prost-types = { workspace = true }
30-
tar = "0.4"
3131
temp-env = "0.3"
3232
tempfile = "3"
3333

crates/openshell-driver-docker/README.md

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -86,9 +86,33 @@ contract:
8686
| `restart_policy = unless-stopped` | Keeps managed sandboxes resumable across daemon or gateway restarts. |
8787
| `PidsLimit` | Enforces the sandbox PID budget at the Docker cgroup layer. Set `[openshell.drivers.docker].sandbox_pids_limit = 0` to inherit the Docker/runtime default. |
8888
| CDI GPU request | Uses opaque `driver_config.cdi_devices` values when set; otherwise selects the requested count of NVIDIA CDI GPUs in round-robin order when daemon CDI support is detected. Docker daemon `/info` can permit `nvidia.com/gpu=all` as a WSL2 all-only compatibility fallback, where it counts as one selectable device. Exact CDI device lists must not contain duplicates and must match the effective GPU count. |
89+
| CDI context upload | For GPU/CDI sandboxes only, mounts daemon-reported CDI spec directories read-only under `/run/openshell/supervisor/cdi-specs/<n>` and uploads `/run/openshell/supervisor/cdi-context.json` after container create and before start. |
8990

9091
The agent child process does not retain these supervisor privileges.
9192

93+
## CDI GPU Metadata
94+
95+
Docker remains the source of truth for GPU injection. The driver selects opaque
96+
CDI device IDs from `driver_config.cdi_devices` or the daemon's discovered CDI
97+
inventory, then passes the same IDs to Docker with a CDI `DeviceRequest`.
98+
99+
When a GPU/CDI request is present, the driver also mounts the Docker
100+
daemon-reported `Info.CDISpecDirs` into supervisor-only paths and uploads a
101+
small versioned CDI context through Docker's container archive API. The context
102+
uses container-side spec paths for resolution and keeps host-side spec sources
103+
diagnostic-only. If the upload fails, the driver removes the created container
104+
and sandbox token file before reporting the failure.
105+
106+
The sandbox supervisor resolves the selected IDs from those mounted specs
107+
before it launches agent processes. CDI device nodes become read-write
108+
Landlock paths, mount destinations default to read-only paths, and
109+
`additionalGids` become supplemental groups for the entrypoint and SSH child
110+
processes. Writable CDI mount destinations are accepted only for exact
111+
single-file paths already listed in the sandbox policy `read_write` list;
112+
writable CDI directory mounts fail closed. Kubernetes, Podman, WSL2 hardware
113+
validation, and Tegra/Jetson hardware validation are separate follow-up
114+
targets.
115+
92116
## Driver Config Mounts
93117

94118
The gateway forwards the `docker` block from `--driver-config-json` to this

0 commit comments

Comments
 (0)