Skip to content
Draft
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 6 additions & 0 deletions architecture/compute-runtimes.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,12 @@ when a sandbox create request asks for GPU resources.

Per-sandbox CPU and memory values currently enter the driver layer through
template resource limits. Docker and Podman apply them as runtime limits.

Docker and Podman operators can configure `sandbox_uid` and `sandbox_gid` in
their driver tables. When unset, the drivers resolve `10001:10001` and inject
it for the agent child; the root supervisor remains the container entry process.
Rootless Podman requires that the chosen values are available in its subordinate
UID/GID mappings.
Kubernetes mirrors each limit into the matching request. VM accepts the fields
but currently ignores them.

Expand Down
7 changes: 7 additions & 0 deletions architecture/sandbox.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,13 @@ container-granted capabilities. This is fail-closed: the supervisor retains
aborts unless the bounding set ends up empty. A `setpcap` `EPERM` is tolerated
only when the set is already empty; any other outcome fails the spawn.

The compute runtime selects the agent's numeric UID and GID; sandbox creators
do not control it. Docker and Podman start the supervisor as root, inject the
driver-resolved identity, and the supervisor drops only the agent child to that
identity. A container image therefore need not contain a matching passwd or
group entry. Agent children receive a stable `HOME=/sandbox`; numeric identities
use the UID as their `USER` and `LOGNAME` presentation value.

## Startup Flow

1. The compute runtime starts the workload with sandbox identity, callback
Expand Down
2 changes: 1 addition & 1 deletion crates/openshell-core/src/sandbox_env.rs
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,7 @@ pub const PROVIDER_SPIFFE_WORKLOAD_API_SOCKET: &str =
/// Resolved sandbox UID used to override `run_as_user` when the policy
/// specifies a numeric value instead of the hardcoded "sandbox" user name.
///
/// Set by compute drivers (Kubernetes, Docker, VM) from resolved config or
/// Set by compute drivers (Kubernetes, Docker, Podman, and VM) from resolved config or
/// cluster autodetection. The supervisor reads this at startup and uses it
/// directly with `setuid()` / `chown()` without requiring an `/etc/passwd`
/// entry in the sandbox image.
Expand Down
1 change: 1 addition & 0 deletions crates/openshell-driver-docker/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ repository.workspace = true

[dependencies]
openshell-core = { path = "../openshell-core", default-features = false }
openshell-policy = { path = "../openshell-policy" }

tokio = { workspace = true }
tonic = { workspace = true }
Expand Down
1 change: 1 addition & 0 deletions crates/openshell-driver-docker/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ contract:
| `apparmor=unconfined` | Avoids Docker's default profile blocking required mount operations. |
| `restart_policy = unless-stopped` | Keeps managed sandboxes resumable across daemon or gateway restarts. |
| `PidsLimit` | Enforces the sandbox PID budget at the Docker cgroup layer. Set `[openshell.drivers.docker].sandbox_pids_limit = 0` to inherit the Docker/runtime default. |
| `sandbox_uid` / `sandbox_gid` | Operator-controlled numeric identity for the agent child. The supervisor still starts as root; the UID defaults to `10001` and the GID defaults to the resolved UID. |
| CDI GPU request | Uses opaque `driver_config.cdi_devices` values when set; otherwise selects the requested count of NVIDIA CDI GPUs in round-robin order when daemon CDI support is detected. Docker daemon `/info` can permit `nvidia.com/gpu=all` as a WSL2 all-only compatibility fallback, where it counts as one selectable device. Exact CDI device lists must not contain duplicates and must match the effective GPU count. |

The agent child process does not retain these supervisor privileges.
Expand Down
40 changes: 40 additions & 0 deletions crates/openshell-driver-docker/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,7 @@ const SUPERVISOR_PATH: &str = "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
const HOST_OPENSHELL_INTERNAL: &str = "host.openshell.internal";
const HOST_DOCKER_INTERNAL: &str = "host.docker.internal";
const DOCKER_NETWORK_DRIVER: &str = "bridge";
const DEFAULT_SANDBOX_UID: u32 = 10_001;

/// Queried by the Docker driver to decide when a sandbox's supervisor
/// relay is live. Implementations return `true` once a sandbox has an
Expand Down Expand Up @@ -136,6 +137,14 @@ pub struct DockerComputeConfig {
/// Set to `0` to leave Docker's runtime/default PID limit unchanged.
pub sandbox_pids_limit: i64,

/// Numeric identity used for the agent process. The root supervisor
/// retains UID 0 while preparing the sandbox.
pub sandbox_uid: Option<u32>,

/// Numeric group used for the agent process. Defaults to the resolved
/// sandbox UID when unset.
pub sandbox_gid: Option<u32>,

/// Allow sandbox requests to attach host bind mounts through
/// `template.driver_config`.
#[serde(default)]
Expand All @@ -158,6 +167,8 @@ impl Default for DockerComputeConfig {
host_gateway_ip: String::new(),
ssh_socket_path: "/run/openshell/ssh.sock".to_string(),
sandbox_pids_limit: DEFAULT_SANDBOX_PIDS_LIMIT,
sandbox_uid: None,
sandbox_gid: None,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

QUestion: Why is the default not DEFAULT_SANDBOX_UID?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the idea was to keep all of the resolution within resolve_sandbox_entity, versus defaulting in multiple locations. But I do think that's a bit confusing, I think it's worth updating here IMO

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated this so the Docker and Podman configs now use DEFAULT_SANDBOX_UID as a concrete UID default. We keep sandbox_gid optional so that, when it is omitted, it follows the selected UID (for example, a UID-only override of 12345 resolves to 12345:12345). That matches the effective UID/GID fallback behavior in the Kubernetes and VM drivers.

enable_bind_mounts: false,
}
}
Expand Down Expand Up @@ -187,6 +198,8 @@ struct DockerDriverRuntimeConfig {
supports_gpu: bool,
allow_all_default_gpu: bool,
sandbox_pids_limit: i64,
sandbox_uid: u32,
sandbox_gid: u32,
enable_bind_mounts: bool,
}

Expand Down Expand Up @@ -307,6 +320,7 @@ impl DockerComputeDriver {
docker_config: &DockerComputeConfig,
supervisor_readiness: Arc<dyn SupervisorReadiness>,
) -> CoreResult<Self> {
let sandbox_identity = resolve_sandbox_identity(docker_config)?;
let docker = Docker::connect_with_local_defaults()
.map_err(|err| Error::execution(format!("failed to create Docker client: {err}")))?;
let version = docker.version().await.map_err(|err| {
Expand Down Expand Up @@ -370,6 +384,8 @@ impl DockerComputeDriver {
supports_gpu,
allow_all_default_gpu,
sandbox_pids_limit: docker_config.sandbox_pids_limit,
sandbox_uid: sandbox_identity.0,
sandbox_gid: sandbox_identity.1,
enable_bind_mounts: docker_config.enable_bind_mounts,
},
events: broadcast::channel(WATCH_BUFFER).0,
Expand Down Expand Up @@ -2162,6 +2178,14 @@ fn build_environment(sandbox: &DriverSandbox, config: &DockerDriverRuntimeConfig
openshell_core::sandbox_env::TELEMETRY_ENABLED.to_string(),
openshell_core::telemetry::enabled_env_value().to_string(),
);
environment.insert(
openshell_core::sandbox_env::SANDBOX_UID.to_string(),
config.sandbox_uid.to_string(),
);
environment.insert(
openshell_core::sandbox_env::SANDBOX_GID.to_string(),
config.sandbox_gid.to_string(),
);
// The root supervisor executes namespace helpers during bootstrap; keep
// their search path driver-owned even when the template/spec set PATH.
environment.insert("PATH".to_string(), SUPERVISOR_PATH.to_string());
Expand Down Expand Up @@ -2619,6 +2643,22 @@ fn validate_sandbox_pids_limit(value: i64) -> CoreResult<()> {
Ok(())
}

fn resolve_sandbox_identity(config: &DockerComputeConfig) -> CoreResult<(u32, u32)> {
let uid = config.sandbox_uid.unwrap_or(DEFAULT_SANDBOX_UID);
let gid = config.sandbox_gid.unwrap_or(uid);
for (field, value) in [("sandbox_uid", uid), ("sandbox_gid", gid)] {
if !(openshell_policy::MIN_SANDBOX_UID..=openshell_policy::MAX_SANDBOX_UID).contains(&value)
{
return Err(Error::config(format!(
"docker {field} must be in [{}, {}]",
openshell_policy::MIN_SANDBOX_UID,
openshell_policy::MAX_SANDBOX_UID,
)));
}
}
Ok((uid, gid))
}

fn docker_pids_limit(value: i64) -> Result<Option<i64>, Status> {
if value < 0 {
return Err(Status::failed_precondition(
Expand Down
68 changes: 68 additions & 0 deletions crates/openshell-driver-docker/src/tests.rs
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,8 @@ fn runtime_config() -> DockerDriverRuntimeConfig {
supports_gpu: false,
allow_all_default_gpu: false,
sandbox_pids_limit: DEFAULT_SANDBOX_PIDS_LIMIT,
sandbox_uid: 10_001,
sandbox_gid: 10_001,
enable_bind_mounts: false,
}
}
Expand Down Expand Up @@ -551,6 +553,50 @@ fn docker_compute_config_disables_bind_mounts_by_default() {
assert!(!cfg.enable_bind_mounts);
}

#[test]
fn docker_sandbox_identity_defaults_to_10001() {
assert_eq!(
resolve_sandbox_identity(&DockerComputeConfig::default()).unwrap(),
(10_001, 10_001)
);
}

#[test]
fn docker_sandbox_identity_uses_resolved_uid_for_default_gid() {
let config = DockerComputeConfig {
sandbox_uid: Some(12_345),
..DockerComputeConfig::default()
};
assert_eq!(resolve_sandbox_identity(&config).unwrap(), (12_345, 12_345));
}

#[test]
fn docker_sandbox_identity_allows_configured_gid_without_uid() {
let config = DockerComputeConfig {
sandbox_gid: Some(12_346),
..DockerComputeConfig::default()
};
assert_eq!(resolve_sandbox_identity(&config).unwrap(), (10_001, 12_346));
}

#[test]
fn docker_sandbox_identity_rejects_system_uid() {
let config = DockerComputeConfig {
sandbox_uid: Some(999),
..DockerComputeConfig::default()
};
assert!(resolve_sandbox_identity(&config).is_err());
}

#[test]
fn docker_sandbox_identity_rejects_gid_above_policy_range() {
let config = DockerComputeConfig {
sandbox_gid: Some(openshell_policy::MAX_SANDBOX_UID + 1),
..DockerComputeConfig::default()
};
assert!(resolve_sandbox_identity(&config).is_err());
}

#[test]
fn container_create_body_sets_driver_owned_pids_limit() {
let body = build_container_create_body(&test_sandbox(), &runtime_config()).unwrap();
Expand All @@ -567,6 +613,28 @@ fn build_environment_sets_docker_tls_paths() {
assert!(env.contains(&"TEMPLATE_ENV=template".to_string()));
assert!(env.contains(&"SPEC_ENV=spec".to_string()));
assert!(env.contains(&"OPENSHELL_SANDBOX_COMMAND=sleep infinity".to_string()));
assert!(env.contains(&"OPENSHELL_SANDBOX_UID=10001".to_string()));
assert!(env.contains(&"OPENSHELL_SANDBOX_GID=10001".to_string()));
}

#[test]
fn build_environment_keeps_sandbox_identity_driver_controlled() {
let mut sandbox = test_sandbox();
let spec = sandbox.spec.as_mut().unwrap();
spec.environment.insert(
openshell_core::sandbox_env::SANDBOX_UID.to_string(),
"1234".to_string(),
);
spec.template.as_mut().unwrap().environment.insert(
openshell_core::sandbox_env::SANDBOX_GID.to_string(),
"1234".to_string(),
);

let env = build_environment(&sandbox, &runtime_config());
assert!(env.contains(&"OPENSHELL_SANDBOX_UID=10001".to_string()));
assert!(env.contains(&"OPENSHELL_SANDBOX_GID=10001".to_string()));
assert!(!env.contains(&"OPENSHELL_SANDBOX_UID=1234".to_string()));
assert!(!env.contains(&"OPENSHELL_SANDBOX_GID=1234".to_string()));
}

#[test]
Expand Down
1 change: 1 addition & 0 deletions crates/openshell-driver-podman/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ path = "src/main.rs"

[dependencies]
openshell-core = { path = "../openshell-core", default-features = false }
openshell-policy = { path = "../openshell-policy" }

tokio = { workspace = true }
tonic = { workspace = true, features = ["transport"] }
Expand Down
6 changes: 6 additions & 0 deletions crates/openshell-driver-podman/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,12 @@ The container spec in `container.rs` sets these security-critical fields:

The restricted agent child does not retain these supervisor privileges.

The gateway operator may set `[openshell.drivers.podman].sandbox_uid` and
`sandbox_gid` for the agent child. The supervisor remains `0:0` during setup;
the UID defaults to `10001` and the GID defaults to the resolved UID. In
rootless mode, configure values that are available in Podman's subordinate
UID/GID mapping.

## Driver Config Mounts

The gateway forwards the `podman` block from `--driver-config-json` to this
Expand Down
74 changes: 74 additions & 0 deletions crates/openshell-driver-podman/src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ use std::str::FromStr;
/// Default Podman bridge network name.
pub const DEFAULT_NETWORK_NAME: &str = "openshell";
pub const MACOS_PODMAN_MACHINE_HOST_GATEWAY_IP: &str = "192.168.127.254";
pub const DEFAULT_SANDBOX_UID: u32 = 10_001;

// Re-export the shared default so existing imports inside this crate keep working.
pub use openshell_core::config::DEFAULT_SANDBOX_PIDS_LIMIT;
Expand Down Expand Up @@ -122,6 +123,11 @@ pub struct PodmanComputeConfig {
///
/// Set to `0` to leave Podman's runtime/default PID limit unchanged.
pub sandbox_pids_limit: i64,
/// Numeric identity used for the agent process. The supervisor remains
/// root while it prepares the sandbox.
pub sandbox_uid: Option<u32>,
/// Numeric group used for the agent process. Defaults to the resolved UID.
pub sandbox_gid: Option<u32>,
/// Allow sandbox requests to attach host bind mounts through
/// `template.driver_config`.
#[serde(default)]
Expand Down Expand Up @@ -189,6 +195,24 @@ impl PodmanComputeConfig {
Ok(())
}

/// Resolve and validate the numeric identity injected into the supervisor.
pub fn resolve_sandbox_identity(&self) -> Result<(u32, u32), crate::client::PodmanApiError> {
let uid = self.sandbox_uid.unwrap_or(DEFAULT_SANDBOX_UID);
let gid = self.sandbox_gid.unwrap_or(uid);
for (field, value) in [("sandbox_uid", uid), ("sandbox_gid", gid)] {
if !(openshell_policy::MIN_SANDBOX_UID..=openshell_policy::MAX_SANDBOX_UID)
.contains(&value)
{
return Err(crate::client::PodmanApiError::InvalidInput(format!(
"{field} must be in [{}, {}]",
openshell_policy::MIN_SANDBOX_UID,
openshell_policy::MAX_SANDBOX_UID,
)));
}
}
Ok((uid, gid))
}

/// Validate optional host gateway override.
pub fn validate_host_gateway_ip(&self) -> Result<(), crate::client::PodmanApiError> {
let trimmed = self.host_gateway_ip.trim();
Expand Down Expand Up @@ -260,6 +284,8 @@ impl Default for PodmanComputeConfig {
guest_tls_cert: None,
guest_tls_key: None,
sandbox_pids_limit: DEFAULT_SANDBOX_PIDS_LIMIT,
sandbox_uid: None,
sandbox_gid: None,
enable_bind_mounts: false,
health_check_interval_secs: DEFAULT_HEALTH_CHECK_INTERVAL_SECS,
}
Expand All @@ -283,6 +309,8 @@ impl std::fmt::Debug for PodmanComputeConfig {
.field("guest_tls_cert", &self.guest_tls_cert)
.field("guest_tls_key", &self.guest_tls_key)
.field("sandbox_pids_limit", &self.sandbox_pids_limit)
.field("sandbox_uid", &self.sandbox_uid)
.field("sandbox_gid", &self.sandbox_gid)
.field("enable_bind_mounts", &self.enable_bind_mounts)
.field(
"health_check_interval_secs",
Expand Down Expand Up @@ -382,6 +410,52 @@ mod tests {
assert!(err.to_string().contains("sandbox_pids_limit"));
}

#[test]
fn sandbox_identity_defaults_to_10001() {
assert_eq!(
PodmanComputeConfig::default()
.resolve_sandbox_identity()
.unwrap(),
(10_001, 10_001)
);
}

#[test]
fn sandbox_identity_uses_uid_for_default_gid() {
let cfg = PodmanComputeConfig {
sandbox_uid: Some(12_345),
..PodmanComputeConfig::default()
};
assert_eq!(cfg.resolve_sandbox_identity().unwrap(), (12_345, 12_345));
}

#[test]
fn sandbox_identity_allows_configured_gid_without_uid() {
let cfg = PodmanComputeConfig {
sandbox_gid: Some(12_346),
..PodmanComputeConfig::default()
};
assert_eq!(cfg.resolve_sandbox_identity().unwrap(), (10_001, 12_346));
}

#[test]
fn sandbox_identity_rejects_system_uid() {
let cfg = PodmanComputeConfig {
sandbox_uid: Some(999),
..PodmanComputeConfig::default()
};
assert!(cfg.resolve_sandbox_identity().is_err());
}

#[test]
fn sandbox_identity_rejects_gid_above_policy_range() {
let cfg = PodmanComputeConfig {
sandbox_gid: Some(openshell_policy::MAX_SANDBOX_UID + 1),
..PodmanComputeConfig::default()
};
assert!(cfg.resolve_sandbox_identity().is_err());
}

#[test]
#[cfg(target_os = "macos")]
fn default_socket_path_uses_podman_machine_on_macos() {
Expand Down
Loading
Loading