bug(driver-podman): per-sandbox secrets and volumes are orphaned when a container disappears without DeleteSandbox

[feloy]

Agent Diagnostic

Reproduced live against main at 8cf2673c (2026-07-17, the commit behind release v0.0.86), with the gateway, CLI, Podman driver, and openshell/supervisor:dev image all rebuilt from that commit so no branch code was involved.

• Traced the create path (PodmanComputeDriver::create_sandbox, crates/openshell-driver-podman/src/driver.rs) and the store-side removal paths (apply_deleted_locked, prune_missing_sandbox, crates/openshell-server/src/compute/mod.rs).
• Ran a standalone Podman-backed gateway (mise run gateway), created a sandbox, removed its container out of band, and observed what survived.
• Confirmed no reconciliation exists for either resource family: git grep list_secrets / reconcile_orphaned in crates/openshell-driver-podman/ returns nothing on main.
• Searched open and closed issues for orphaned secret, secret reconciliation, podman secret cleanup, openshell-token, secret leak: no matching issue. The search was full-text, so an issue using different vocabulary could have been missed.
• Corroborating evidence that this happens in normal use, not just under a synthetic trigger: before starting, this machine had four openshell-token-* secrets, ~10 days old, with zero containers present. They were removed to get a clean baseline for the reproduction below.
• Surfaced during review of feat(sandbox,gateway): route sandbox egress through corporate HTTP proxy #2245, which adds a third per-sandbox secret with the same lifecycle. That PR's family is deliberately out of scope here — see the Scope note.

Description

The Podman driver creates two per-sandbox resources before it creates the container, and removes them in exactly two places: the create path's own error cleanup, and delete_sandbox. Neither runs if the container disappears without a DeleteSandbox call — whether because the gateway died mid-create, or the container was removed out of band.

Affected resources on main:

• Workspace volume — created at driver.rs:509, removed only in create-error cleanup (driver.rs:515,528,543,557,564,580) and delete_sandbox (driver.rs:672).
• openshell-token-* secret — created at driver.rs:101, removed only in those same paths and delete_sandbox (driver.rs:681).

When the container goes missing, the server removes its store record through apply_deleted_locked (compute/mod.rs:1381) without ever asking the driver to clean up. Two paths reach it:

• The watch path — the Podman watcher emits a deleted event, apply_deleted fires immediately, no grace period.
• The prune sweep — prune_missing_sandbox (compute/mod.rs:1476-1517) after ORPHAN_GRACE_PERIOD (300s).

apply_deleted_locked calls cleanup_sandbox_owned_records, which deletes only store objects (SSH sessions, sandbox settings). Nothing in either path touches Podman. The record naming the sandbox disappears while its volume and secret persist, leaving a bearer token and a workspace volume with nothing left in the system to attribute them to.

For the token secret this is a leaked credential (CWE-459, missing release of resource after effective lifetime). For the volume it is unbounded disk growth.

Expected: per-sandbox resources whose container no longer exists are reclaimed — either by the driver reconciling against live container IDs, or by the removal path asking the driver to clean up.

Scope note

PR #2245 adds a third family, openshell-proxy-auth-*, with the same lifecycle and a higher-value credential. It is not in scope here because it is not on main yet, but a fix should cover it too rather than be redone later. See #2245 (comment) for the scope discussion.

Reproduction Steps

Verified end to end on main at 8cf2673c.

1. Start a standalone Podman-backed gateway: mise run gateway
2. Create a sandbox: openshell sandbox create --name orphan-repro-main
3. Note the per-sandbox resources: podman secret ls, podman volume ls
4. Remove the container out of band: podman rm -f openshell-sandbox-orphan-repro-main
5. Wait ~10 seconds, then: openshell sandbox list (empty), podman secret ls and podman volume ls (both still present)

Observed timeline, sandbox ID 1eadfcbe-ad00-4ef9-9c95-1f92be936441:

Time (UTC)	Event
16:32:37	CreateSandbox — secret, volume, container created
16:35:38	podman rm -f (out-of-band removal)
16:35:48	Watcher: "Container already removed when inspecting after event, emitting deleted event"
16:35:48	Store record deleted via apply_deleted_locked
16:37:41	Secret and volume still present
16:39:44	Still present, past the 300s grace and ~4 reconcile sweeps; zero cleanup log lines

The same end state is reachable by killing the gateway between secret creation and container creation (driver.rs:509-580), which additionally strands the volume created moments earlier.

Environment

• OS: macOS (Darwin 25.5.0), Apple Silicon; the affected code is platform-independent
• Podman: 5.8.1, rootless, podman machine (libkrun)
• OpenShell: main at 8cf2673c, matching release v0.0.86 — gateway, CLI, driver, and supervisor image all rebuilt from this commit
• Latest release checked: yes — v0.0.86 is latest and contains the affected code
• Possible duplicates checked: yes — none found (see diagnostic for terms and caveat)

Logs

2026-07-19T16:35:48.490839Z  INFO openshell_driver_podman::watcher: Container already removed when inspecting after event, emitting deleted event container_id=a9f3aafedbcd… action=stop
2026-07-19T16:35:48.491922Z  INFO openshell_driver_podman::watcher: Container already removed when inspecting after event, emitting deleted event container_id=a9f3aafedbcd… action=die

$ openshell sandbox list
No sandboxes found.

$ podman secret ls --format '{{.Name}}'
openshell-token-1eadfcbe-ad00-4ef9-9c95-1f92be936441

$ podman volume ls --format '{{.Name}}'
openshell-sandbox-1eadfcbe-ad00-4ef9-9c95-1f92be936441-workspace

[elezar]

One question I would have here is whether this indicates a general driver-agnostic gap in the existing gateway <-> driver API?

[feloy]

One question I would have here is whether this indicates a general driver-agnostic gap in the existing gateway <-> driver API?

Yes — this is a driver-agnostic gap in the gateway↔driver API, not just a Podman-specific bug.

The ComputeDriver gRPC service has no cleanup or reconciliation hook. The only way to release driver-side resources is DeleteSandbox, and the server never calls it when a sandbox disappears out-of-band. apply_deleted_locked and prune_missing_sandbox only clean the store — they never reach back into the driver.

Every driver is affected:

Driver	Orphaned resources	Impact
Podman	Named volume + bearer-token secret	Disk growth + credential leak
VM	State dir (GiB), GPU reservation, subnet, TAP	Disk + hardware resource leak
Docker	Host-side token file	Small file but credential material
Kubernetes	Sandbox CR left behind (reverse orphan)	Low — K8s GC handles downstream pods

A durable fix should live at the API boundary with two complementary pieces:

1. Reactive cleanup — add a CleanupSandboxResources RPC (or make DeleteSandbox idempotent/best-effort) that apply_deleted_locked calls into the driver before dropping the store record.

2. Startup/crash-recovery reconciliation — add a ReconcileResources or ListOwnedResources RPC so the server's periodic sweep can ask the driver "what resource families do you still have?" and prune anything that no longer maps to a known sandbox.

Option 1 alone doesn't cover gateway crashes mid-create (the store record may never have existed), so both are needed. The VM driver's existing cleanup_stale_tap_interfaces at startup is a driver-local version of option 2 — the need has been felt before but solved ad-hoc rather than at the trait level.

[krishicks]

📋 triage-agent

Removing state:triage-needed as workflow cleanup. Agent review of the issue's current state found substantive maintainer discussion, indicating that it is no longer awaiting initial triage. This label removal does not imply roadmap acceptance or implementation priority.

[politerealism]

I'd like to pick this up. Building on @feloy's driver-agnostic diagnosis above, I'll implement the two RPCs proposed in this thread at the ComputeDriver service boundary rather than a Podman-only patch:

• CleanupSandboxResources (or an idempotent DeleteSandbox) — called from apply_deleted_locked before the store record is dropped, so reactive cleanup happens driver-side instead of only in the store.
• ReconcileResources / ListOwnedResources — called from the periodic sweep so the server can ask each driver what it still holds and prune anything unmapped, covering crash-recovery cases where the deletion event itself was missed.

I'll implement this for the Podman driver first, and design the contract so Docker, VM, and Kubernetes can adopt it as explicit follow-up work rather than needing a second redesign later.

One update to scope: #2245 has since merged, so the openshell-proxy-auth-* secret family mentioned in the scope note above is no longer hypothetical — I checked crates/openshell-driver-podman/src/driver.rs directly and confirmed cleanup_sandbox_proxy_auth_secret has the identical gap (only called from create-error cleanup and delete_sandbox, never on out-of-band removal). So Podman's fix needs to reconcile three resource families, not two: workspace volume, token secret, and proxy-auth secret.

Also worth noting: #2591 (server-managed expiration leases) explicitly defers to "driver-specific orphan cleanup" for resources left behind after a sandbox record is lost — so this reconciliation primitive already has a second consumer beyond this issue, which makes the case for the durable API-boundary fix stronger rather than a one-off patch.

Considered a Podman-only stopgap first, but decided against it — it'd be throwaway work once the real contract lands, and I'd rather put review effort into the durable fix directly. Happy to adjust if there's a reason to want an interim fix while the API design gets reviewed.

[politerealism]

Update to my plan above — I traced the actual code paths before implementing and found a smaller, better fix than the two-RPC design I proposed.

DeleteSandbox is already idempotent for Podman and already does the full cleanup. crates/openshell-driver-podman/src/driver.rs's delete_sandbox already reclaims the workspace volume, token secret, and proxy-auth secret even when the container is already gone — it doesn't error, it just cleans up and returns deleted: false. There's an existing test proving this (delete_sandbox_cleans_up_volume_when_container_is_already_gone).

The actual bug is entirely server-side: I traced both places that discover a sandbox is gone out-of-band — the watch-path (apply_deleted_locked) and the periodic sweep (remove_sandbox_record_if_version_locked, reached via prune_missing_sandbox) — and neither one ever calls the driver's DeleteSandbox RPC. Both only clean up store-side objects, through a shared helper, cleanup_sandbox_owned_records. The explicit, user-initiated delete path already calls driver.delete_sandbox correctly and doesn't go through that helper, so this isn't a regression risk for normal deletes.

Revised fix: add one self.driver.call("driver.delete_sandbox", ...) invocation inside cleanup_sandbox_owned_records, using the same call pattern already used twice elsewhere in this file, made best-effort (log and continue on failure, since these paths already believe the sandbox is gone). No proto changes, no new RPCs.

This also covers the mid-create crash reproduction, not just the out-of-band-removal one: the server writes the sandbox's store record via put_if before ever calling the driver's CreateSandbox (compute/mod.rs:961-1015), so even in the "gateway killed between secret creation and container creation" case, a store record already exists. The existing periodic sweep already walks every store record and queries the driver via GetSandbox — once DeleteSandbox gets wired into the cleanup path that sweep already reaches, both reproduction scenarios close.

Deliberately deferring the ReconcileResources/ListOwnedResources piece. That RPC would matter for a genuinely different failure mode — a driver resource with no store record at all to reconcile against (e.g. the store record itself was lost or deleted outside DeleteSandbox). That's real defense-in-depth, but it's not what this issue reproduces, and I'd rather not build it speculatively. Happy to open a follow-up issue for it if anyone has a concrete case where it's needed.

[politerealism]

Opened #3042 implementing the revised plan above. Also filed #3041 for a related-but-separate gap in the Docker driver's own DeleteSandbox idempotency, surfaced while verifying this fix reaches every driver's leak, not just Podman's.