feat(sandbox): support rootfs tar as --from source for VM driver - #2863
Open
feloy wants to merge 7 commits into
Open
feat(sandbox): support rootfs tar as --from source for VM driver#2863feloy wants to merge 7 commits into
feloy wants to merge 7 commits into
Conversation
feloy
requested review from
a team,
derekwaynecarr,
mrunalp and
sjenning
as code owners
August 21, 2026 07:14
6 tasks
johntmyers
reviewed
Aug 25, 2026
johntmyers
left a comment
johntmyers
left a comment
Collaborator
There was a problem hiding this comment.
gator-agent
PR Review Status
The VM rootfs-tar feature is project-valid and documented, but this head has three blocking implementation issues: the new raw host-path field crosses the driver trust boundary, the CLI encodes that field outside the required vm configuration object, and the cache key can collide or remain stale.
Action required: @feloy, please address GATOR-28b5152e-01, GATOR-28b5152e-02, and GATOR-28b5152e-03, including the requested regression coverage.
Blocking findings:
GATOR-28b5152e-01: caller-controlled host paths reach privileged VM-driver filesystem reads.GATOR-28b5152e-02: the CLI-generated rootfs path is discarded by driver-config selection.GATOR-28b5152e-03: rootfs archive cache identities are collision-prone, stale at sub-second updates, and unbounded.
Carried findings:
- None
Gator metadata
- Validation: Project-valid implementation of maintainer-supported issue #2175, scoped to local VM gateways.
- Docs: Fern sandbox documentation updated for the direct UX change.
- Checks: Current required Branch Checks and Helm Lint have not run and remain pending.
- E2E:
test:e2eis required for VM sandbox lifecycle behavior but is not dispatched while review blockers remain. - Head SHA:
28b5152ee0e3bc377b555cdbb89faa7cc8bc1f5c - Base SHA:
40d1b4866691be03b5617089739c7f2325487479 - Merge base SHA:
40d1b4866691be03b5617089739c7f2325487479 - Patch ID:
4f052d6ec908bc7bf57cfeb49f381929c826deee - Gator payload:
7 - Review mode:
initial - Previous reviewed SHA: none
- Review budget exhausted: no
- Maintainer decision required: no
- Next state:
gator:in-review
johntmyers
added
gator:in-review
johntmyers
reviewed
Aug 26, 2026
johntmyers
left a comment
johntmyers
left a comment
Collaborator
There was a problem hiding this comment.
gator-agent
PR Review Status
Thanks @feloy, I checked the new staging-boundary validation against the three existing findings. The new head narrows arbitrary host-path access, but the documented CLI still copies archives into shared persistent staging before enforcing the configured limit, still encodes the rootfs path outside driver_config.vm, and leaves the cache identity and copy-consistency defects unchanged. No new blockers were added.
Action required: @feloy, please finish the three carried obligations and their requested regression coverage.
Blocking findings:
- No new blocking findings.
Carried findings:
GATOR-28b5152e-01: use request-bound server-controlled staging, enforce the byte limit while receiving the archive, prevent cross-request path selection, and clean up staged data.GATOR-28b5152e-02: nestrootfs_tar_pathunderdriver_config.vm, deep-merge existing VM settings, and cover the CLI-to-driver transport.GATOR-28b5152e-03: use a fixed-length collision-resistant version identity and reject or retry if the archive changes during copy.
Gator metadata
- Validation: Project-valid implementation of maintainer-supported issue #2175, scoped to local VM gateways.
- Docs: Fern sandbox documentation is present for the direct UX change.
- Checks: Current-head Branch Checks and Helm Lint are pending; pipeline dispatch waits for review blockers to resolve.
- E2E:
test:e2eremains required for VM sandbox lifecycle behavior and will be dispatched after review blockers resolve. - Head SHA:
75e1f2be4b522e53381ab4e2a958c6dba7e031ff - Base SHA:
8be8b62ab58e42ee08213494b0d556ebc829e5de - Merge base SHA:
8be8b62ab58e42ee08213494b0d556ebc829e5de - Patch ID:
2c200e7c9e38781640f7fbeaa9eab2919ebce81b - Gator payload:
7 - Review mode:
follow_up - Previous reviewed SHA:
28b5152ee0e3bc377b555cdbb89faa7cc8bc1f5c - Review budget exhausted:
no - Maintainer decision required:
no - Next state:
gator:in-review
johntmyers
added
gator:in-review
johntmyers
reviewed
Aug 26, 2026
johntmyers
left a comment
johntmyers
left a comment
Collaborator
There was a problem hiding this comment.
gator-agent
PR Review Status
Thanks @feloy, I checked the latest staging-limit, request-directory, and cleanup changes against the three existing obligations. The raw staging reference is still not request-bound or bounded while copying, the CLI transport and cache identity remain unchanged, and the new cleanup makes a successfully created rootfs-tar sandbox unable to restart or restore.
Action required: @feloy, please address GATOR-168b9210-01 and finish the three carried obligations with their requested regression coverage.
Blocking findings:
GATOR-168b9210-01: deleting the one-shot staged archive leaves persisted sandbox state pointing to a path that restart and restoration must canonicalize.
Carried findings:
GATOR-28b5152e-01: replace caller-selected staging paths with request-bound server-controlled staging, enforce the byte limit while receiving, and clean up every terminal path safely.GATOR-28b5152e-02: nestrootfs_tar_pathunderdriver_config.vm, deep-merge existing VM settings, and cover CLI-to-driver transport.GATOR-28b5152e-03: use a fixed-length collision-resistant archive-version identity and reject or retry if the source changes during copy.
Gator metadata
- Validation: Project-valid implementation of maintainer-supported issue #2175, scoped to local VM gateways.
- Docs: Fern sandbox documentation remains present for the direct UX change.
- Checks: Current-head Branch Checks and Helm Lint are pending; pipeline dispatch remains gated on review blockers.
- E2E:
test:e2eis required for VM sandbox lifecycle behavior but must not be dispatched while review blockers remain. - Head SHA:
168b9210cc70a99b003a18b18b2bead91d30f0b9 - Base SHA:
8be8b62ab58e42ee08213494b0d556ebc829e5de - Merge base SHA:
8be8b62ab58e42ee08213494b0d556ebc829e5de - Patch ID:
1f7bcbdefada29ff2276cb444f62a4b8984857ae - Gator payload:
7 - Review mode:
follow_up - Previous reviewed SHA:
75e1f2be4b522e53381ab4e2a958c6dba7e031ff - Review budget exhausted:
yes - Maintainer decision required:
no - Next state:
gator:in-review
Collaborator
Maintainer Convergence DecisionThanks @feloy. I checked the new persisted-image restoration path at head Root-cause findings:
Scope growth:
Reviewer-quality signals:
Maintainer action: @NVIDIA/openshell-maintainers, please decide whether Gator metadata
|
johntmyers
removed
the
gator:in-review
johntmyers
added
gator:watch-pipeline
|
Label |
johntmyers
added
gator:blocked
Accept flat rootfs tar archives (.tar, .tar.gz, .tgz) via the --from flag for VM-backed gateways. The CLI detects the archive extension, validates that the gateway uses the VM compute driver, and passes the tar path through driver_config. The VM driver copies the tar into its staging area and feeds it into the existing rootfs extraction and ext4 disk creation pipeline, skipping the container image pull/export steps. Closes NVIDIA#2175 Signed-off-by: Philippe Martin <phmartin@redhat.com>
The rootfs_tar_path field in driver_config was passed from the API caller directly to tokio::fs::copy without validation. An authenticated user bypassing the CLI could supply arbitrary host paths (e.g. /dev/zero for disk exhaustion, or readable host files for data exfiltration). Introduce a trusted staging directory that the VM driver creates on startup and advertises via GetCapabilities. The CLI now copies the tar into the staging directory before creating the sandbox, and the driver validates that the received path is a regular file inside the staging root and within a configurable size limit (default 10 GiB) before any I/O. New VmDriverConfig options: - rootfs_tar_staging_dir: override the staging directory (default: <state_dir>/rootfs-tar-staging) - rootfs_tar_max_bytes: override the size limit (default: 10 GiB) Addresses GATOR-28b5152e-01. Signed-off-by: Philippe Martin <phmartin@redhat.com>
… rootfs tar Tighten the rootfs tar staging flow to address the remaining GATOR-01 obligations: - Request-scoped staging: the CLI creates a unique per-request subdirectory (req-<pid>) under the staging root instead of placing files directly in the shared directory. The driver enforces that the tar path is at depth 2 (staging_root/<subdir>/<file>), preventing cross-request path selection. - Size pre-check: the driver advertises rootfs_tar_max_bytes via GetCapabilities. The CLI reads this limit and rejects oversized files before copying, avoiding disk exhaustion in the staging directory. - Cleanup: the driver removes the request staging subdirectory after consuming the tar (on cache hit, copy success, or copy failure), ensuring staged data does not persist beyond the request. Signed-off-by: Philippe Martin <phmartin@redhat.com>
On restore or restart, the one-shot staged tar archive has already been cleaned up. Reading the persisted image identity from the sandbox state directory and resolving the cached disk path directly avoids re-accessing the deleted staging path. Addresses GATOR-168b9210-01. Signed-off-by: Philippe Martin <phmartin@redhat.com>
…s tar copy Replace PID-based request staging directories with tempfile-generated random names to prevent collisions and make paths unpredictable. Replace bare tokio::fs::copy with a streaming copy loop that enforces the advertised max_bytes limit during transfer, closing the TOCTOU gap between the pre-copy size check and the actual copy. Signed-off-by: Philippe Martin <phmartin@nvidia.com> Signed-off-by: Philippe Martin <phmartin@redhat.com>
A caller could name any host path in `driver_config.vm.rootfs_tar_path`, which the privileged VM driver then read. The CLI-side locality check did not apply to direct API requests. The gateway now owns staging. `BeginRootfsTarStaging` allocates a request-scoped directory under the driver-advertised staging root and returns an opaque single-use token; `CreateSandbox` carries the token, and the gateway substitutes the path it allocated before dispatching to the driver. `template.driver_config.<driver>.rootfs_tar_path` is rejected outright in request validation, so a caller-supplied path never reaches privileged I/O. Tokens are bound to the issuing workspace and subject, consumed once, and expire after 30 minutes. Outstanding slots are capped per caller and overall, so one caller can neither exhaust the staging filesystem nor starve others. An RAII guard reclaims the directory on every failure path after consumption, and an age-gated sweep runs at startup and on each reconcile pass for directories whose driver died before its own cleanup. The token is stripped from the public sandbox before persistence: the stored copy is returned verbatim by GetSandbox, ListSandboxes and WatchSandbox to every member of the workspace. Also fixes two defects this exposed: - The CLI wrote `rootfs_tar_path` at the top level of `driver_config`, but the gateway forwards only `driver_config.<driver_name>`, silently dropping unmatched keys. The archive never reached the VM driver, so the documented `--from ./rootfs.tar` flow did not work at all. Config is now nested under `vm` and deep-merged, so a caller's existing VM settings survive instead of being clobbered by a shallow extend. - Staging previously required `GetGatewayInfo`, which is restricted to `platform_admin`, making the feature unusable for ordinary users on any RBAC-enabled gateway. The new RPC matches CreateSandbox at `sandbox:write` / `workspace_role: user`. `compute_driver.proto` is unchanged; the gateway reads the staging root from the capabilities it already stores. Refs NVIDIA#2175 Signed-off-by: Philippe Martin <phmartin@redhat.com>
The prepared-disk cache key combined the archive's full path with an mtime truncated to seconds, then mapped punctuation to `-`. Distinct paths such as `/tmp/a/b.tar` and `/tmp/a-b.tar` collapsed onto the same key and reused each other's disk, a rewrite within the same second kept stale contents, and a long path could exceed filesystem component limits. Identity is now a SHA-256 of the archive contents. This is also what makes the cache work at all now that the gateway allocates a fresh staging directory per request: a path-derived key would miss on every create. The archive is hashed, the cache checked, and only on a miss copied — so a hit skips writing a multi-gigabyte file. The copy is hashed as it is written and rejected if the digest differs from the first pass, which closes the window where the source changes during staging rather than approximating it with a re-stat. Refs NVIDIA#2175 Signed-off-by: Philippe Martin <phmartin@redhat.com>
Collaborator
|
/ok to test 3ed0174 |
johntmyers
reviewed
Sep 1, 2026
johntmyers
left a comment
johntmyers
left a comment
Collaborator
There was a problem hiding this comment.
gator-agent
PR Review Status
Thanks @feloy. I checked the rebased author series at head 3ed017419a0af3245d4a19d65c05988aa64c14ac against your gateway-issued staging, nested VM configuration, and content-addressed cache fixes. The range-diff preserves the four resolved Gator obligations, and the required critical-only review found no newly introduced Critical defect.
Blocking findings:
- No blocking findings remain.
Carried findings:
- None;
GATOR-168b9210-01,GATOR-28b5152e-01,GATOR-28b5152e-02, andGATOR-28b5152e-03remain resolved.
Gator metadata
- Validation: Project-valid implementation of maintainer-supported issue #2175, scoped to local VM gateways.
- Docs: Fern sandbox and gateway configuration documentation remain updated for the direct UX change and managed-driver constraint.
- Checks: Current-head Branch Checks run
33490039963, Helm Lint run33490039891, and Branch E2E run33490040676are queued or running. - E2E:
test:e2eremains applied;/ok to test 3ed017419a0af3245d4a19d65c05988aa64c14accreated the current-head mirror and dispatched Branch E2E. - Head SHA:
3ed017419a0af3245d4a19d65c05988aa64c14ac - Base SHA:
8ffc6c2a13cd482dea9e2195058309344c274756 - Merge base SHA:
8ffc6c2a13cd482dea9e2195058309344c274756 - Patch ID:
4f57b99546fb06cde7c7c68a6a4213035780a994 - Gator payload:
7 - Review mode:
critical_only - Previous reviewed SHA:
e17ee02d5d894ad143dc3f9a3bcaaeeb704bb5df - Review budget exhausted:
yes - Maintainer decision required:
no — all prior findings remain resolved, there is no qualifying scope growth, and no new Critical was found - Review telemetry: 0 proposed findings, 0 blockers, 0 duplicate proposals, and 0 unchanged-code proposals on this head; ledger history records 4 finding-bearing rounds and 4 unique historical findings.
- Next state:
gator:watch-pipeline
johntmyers
added
gator:watch-pipeline
Contributor
Author
|
the failing test seems flaky, is it possible to restart it? |
johntmyers
added
gator:approval-needed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
.tar,.tar.gz,.tgz) as a new--fromsource for VM-driver sandboxes, enabling daemon-free sandbox creation from flat filesystem archives produced bydocker export,podman export, orbuildah mount+tardriver_configRelated Issue
Closes #2175
Changes
CLI (
crates/openshell-cli/)ResolvedSource::RootfsTarvariant inresolve_from()with extension-based detectionvalidate_rootfs_tar_source()checks local-gateway and VM-driver constraints viaGetGatewayInfoRPCrootfs_tar_driver_config()andmerge_driver_config()encode the tar path intodriver_config--fromhelp text and error messagesVM driver (
crates/openshell-driver-vm/)rootfs_tar_pathfield onVmSandboxDriverConfigensure_prepared_rootfs_tar_disk()with double-checked cache locking and mtime-based identityvalidate_sandbox(),create_sandbox(),prepare_runtime_images(), and reconciliation to accept rootfs tar as an alternative to image referencesbootstrap_image_ref_default()for reuse in rootfs tar and reconciliation pathsDocs (
docs/sandboxes/manage-sandboxes.mdx)--fromdocumentation with rootfs tar example and VM-driver constraintTests
run.rs: extension detection for.tar/.tar.gz/.tgz, missing archive rejection,filename_looks_like_rootfs_tarcoveragee2e/rust/tests/rootfs_tar.rs): builds a Docker image, exports a flat rootfs tar, creates a VM sandbox from it, and verifies a marker fileTesting
mise run pre-commitpassesChecklist
driver_configcarries the tar path)