feat(test-classifier): sandboxed OBSERVED runs via smolvm (control plane + plan) - #65
Open
fg-nava wants to merge 2 commits into
Open
feat(test-classifier): sandboxed OBSERVED runs via smolvm (control plane + plan)#65fg-nava wants to merge 2 commits into
fg-nava wants to merge 2 commits into
Conversation
Design doc (no implementation) for running OBSERVED (AI_RUN_SUITE=1) inside a disposable on-device smolvm VM instead of directly on the developer's machine. Resolves the no-sandbox risk flagged in LOCAL_TEST_CLASSIFIER.md: OBSERVED installs + runs a PR branch's code with the developer's full ambient creds and leaves build artifacts in the working tree. Decisions captured: credentials cross via an ephemeral 0600 env-file + a copied per-tool CLI config dir in the mount (no --env flag in smolvm; secrets kept off the command line); strict --allow-host egress allowlist; ephemeral machine run only (no named VMs); strict stage→run→teardown lifecycle with trap-based cleanup and secret shredding. Documents the undocumented-smolvm gaps as spikes to de-risk (exit-code propagation, browser-mode in-guest, cred-dir copy), and a phased delivery where each phase is its own approval-gated PR.
Implements Phase 1 of the smolvm sandbox plan: an opt-in control plane that
runs OBSERVED (AI_RUN_SUITE=1) inside a disposable on-device smolvm VM instead
of directly on the developer's machine.
scripts/sandbox-run.sh (the control plane / trust boundary):
• stages a CLEAN copy of the repo at HEAD into a 0700 temp dir mounted at
/workspace — never the real checkout, so install artifacts (node_modules/,
browsers, caches) stay ephemeral and vanish on teardown;
• writes a 0600 env-file with ONLY the vars the classifier reads (AI_REVIEW_*,
GH_TOKEN, METRICSAI_*, provider API keys) — smolvm has no --env flag, so
secrets stay off the command line; sourced in-VM, never argv;
• copies the RESOLVED tool's CLI config dir (~/.claude / ~/.codex / gh) into a
staged HOME so claude/codex authenticate in-guest (key- or OAuth-based);
• runs an ephemeral with --net + a strict --allow-host
allowlist (GitHub, provider, metricsai incl. the googleusercontent 302 hop,
package registries; override via AI_SANDBOX_ALLOW_HOSTS);
• trap cleanup EXIT INT TERM shreds the env-file and removes the staging dir
even on interrupt; preflight fails CLOSED if smolvm is absent (no silent
unsandboxed fallback).
dispatcher: routes OBSERVED+AI_SANDBOX=1 through the control plane, which
re-invokes the dispatcher in-VM with AI_SANDBOX_ACTIVE=1 set (recursion guard).
INFERRED is never routed (it executes nothing); CI path unchanged.
docs: LOCAL_TEST_CLASSIFIER.md gains a 'Sandboxed OBSERVED' section; the design
doc's status moves to Phase-1-implemented with Phase-0 spikes still pending a
real smolvm install.
Tested host-side (smolvm not installed here, so the in-VM path is unverified —
that's the Phase-0 spike): routing matrix (in-VM guard / INFERRED / OBSERVED+
sandbox / OBSERVED-no-sandbox all correct), env-file contents, staging layout,
allowlist assembly, and teardown leaving nothing behind.
fg-nava
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Opt-in sandbox control plane that runs OBSERVED (
AI_RUN_SUITE=1) inside a disposable on-device smolvm VM, plus the design doc behind it. Resolves the no-sandbox risk: OBSERVED installs and runs a PR branch's code on the laptop with the developer's full ambient creds and litters the working tree.Implementation
scripts/sandbox-run.sh— the control plane / trust boundary:0700temp dir mounted at/workspace— never the real checkout, so artifacts stay ephemeral and are removed on teardown (no mess);0600env-file with only the vars the classifier reads + provider keys (smolvm has no--envflag, so secrets stay off the command line); sourced in-VM;~/.claude/~/.codex/ gh) into a stagedHOMEso the CLI authenticates in-guest;smolvm machine runwith--net+ a strict--allow-hostallowlist (override viaAI_SANDBOX_ALLOW_HOSTS);trap cleanup EXIT INT TERMshreds the env-file + removes staging even on interrupt; preflight fails closed if smolvm is absent (no silent unsandboxed fallback).Dispatcher routes OBSERVED+
AI_SANDBOX=1through the control plane, which re-invokes the dispatcher in-VM withAI_SANDBOX_ACTIVE=1(recursion guard). INFERRED is never routed; CI unchanged.Docs —
LOCAL_TEST_CLASSIFIER.mdgains a Sandboxed OBSERVED section;SANDBOXED_OBSERVED.mdis the full design.Tested vs. not — read this
smolvmis not installed in my environment, so host-side logic is verified but not the in-VM execution:machine run; whether a copied~/.claude/~/.codexauthenticates after theHOMEremap; Playwright browser-mode in-guest.Scope
Opt-in, default off — nothing changes for existing runs. Not for merge yet — review of approach + implementation; in-VM path needs a real smolvm run first.