Skip to content

Machine-checkable sandbox env-var contract (manifest of exported/reserved sandbox variables) #8465

Open

Description

@jflowers

Machine-checkable sandbox env-var contract (manifest of exported/reserved sandbox variables)

Problem

The set of environment variables the runner injects into — and reserves from — the sandbox is hand-synced knowledge in three places across two repos, and none of them is machine-checked against the actual code:

  1. fullsend, against itself. reservedSandboxKeys (internal/cli/run.go:3227) carries a literal note: "NOTE: keep in sync with bootstrapEnv exports below for FULLSEND_ keys."* No test enforces that sync.
  2. In consumers. fullsend-ai/agents maintains SANDBOX_RUNTIME_VARS in scripts/skill-env-reachability-test.sh — a hand-maintained mirror of what the runner injects — used to validate that every SKILL.md env-var reference is reachable at runtime.
  3. In prose. Runtime-specific exports (CLAUDE_CONFIG_DIR, CODEX_HOME, PI_CODING_AGENT_DIR via EnvExports(), internal/runtime/runtime.go:162) are discoverable only by reading internal/runtime/*.go.

This has drifted in practice. fullsend-ai/agents#1677 (migrating the pr-risk-assessment skill from CLAUDE_CONFIG_DIR to a proposed FULLSEND_SKILLS_DIR) passed the agents repo's own reachability gate because the same PR edited the hand-maintained allowlist — the test validated the repo's claim against its own assertion. No released runner exports FULLSEND_SKILLS_DIR (zero hits in this repo), so the change would have silently degraded risk signals at the next lockstep release (risk-tier1.sh exits 0 on failure by design). The breakage was caught by a bot reviewer, not by either repo's CI.

The general failure mode: content repos cannot verify, at PR time, that env vars their skills reference are exported by any released runner.

Proposal

Emit the sandbox env-var contract as a machine-readable manifest generated from a single source of truth:

  1. Single source of truth. A generator derives the manifest from the same data bootstrapEnv (internal/cli/run.go:3625) and reservedSandboxKeys already use: bootstrap export lines, reserved keys + oidcDenyKeys/providerOnlyKeys merges, per-runtime EnvExports(), and host-file injection (GOOGLE_APPLICATION_CREDENTIALS via gcp-vertex.env).
  2. Golden test. A committed manifest is diffed against generator output on every PR touching run.go, internal/runtime, or sandbox env code. The "keep in sync" NOTE becomes a checked invariant.
  3. Stable committed path. The manifest lives at a stable repo path (e.g. docs/contracts/sandbox-env.json — location bikesheddable) so consumers can fetch it via raw.githubusercontent.com at a tag — the same pinned-URL mechanism harnesses already use (ADR 0038). No release-asset machinery, no binary download.
  4. Phase 2 (optional). fullsend sandbox env-manifest --json subcommand for humans/debugging.

Manifest shape

{
  "schema_version": 1,
  "fullsend_version": "0.47.1",
  "exports": {
    "always": ["PATH", "FULLSEND_OUTPUT_DIR", "FULLSEND_TARGET_REPO_DIR", "FULLSEND_ROLE", "FULLSEND_SLUG", "TRACEPARENT"],
    "conditional": {
      "FULLSEND_OUTPUT_SCHEMA": "validation_loop.schema present",
      "FULLSEND_OUTPUT_FILE": "legacy RunnerEnv",
      "FULLSEND_FETCH_URL": "fetch service enabled",
      "FULLSEND_FETCH_TOKEN": "fetch service enabled",
      "GOOGLE_APPLICATION_CREDENTIALS": "gcp-vertex.env host file"
    },
    "runtime_specific": {
      "CLAUDE_CONFIG_DIR": "claude",
      "CODEX_HOME": "codex",
      "PI_CODING_AGENT_DIR": "pi"
    }
  },
  "reserved": ["HOME", "SHELL", "LD_PRELOAD", "OPENAI_API_KEY"]
}

Categories preserve semantics: always + conditional = universally reachable (modulo feature gates); runtime_specific = exported only by a given runtime — consumers needing portability should treat these as gaps, not guarantees.

Out of scope

Adding, renaming, or reserving any new variable (e.g. FULLSEND_SKILLS_DIR, fullsend-ai/agents#1672). That's a follow-up once the contract mechanism is agreed, so any new variable arrives with its contract surface already machine-checked.

Acceptance criteria

  • Golden test fails when bootstrapEnv, reservedSandboxKeys, EnvExports, or host-file injection changes without a manifest bump
  • All EnvExports() implementers represented (claude, codex, pi; opencode is nil)
  • Conditional exports represented with their gating conditions
  • Host-file injection represented
  • Manifest committed at a stable path, fetchable at tags/<version> via raw.githubusercontent.com
  • Downstream consumption proven: agents' skill-env-reachability-test.sh reads the released-tag manifest (agents-side change tracked separately)

Proposed ADR 0142 — "Sandbox environment contract manifest"

  • Context: three hand-synced descriptions of sandbox env; incident refactor(skills): migrate pr-risk-assessment from CLAUDE_CONFIG_DIR to FULLSEND_SKILLS_DIR agents#1677.
  • Decision: single-source generator + golden test + committed manifest at a stable path; consumers validate against the latest released tag at PR time and against main nightly (ADR 0139 bridge).
  • Consequences: new runner env vars require a manifest bump in the same PR (visible in diff); content repos gain a fail-closed gate; the transitional lockstep coupling (ADR 0058) becomes verifiable instead of tribal knowledge; manifest is contract data, not behavior — no runtime cost.

References

Line numbers verified against origin/main 9f188578.

Activity

  1. fullsend-ai-triage commented on Oct 15, 2026

    @fullsend-ai-triage

    🤖 Finished Triage · ✅ Success · Started 4:27 PM UTC · Completed 4:29 PM UTC

    Commit: 99417cc · View workflow run →

    Runtime: claude · Model: sonnet → claude-sonnet-5-5 · Effort: high · Cost: $0.21

  2. added
    featureFeature-category issue awaiting human prioritization
    triagedTriaged but awaiting human prioritization
    on Oct 15, 2026
  3. fullsend-ai-triage commented on Oct 15, 2026

    @fullsend-ai-triage

    This reads as a well-scoped feature proposal, and the code references check out against the current tree: reservedSandboxKeys carries the sync NOTE, bootstrapEnv exists, and EnvExports() is implemented by the claude, codex and pi runtimes (opencode and the dummy runtime return nil). No open PR or sub-issue addresses this, and no duplicate was found. #7010 (reservedSandboxKeys only guards env.sandbox) is related but not blocking.

    Notes for the implementer:

    • bootstrapEnv builds many exports as inline lines. To make the single-source-of-truth claim real, the always/conditional keys and their gating conditions probably need to be declared as data that both bootstrapEnv and the generator consume. A generator that re-lists the keys would just be a fourth hand-synced copy.
    • docs/contracts/ does not exist yet, so it is a new path. Check the docs-site sidebar config if the manifest or its description should be surfaced there.
    • fullsend_version in the committed manifest would make the golden test fail on every release unless it is stamped at release time or excluded from the diff.
    • The latest ADR in the repo is 0141. The proposed 0142 looks free, but confirm there is no collision before merging.
    • Per AGENTS.md, add or update docs and keep patch coverage at or above the Codecov gate.

    Proposed test case:

    TestSandboxEnvManifest_Golden: generate manifest in-process, compare to committed JSON, fail with regenerate hint on diff.
    TestBootstrapFullsendKeysAreReserved: every FULLSEND_* key exported by bootstrapEnv is in reservedSandboxKeys.
    Edge cases: nil EnvExports (opencode), conditional keys with gating text, deterministic ordering.
    

    The agents-repo change that reads the released-tag manifest is tracked separately and is not a prerequisite for the fullsend-side work. Because this is a feature, it is not eligible for automatic promotion to ready-to-code, so a maintainer should review the design (manifest path, ADR) first.


    Next steps:

    • /fs-code — agent creates a PR to implement this issue
    • /fs-code <your instruction> — agent implements with your specific guidance
  4. jflowers commented on Oct 15, 2026

    @jflowers
    ContributorAuthor

    Divisor review panel triage: #8465

    Seven reviewers assessed this issue. All seven read it as a valid, well-scoped feature proposal, with code references verified against the current tree. The consolidated verdict is NEEDS-CLARIFICATION. The proposal looks solid overall; a few design details are worth settling before implementation begins.

    Questions from the panel:

    1. fullsend_version in the manifest. The golden test compares the committed manifest against generator output. What happens to that comparison on every release bump? Worth deciding whether the field is excluded from the diff or stamped at release time, so the gate does not fail on each release.

    2. A fifth export site. The proposal names four sources: bootstrapEnv, reservedSandboxKeys, per-runtime EnvExports(), and gcp-vertex.env host-file injection. iterationEnvCommand (internal/cli/run.go:3507) also writes FULLSEND_TIMEOUT_MINUTES, FULLSEND_ITERATION_DEADLINE, and TRACEPARENT to iteration.env, which the sandbox .env sourcing chain picks up. The example manifest lists TRACEPARENT under always but omits the two timing vars, which are both exported and reserved. Would a generator scoped to the four named sources be born incomplete?

    3. Dynamic reserved keys. init() merges oidcDenyKeys and providerOnlyKeys into reservedSandboxKeys at load time, and the map contains the computed runnerPathVar key. Should the generator consume the merged runtime state rather than the static map literal, so dynamic entries survive?

    4. Runtime-implementer scope. dummy.go and dummy_playback.go also implement EnvExports(), returning nil. Does the "all implementers represented" criterion include the test runtimes?

    5. Pi's full export set. PiRuntime exports six variables: PI_CODING_AGENT_DIR, PI_CODING_AGENT_SESSION_DIR, PI_OFFLINE, PI_SKIP_VERSION_CHECK, PI_TELEMETRY, and JITI_FS_CACHE. The example manifest shows only the first.

    6. New docs path. docs/contracts/ does not exist yet. Per AGENTS.md, the implementer should check the docs/.vitepress/config.ts sidebar config and add the folder prefix to search.options.scopes.

    Duplicate check: the search surfaced #303 (agent permissions migration), which the panel judged topically unrelated. #7010 (reservedSandboxKeys guards env.sandbox only) is related but distinct, as the earlier triage noted.

    Classification: feature (unanimous), objective, no split recommended, no duplicate. ADR 0142 is free; the latest ADR is 0141.

    This triage was performed by the Divisor review panel.

  5. jflowers commented on Oct 16, 2026

    @jflowers
    ContributorAuthor

    Answers to the panel's questions, verified against the current tree:

    1. fullsend_version in the manifest. Dropping the field. The version variable is "dev" unless ldflags stamp it at release build time (internal/cli/root.go:12), so a golden test can never generate a real value — committed output would always read "dev". Stamping at release time conflicts with the fetch mechanism: the manifest is pinned at tags/<version>, the tag tree is frozen at tag time, and there is no pre-tag version-bump commit to carry the stamp (the version is ldflags-injected by GoReleaser, not a committed file). The consumer already knows the version by construction — it is the tag in the URL it pinned. schema_version stays for shape.

    2. Fifth export site. Correct — iterationEnvCommand (internal/cli/run.go:3506) joins the generator's source list as a fifth site, alongside bootstrapEnv, the reservedSandboxKeys merges, per-runtime EnvExports(), and gcp-vertex.env. It writes FULLSEND_TIMEOUT_MINUTES, FULLSEND_ITERATION_DEADLINE, and TRACEPARENT to iteration.env, which the sandbox .env chain sources. All three were already reservation-checked (run_test.go:4522-4524); the export side was the gap. The example manifest's always list showing TRACEPARENT without the two timing vars was this same incompleteness — the generator derives the full set from the template production writes.

    3. Dynamic reserved keys. The generator consumes the merged runtime state. init() merges oidcDenyKeys and providerOnlyKeys into reservedSandboxKeys at load (run.go:3270-3293), and runnerPathVar is already in the map literal (run.go:3264). Running the generator in-process (a Go test in internal/cli) means init() has already executed — reading the map yields merged state with no re-derivation. Re-deriving the union from the literals would duplicate the merge logic, which is the hand-sync failure mode this proposal exists to eliminate.

    4. Runtime-implementer scope. The criterion changes from "all implementers" to the config-selectable set: the manifest's runtime section derives from config.ValidRuntimes() (internal/config/config.go:300) — claude, pi, codex, dummy, dummy-playback. The dummy runtimes are config-selectable for behaviour test orgs (registry.go:20-27) and return nil from EnvExports(); opencode also returns nil but is a registered stub excluded from ValidRuntimes (registry.go:85-87) — it cannot be activated through config, so the original criterion had the two backwards. Nil-returners appear explicitly (e.g. "dummy": []) so consumers can distinguish "exports nothing" from "not represented", and the golden test asserts the manifest's runtime set equals ValidRuntimes(), so a newly added runtime fails the test until represented.

    5. Pi's full export set. Confirmed — PiRuntime.EnvExports() (internal/runtime/pi.go:145-153) exports six variables: PI_CODING_AGENT_DIR, PI_CODING_AGENT_SESSION_DIR, PI_OFFLINE, PI_SKIP_VERSION_CHECK, PI_TELEMETRY, JITI_FS_CACHE. The example manifest showed only the first for brevity; the generator parses EnvExports() output directly, so the full set appears automatically. The manifest will annotate that JITI_FS_CACHE is jiti's variable pinned by the runner, not one of pi's own, so the list is not misread.

    6. New docs path. Keeping docs/contracts/. The JSON itself needs no sidebar entry (the sidebar lists .md pages) and raw.githubusercontent.com serves it regardless of the site build. The change adds a human-readable docs/contracts/README.md describing the contract, wires the section via getMarkdownFiles("contracts", "contracts") in docs/.vitepress/config.ts, and adds "/docs/contracts/" to the Design Docs search scope prefixes (config.ts:469-482).

  6. self-assigned this
    on Oct 21, 2026
  7. rh-hemartin commented on Oct 22, 2026

    @rh-hemartin
    Member

    Adding, renaming, or reserving any new variable (e.g. FULLSEND_SKILLS_DIR, fullsend-ai/agents#1672). That's a follow-up once the contract mechanism is agreed, so any new variable arrives with its contract surface already machine-checked.

    Adding FULLSEND_SKILLS_DIR can be done before this, so don't block that on this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementfeatureFeature-category issue awaiting human prioritizationtriagedTriaged but awaiting human prioritization

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions