Merge pull request #6975 from fullsend-ai/agent/6971-scaffold-vertex-binary-allowlist

fix(#6971): allow claude.exe and pi on scaffold Vertex profile, warn on profile shadowing

Author: waynesun09

docs/contributing/runtime-implementation.md

@@ -34,6 +34,11 @@ On this page:
 5. If the runtime's binary ships in the sandbox image, pin it the way the
    existing ones are pinned and prove the pin at build time
    ([Pinned runtime binaries](#pinned-runtime-binaries-in-the-sandbox-image)).
+6. Name the process that opens the model connection in `runtimeEgressBinaries`
+   (`internal/cli/runtime_binaries_test.go`) and make sure the inference
+   profiles' `binaries:` globs match it — a runtime exec'd directly (no node
+   wrapper) is not covered by `**/node`
+   ([Egress binary identity](#egress-binary-identity-per-runtime)).
 
 ## Security feature matrix
 
@@ -55,7 +60,7 @@ flowchart TB
   end
   subgraph SB["Sandbox boundary — OpenShell + L7 egress policy (containment)"]
     direction TB
-    EG["egress allowlist: *.googleapis.com · api.anthropic.com\n(+ api.openai.com POST /v1/responses with the openai provider)\nbinaries: **/claude · **/node (pi runs via node) · **/codex"]
+    EG["egress allowlist: *.googleapis.com · api.anthropic.com\n(+ api.openai.com POST /v1/responses with the openai provider)\nbinaries: **/claude · **/claude.exe · **/node (pi runs via node) · **/pi (fleet-profile parity) · **/codex"]
     subgraph PROC["Runtime process — steering, defense in depth"]
       direction LR
       PRE["PreToolUse\nTirith · SSRF\ncanary · allowlist"]
@@ -269,6 +274,22 @@ The run log's `Agent: <model> (vX.Y.Z)` line is the ground truth for which Claud
 
 The image also creates each runtime's config directory when the binary refuses to start without one: `CODEX_HOME` (`sandbox.SandboxCodexConfig`) is created owned by the sandbox user and baked as an `ENV` default, so an ad-hoc `codex` invocation from Bash behaves like the runtime's own `EnvExports()`. `TestSandboxImageCodexDefaults` keeps the path in the image and the Go constant in step (the pi equivalent is `TestSandboxImagePiDefaults`).
 
+## Egress binary identity per runtime
+
+The inference profiles (`profiles/fullsend-vertex-ai.yaml` in this repo and in fullsend-ai/agents; `profiles/fullsend-openai.yaml`, which has no fleet copy — the runner imports the embedded scaffold) carry a `binaries:` list. OpenShell's OPA (`sandbox-policy.rego`, `binary_allowed`) matches each glob against the kernel-resolved `/proc/<pid>/exe` of the process that opens the connection **or of any of its ancestors**. That gives two kinds of runtime:
+
+- **Wrapped by node** (pi, codex): `**/node` admits them through the ancestor, so a pin bump cannot take them off the allowlist unless the wrapper disappears.
+- **Exec'd directly** (Claude Code): `claude` on PATH is a symlink to the npm package's `bin/claude.exe`, so the connecting process has no wrapper ancestor and only a glob on the real file name admits it. A renamed native binary breaks every run of that runtime on its first model call with `API Error: Error code policy_denied`, 0 tokens, and the only explanation is the gateway's `NET:OPEN … DENIED … binary '…' not allowed in policy '_provider_<name>'` line (#6971).
+
+`TestScaffoldProfilesAllowRuntimeBinaries` (`internal/cli/runtime_binaries_test.go`) fails for any selectable runtime without a mapping and pins the ones below.
+
+| Runtime | Process that opens the connection | Profile glob(s) | Evidence |
+|---------|-----------------------------------|-----------------|----------|
+| Claude Code | `bin/claude.exe` in the npm package — 2.1.2xx's `install.cjs` places the native binary there ("Always write to bin/claude.exe"); the Containerfile installs it *with* scripts so that runs | `**/claude`, `**/claude.exe` | gateway deny log in #6971; `npm view @anthropic-ai/claude-code@<pin> bin` |
+| pi | `node` (`bin = dist/bundle/cli.js`; no native network path in the package) | `**/node` | `images/sandbox/Containerfile`, `npm view @earendil-works/pi-coding-agent@<pin> bin` |
+| Codex | `vendor/<triple>/bin/codex`, spawned by the npm launcher `bin/codex.js` under node; codex also spawns `codex-code-mode-host` (default-enabled in 0.152.1), covered by ancestor matching, not by name | `**/node` (ancestor), `**/codex` (the process) | `npm pack --dry-run "@openai/codex@<pin>-linux-x64"` |
+| OpenCode (stub) | exec'd directly, like Claude Code: `opencode-ai` ships a stub `bin/opencode.exe` that `postinstall.mjs` replaces with `opencode-linux-x64/bin/opencode`; with `--ignore-scripts` the stub stays | to be decided against the Containerfile install (`**/opencode.exe` or `**/opencode`); the test fails the moment `opencode` joins `config.ValidRuntimes` until a mapping exists | `npm view opencode-ai bin optionalDependencies`, `postinstall.mjs` |
+
 ## Sandbox workspace layout
 
 The sandbox has two key directories that map to Claude Code's config levels (plus a runner-owned config directory per additional runtime, e.g. `pi-config/` for pi and `codex-config/` for codex):
@@ -916,6 +937,7 @@ Two artefacts of the run are worth knowing about:
 | `auth.command` semantics (trimmed stdout, non-zero exit fails, no env fallback) | the whole credential path | `codex-rs/login/src/auth/external_bearer.rs` |
 | `supports_websockets` default for custom providers | a true default would take traffic off `POST /v1/responses` and break the egress profile | `codex-rs/model-provider-info/src/lib.rs` |
 | `[skills.bundled]` and skill discovery | the bundled skills are disabled by the runner-owned config; a renamed key would silently bring `skill-installer` and friends back into the agent's roster | `codex-rs/config/src/skills_config.rs` |
+| The native binary's path inside the platform package (`vendor/<triple>/bin/codex` at 0.152.1) | the `fullsend-openai` profile names it as `**/codex`; the node ancestor still admits a renamed file, but the pin in `runtimeEgressBinaries` should follow the rename | `npm pack --dry-run "@openai/codex@<pin>-linux-x64"` |
 | Whether a custom provider still issues `GET /v1/models` at startup | the `fullsend-openai` egress profile denies it; if the request ever became fatal or retried, it would delay or fail every first turn | `codex-rs/models-manager/` |
 | `ConfigToml` keys and the `ReasoningEffort` enum | a renamed or removed key silently changes behaviour; `--strict-config` reports it | `codex-rs/config/src/config_toml.rs`, `codex-rs/protocol/src/openai_models.rs` |
 | JSONL event structs and rollout file naming | the stream parser and transcript extraction | `codex-rs/exec/src/exec_events.rs`, `codex-rs/thread-store/src/local/helpers.rs` |

docs/guides/user/bring-your-own-agent.md

@@ -132,7 +132,9 @@ endpoints:
     enforcement: enforce
 binaries:
   - "**/claude"
+  - "**/claude.exe"   # Claude Code 2.1.2xx runs as claude.exe, even on Linux
   - "**/node"
+  - "**/pi"
 ```
 
 > **Note (CI only):** the provider profile above controls network access only; real credentials are delivered via `host_files` (see [real-world example](#real-world-example-the-triage-agent)). Make sure you've completed the GCP prerequisites in [Before you begin](#before-you-begin).
@@ -355,6 +357,7 @@ allowed_remote_resources:
 
 | Symptom | Fix |
 |---------|-----|
+| `API Error: Error code policy_denied` on the first model call (agent exits after ~2 s, 0 tokens) | The sandbox gateway denied the agent's *binary*, not the model. Check your profile's `binaries:` list has both `**/claude` and `**/claude.exe` (Claude Code 2.1.2xx runs as `claude.exe`). To see exactly which binary was denied: `grep DENIED <run-dir>/logs/openshell-sandbox.log` — see [Debugging network policies locally](running-agents-locally.md#debugging-network-policies-locally) |
 | Agent crashes at 0s | Sandbox can't reach Vertex AI — verify that `providers/vertex-ai.yaml` is listed in your harness `providers:` and that `ANTHROPIC_VERTEX_PROJECT_ID`/`CLOUD_ML_REGION` are set (in your `--env-file` for local runs, or in the workflow `env` block for CI) |
 | "role field is required" | Add `role:` to harness |
 | `403` / "role not allowed" from the mint | Your `role:` is not one the mint serves. On the hosted mint use a built-in role (`triage`, `coder`, `review`, `retro`, `prioritize`, `fullsend`); for a custom role, point `FULLSEND_MINT_URL` at your own mint — see [Custom Agent Identity](custom-agent-identity.md) |

docs/guides/user/customizing-agents.md

@@ -135,6 +135,14 @@ When using `base:` composition, the base harness can declare its own providers a
 - **Profiles:** base + child lists are concatenated; deduplicated by profile `id` (child wins)
 - **Providers:** base + child lists are concatenated; local names shadow URL-resolved names of the same `name`
 
+If the `profiles/` directory next to the harness also contains a file with the same `id` as a profile the harness already resolves, `fullsend run` warns:
+
+```text
+  ! Profile "fullsend-vertex-ai" is defined both in /work/.fullsend/profiles and by the harness (/work/.fullsend/.fullsend-cache/resources/sha256/fe4f748d…/content); whichever copy was imported most recently is live — delete the directory copy or keep it in sync
+```
+
+Delete the directory copy unless you mean to override the harness's. A stale copy is how a fix that already landed in the harness (for example the `**/claude.exe` entry on the Vertex profile) silently stops applying.
+
 Remote URLs must include a `#sha256=...` integrity hash and match an `allowed_remote_resources` prefix in the same config. The integrity hash is checked on every resolution to ensure the content hasn't been tampered with since it was pinned.
 
 ### Tuning agents with augmentation skills

docs/guides/user/running-agents-locally.md

@@ -444,6 +444,10 @@ to the server (gateway). It is likely that you need to bind the gateway to `0.0.
 **`Syntax error: "(" unexpected` inside sandbox**
 - The macOS Mach-O binary was injected instead of a Linux ELF. Update to fullsend 0.4.0+ which auto-resolves the correct binary, or provide one explicitly with `--fullsend-binary`
 
+**`API Error: Error code policy_denied` on the first model call (agent exits after ~2 s, 0 tokens)**
+- The gateway denied the agent's binary, not the model. Run `grep DENIED <run-dir>/logs/openshell-sandbox.log`; a line ending in `binary '…/claude.exe' not allowed in policy '_provider_vertex_ai'` means the Vertex profile lacks `**/claude.exe` (Claude Code 2.1.2xx runs as `claude.exe`, even on Linux)
+- If `--fullsend-dir` contains a `profiles/` directory, its copy of the profile is imported after the harness's and is the one to fix; `fullsend run` prints a `Profile "…" is defined both in … and by the harness` warning when that happens
+
 **Agent fails with missing environment variable**
 - Check your env file contains all variables listed in the agent's harness YAML (`harness/{agent}.yaml` in the `.fullsend` config directory)
 

internal/cli/run.go

@@ -1211,6 +1211,10 @@ func runAgent(ctx context.Context, agentName, fullsendDir, outputBase, targetRep
 	// GitLab instance (#6615). Prepended so that a user-defined profile
 	// with the same ID wins via last-wins dedup. Inserted before the
 	// integrity check so providers referencing this ID are valid.
+	// generatedProfileIDs records profiles the runner synthesized itself;
+	// a profiles/ directory copy overriding one of these is the documented
+	// path, not a shadowing worth warning about.
+	generatedProfileIDs := map[string]bool{}
 	if forgePlatform == "gitlab" {
 		if profilePath, cleanupProfile, err := generateGitLabForgeProfile(); err != nil {
 			printer.StepWarn("Failed to auto-generate GitLab forge profile: " + err.Error())
@@ -1220,6 +1224,7 @@ func runAgent(ctx context.Context, agentName, fullsendDir, outputBase, targetRep
 				ID:        "fullsend-gitlab-forge",
 				LocalPath: profilePath,
 			}}, result.Profiles...)
+			generatedProfileIDs["fullsend-gitlab-forge"] = true
 		}
 	}
 
@@ -1283,8 +1288,18 @@ func runAgent(ctx context.Context, agentName, fullsendDir, outputBase, targetRep
 			printer.StepDone(fmt.Sprintf("Profile imported: %s (%.1fs)", rp.ID, time.Since(profileStart).Seconds()))
 		}
 
-		// Import provider profiles (if profiles/ directory exists).
+		// Warn when a profiles/ directory copy and a harness-resolved profile
+		// share an id. The directory import below runs after the harness
+		// import, but each has its own hash cache, so either copy can end up
+		// live on the gateway; a stale directory copy can silently undo a fix
+		// the harness already carries (#6971). Per-repo customization relies
+		// on the override, so this only makes it visible.
 		profilesDir := filepath.Join(absFullsendDir, "profiles")
+		for _, sp := range shadowedProfiles(dirProfileIDs, result.Profiles, profilesDir, generatedProfileIDs) {
+			printer.StepWarn(fmt.Sprintf("Profile %q is defined both in %s and by the harness (%s); whichever copy was imported most recently is live — delete the directory copy or keep it in sync", sp.ID, profilesDir, sp.LocalPath))
+		}
+
+		// Import provider profiles (if profiles/ directory exists).
 		dirProfileStart := time.Now()
 		printer.StepStart("Importing provider profiles")
 		if err := sandbox.ImportProfiles(profilesDir); err != nil {
@@ -5282,6 +5297,34 @@ func dedupResolvedProfiles(profiles []resolve.ResolvedProfile) []resolve.Resolve
 	return deduped
 }
 
+// shadowedProfiles returns, sorted by ID, the harness-resolved profiles
+// whose ID also appears in profilesDir. ImportProfiles(profilesDir) runs
+// after the harness-resolved imports, but the two imports keep independent
+// hash caches, so the copy imported most recently is the live one. A
+// resolved profile that already lives in profilesDir (a local-path entry,
+// ADR 0075) is the same file, not a shadow, and runner-generated profiles
+// (generatedIDs) are meant to be overridden, so both are skipped. Duplicate
+// IDs in the directory are reported once.
+func shadowedProfiles(dirIDs []string, resolved []resolve.ResolvedProfile, profilesDir string, generatedIDs map[string]bool) []resolve.ResolvedProfile {
+	byID := make(map[string]resolve.ResolvedProfile, len(resolved))
+	for _, rp := range resolved {
+		if generatedIDs[rp.ID] || (!rp.FromURL && filepath.Dir(rp.LocalPath) == profilesDir) {
+			continue
+		}
+		byID[rp.ID] = rp
+	}
+	seen := make(map[string]bool, len(dirIDs))
+	var shadowed []resolve.ResolvedProfile
+	for _, id := range dirIDs {
+		if rp, ok := byID[id]; ok && !seen[id] {
+			seen[id] = true
+			shadowed = append(shadowed, rp)
+		}
+	}
+	sort.Slice(shadowed, func(i, j int) bool { return shadowed[i].ID < shadowed[j].ID })
+	return shadowed
+}
+
 // mergeProviderDefs merges local and URL-resolved provider definitions.
 // Local defs have highest precedence; among URL-resolved defs, last
 // occurrence wins (child over base). The returned slice is deterministically

internal/cli/run_test.go

@@ -5830,6 +5830,102 @@ func TestDedupResolvedProfiles(t *testing.T) {
 	}
 }
 
+func TestShadowedProfiles(t *testing.T) {
+	const profilesDir = "/ws/.fullsend/profiles"
+	url := func(id string) resolve.ResolvedProfile {
+		return resolve.ResolvedProfile{ID: id, LocalPath: "/cache/sha256/" + id + "/content.yaml", FromURL: true}
+	}
+	ids := func(profiles []resolve.ResolvedProfile) []string {
+		var out []string
+		for _, p := range profiles {
+			out = append(out, p.ID)
+		}
+		return out
+	}
+	tests := []struct {
+		name      string
+		dirIDs    []string
+		resolved  []resolve.ResolvedProfile
+		generated map[string]bool
+		want      []string
+	}{
+		{
+			name:     "no overlap",
+			dirIDs:   []string{"local-only"},
+			resolved: []resolve.ResolvedProfile{url("remote-only")},
+			want:     nil,
+		},
+		{
+			name:     "empty dir",
+			dirIDs:   nil,
+			resolved: []resolve.ResolvedProfile{url("a")},
+			want:     nil,
+		},
+		{
+			name:     "empty resolved",
+			dirIDs:   []string{"a"},
+			resolved: nil,
+			want:     nil,
+		},
+		{
+			name:     "one shadow",
+			dirIDs:   []string{"fullsend-vertex-ai"},
+			resolved: []resolve.ResolvedProfile{url("fullsend-vertex-ai")},
+			want:     []string{"fullsend-vertex-ai"},
+		},
+		{
+			name:     "multiple shadows sorted",
+			dirIDs:   []string{"z-profile", "a-profile", "local-only"},
+			resolved: []resolve.ResolvedProfile{url("z-profile"), url("a-profile"), url("remote-only")},
+			want:     []string{"a-profile", "z-profile"},
+		},
+		{
+			name:   "local-path profile in the same directory is not a shadow",
+			dirIDs: []string{"byo"},
+			resolved: []resolve.ResolvedProfile{
+				{ID: "byo", LocalPath: profilesDir + "/byo.yaml", FromURL: false},
+			},
+			want: nil,
+		},
+		{
+			name:   "local-path profile elsewhere in the workspace is a shadow",
+			dirIDs: []string{"byo"},
+			resolved: []resolve.ResolvedProfile{
+				{ID: "byo", LocalPath: "/ws/custom/byo.yaml", FromURL: false},
+			},
+			want: []string{"byo"},
+		},
+		{
+			name:     "duplicate directory ids reported once",
+			dirIDs:   []string{"dup", "dup"},
+			resolved: []resolve.ResolvedProfile{url("dup")},
+			want:     []string{"dup"},
+		},
+		{
+			name:   "runner-generated gitlab forge profile is not a shadow",
+			dirIDs: []string{"fullsend-gitlab-forge"},
+			resolved: []resolve.ResolvedProfile{
+				{ID: "fullsend-gitlab-forge", LocalPath: "/tmp/fullsend-gitlab-profile-123/fullsend-gitlab-forge.yaml"},
+			},
+			generated: map[string]bool{"fullsend-gitlab-forge": true},
+			want:      nil,
+		},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			got := shadowedProfiles(tt.dirIDs, tt.resolved, profilesDir, tt.generated)
+			assert.Equal(t, tt.want, ids(got))
+		})
+	}
+}
+
+func TestShadowedProfiles_ReturnsResolvedCopy(t *testing.T) {
+	rp := resolve.ResolvedProfile{ID: "fullsend-vertex-ai", LocalPath: "/cache/x/content.yaml", FromURL: true}
+	got := shadowedProfiles([]string{"fullsend-vertex-ai"}, []resolve.ResolvedProfile{rp}, "/ws/profiles", nil)
+	require.Len(t, got, 1)
+	assert.Equal(t, rp, got[0], "the returned entry must carry the shadowed copy's path so the warning can name it")
+}
+
 func TestDedupResolvedProviders(t *testing.T) {
 	tests := []struct {
 		name      string