Skip to content

Commit e75bc41

Browse files
committed
test(profiles): pin the egress binary each runtime needs on its profile
The claude.exe gap (#6971) was a class of failure, not a one-off: the gateway's OPA matches a profile's binaries: glob against the kernel-resolved exe of the process that opens the connection, so every runtime has to name the file that actually runs, and a runtime pin bump or a new runtime can silently break that. Cover the other runtimes the same way: - codex: bin/codex.js spawns vendor/<triple>/bin/codex (verified with npm pack --dry-run on the 0.152.1 pin), matched by the **/codex the OpenAI profile already carries. - pi: an npm package executed by node, matched by **/node. - opencode: follows the claude.exe pattern (opencode-ai ships a stub bin/opencode.exe that postinstall replaces with the platform binary, opencode-linux-x64/bin/opencode). The runtime is still a stub and not in the sandbox image, so nothing is added to the profiles; the mapping is pre-declared so the test fails the moment opencode joins config.ValidRuntimes until both inference profiles allow **/opencode and **/opencode.exe. TestScaffoldProfilesAllowRuntimeBinaries walks config.ValidRuntimes and requires each selectable runtime to have a declared mapping whose globs are present in the scaffold profiles it uses; the runtime matrix in docs/contributing/runtime-implementation.md gains an "egress binary identity" row with the same facts and the verification recipe. Refs #6971 Assisted-by: Claude (code, review), Grok (review) Signed-off-by: Wayne Sun <gsun@redhat.com>
1 parent 089e0ef commit e75bc41

3 files changed

Lines changed: 127 additions & 0 deletions

File tree

‎docs/contributing/runtime-implementation.md‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,11 @@ On this page:
3434
5. If the runtime's binary ships in the sandbox image, pin it the way the
3535
existing ones are pinned and prove the pin at build time
3636
([Pinned runtime binaries](#pinned-runtime-binaries-in-the-sandbox-image)).
37+
6. Name the process that opens the model connection in `runtimeEgressBinaries`
38+
(`internal/cli/runtime_binaries_test.go`) and make sure the inference
39+
profiles' `binaries:` globs match it — a runtime exec'd directly (no node
40+
wrapper) is not covered by `**/node`
41+
([Egress binary identity](#egress-binary-identity-per-runtime)).
3742

3843
## Security feature matrix
3944

@@ -269,6 +274,22 @@ The run log's `Agent: <model> (vX.Y.Z)` line is the ground truth for which Claud
269274

270275
The image also creates each runtime's config directory when the binary refuses to start without one: `CODEX_HOME` (`sandbox.SandboxCodexConfig`) is created owned by the sandbox user and baked as an `ENV` default, so an ad-hoc `codex` invocation from Bash behaves like the runtime's own `EnvExports()`. `TestSandboxImageCodexDefaults` keeps the path in the image and the Go constant in step (the pi equivalent is `TestSandboxImagePiDefaults`).
271276

277+
## Egress binary identity per runtime
278+
279+
The inference profiles (`profiles/fullsend-vertex-ai.yaml` in this repo and in fullsend-ai/agents; `profiles/fullsend-openai.yaml`, which has no fleet copy — the runner imports the embedded scaffold) carry a `binaries:` list. OpenShell's OPA (`sandbox-policy.rego`, `binary_allowed`) matches each glob against the kernel-resolved `/proc/<pid>/exe` of the process that opens the connection **or of any of its ancestors**. That gives two kinds of runtime:
280+
281+
- **Wrapped by node** (pi, codex): `**/node` admits them through the ancestor, so a pin bump cannot take them off the allowlist unless the wrapper disappears.
282+
- **Exec'd directly** (Claude Code): `claude` on PATH is a symlink to the npm package's `bin/claude.exe`, so the connecting process has no wrapper ancestor and only a glob on the real file name admits it. A renamed native binary breaks every run of that runtime on its first model call with `API Error: Error code policy_denied`, 0 tokens, and the only explanation is the gateway's `NET:OPEN … DENIED … binary '…' not allowed in policy '_provider_<name>'` line (#6971).
283+
284+
`TestScaffoldProfilesAllowRuntimeBinaries` (`internal/cli/runtime_binaries_test.go`) fails for any selectable runtime without a mapping and pins the ones below.
285+
286+
| Runtime | Process that opens the connection | Profile glob(s) | Evidence |
287+
|---------|-----------------------------------|-----------------|----------|
288+
| Claude Code | `bin/claude.exe` in the npm package — 2.1.2xx's `install.cjs` places the native binary there ("Always write to bin/claude.exe"); the Containerfile installs it *with* scripts so that runs | `**/claude`, `**/claude.exe` | gateway deny log in #6971; `npm view @anthropic-ai/claude-code@<pin> bin` |
289+
| pi | `node` (`bin = dist/bundle/cli.js`; no native network path in the package) | `**/node` | `images/sandbox/Containerfile`, `npm view @earendil-works/pi-coding-agent@<pin> bin` |
290+
| Codex | `vendor/<triple>/bin/codex`, spawned by the npm launcher `bin/codex.js` under node; codex also spawns `codex-code-mode-host` (default-enabled in 0.152.1), covered by ancestor matching, not by name | `**/node` (ancestor), `**/codex` (the process) | `npm pack --dry-run "@openai/codex@<pin>-linux-x64"` |
291+
| OpenCode (stub) | exec'd directly, like Claude Code: `opencode-ai` ships a stub `bin/opencode.exe` that `postinstall.mjs` replaces with `opencode-linux-x64/bin/opencode`; with `--ignore-scripts` the stub stays | to be decided against the Containerfile install (`**/opencode.exe` or `**/opencode`); the test fails the moment `opencode` joins `config.ValidRuntimes` until a mapping exists | `npm view opencode-ai bin optionalDependencies`, `postinstall.mjs` |
292+
272293
## Sandbox workspace layout
273294

274295
The sandbox has two key directories that map to Claude Code's config levels (plus a runner-owned config directory per additional runtime, e.g. `pi-config/` for pi and `codex-config/` for codex):
@@ -916,6 +937,7 @@ Two artefacts of the run are worth knowing about:
916937
| `auth.command` semantics (trimmed stdout, non-zero exit fails, no env fallback) | the whole credential path | `codex-rs/login/src/auth/external_bearer.rs` |
917938
| `supports_websockets` default for custom providers | a true default would take traffic off `POST /v1/responses` and break the egress profile | `codex-rs/model-provider-info/src/lib.rs` |
918939
| `[skills.bundled]` and skill discovery | the bundled skills are disabled by the runner-owned config; a renamed key would silently bring `skill-installer` and friends back into the agent's roster | `codex-rs/config/src/skills_config.rs` |
940+
| The native binary's path inside the platform package (`vendor/<triple>/bin/codex` at 0.152.1) | the `fullsend-openai` profile names it as `**/codex`; the node ancestor still admits a renamed file, but the pin in `runtimeEgressBinaries` should follow the rename | `npm pack --dry-run "@openai/codex@<pin>-linux-x64"` |
919941
| Whether a custom provider still issues `GET /v1/models` at startup | the `fullsend-openai` egress profile denies it; if the request ever became fatal or retried, it would delay or fail every first turn | `codex-rs/models-manager/` |
920942
| `ConfigToml` keys and the `ReasoningEffort` enum | a renamed or removed key silently changes behaviour; `--strict-config` reports it | `codex-rs/config/src/config_toml.rs`, `codex-rs/protocol/src/openai_models.rs` |
921943
| JSONL event structs and rollout file naming | the stream parser and transcript extraction | `codex-rs/exec/src/exec_events.rs`, `codex-rs/thread-store/src/local/helpers.rs` |
Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
package cli
2+
3+
import (
4+
"strings"
5+
"testing"
6+
7+
"github.com/stretchr/testify/assert"
8+
"github.com/stretchr/testify/require"
9+
"gopkg.in/yaml.v3"
10+
11+
"github.com/fullsend-ai/fullsend/internal/config"
12+
"github.com/fullsend-ai/fullsend/internal/scaffold"
13+
)
14+
15+
// runtimeEgressBinaries names, per runtime, the profile binaries: globs the
16+
// sandbox gateway must carry for that runtime's model calls. OpenShell's OPA
17+
// (sandbox-policy.rego, binary_allowed) matches each glob against the
18+
// kernel-resolved /proc/<pid>/exe of the process that opens the connection
19+
// OR of any of its ancestors. That makes two kinds of runtime:
20+
//
21+
// - Wrapped by node: pi is a script run by node, and codex's npm launcher
22+
// bin/codex.js spawns the native vendor/<triple>/bin/codex under node.
23+
// **/node admits both through the ancestor; **/codex names the process
24+
// itself. A pin bump cannot break these unless the wrapper goes away.
25+
// - Exec'd directly: `claude` on PATH is a symlink to the npm package's
26+
// bin/claude.exe (2.1.2xx places the native binary there — install.cjs,
27+
// "Always write to bin/claude.exe"), so the connecting process has no
28+
// wrapper ancestor and only a glob on the real file name admits it. This
29+
// is what broke every 0.40.0-image run (fullsend#6971): **/claude alone
30+
// never matched.
31+
//
32+
// A new runtime must add its mapping here; TestScaffoldProfilesAllowRuntimeBinaries
33+
// fails for any selectable runtime without one. Decide the globs against the
34+
// actual Containerfile install (opencode, still a stub, follows the claude.exe
35+
// pattern — see the note on OpenCodeRuntime). The two exact-list tests
36+
// (TestScaffoldVertexProfile_BinaryAllowlist, TestEmbeddedOpenAIProfileBinaries)
37+
// pin the full lists; the walk over config.ValidRuntimes is what this test adds.
38+
var runtimeEgressBinaries = map[string]map[string][]string{
39+
"claude": {
40+
"fullsend-vertex-ai": {"**/claude", "**/claude.exe"},
41+
},
42+
"pi": {
43+
"fullsend-vertex-ai": {"**/node"},
44+
"fullsend-openai": {"**/node"},
45+
},
46+
"codex": {
47+
"fullsend-openai": {"**/node", "**/codex"},
48+
},
49+
}
50+
51+
func scaffoldProfileBinaries(t *testing.T, id string) []string {
52+
t.Helper()
53+
data, err := scaffold.FullsendRepoFile("profiles/" + id + ".yaml")
54+
require.NoError(t, err, "scaffold profile %s", id)
55+
var profile struct {
56+
ID string `yaml:"id"`
57+
Binaries []string `yaml:"binaries"`
58+
}
59+
require.NoError(t, yaml.Unmarshal(data, &profile))
60+
require.Equal(t, id, profile.ID)
61+
return profile.Binaries
62+
}
63+
64+
// TestScaffoldProfilesAllowRuntimeBinaries fails when a selectable runtime
65+
// has no declared egress binary, or when a scaffold profile it uses lacks
66+
// one of them — the gap that let every 0.40.0-image Claude run die with
67+
// "API Error: Error code policy_denied" (fullsend#6971, #6962).
68+
func TestScaffoldProfilesAllowRuntimeBinaries(t *testing.T) {
69+
for _, rt := range config.ValidRuntimes() {
70+
if strings.HasPrefix(rt, "dummy") {
71+
continue // no sandbox process, no egress
72+
}
73+
profiles, ok := runtimeEgressBinaries[rt]
74+
require.True(t, ok, "runtime %q is selectable but has no runtimeEgressBinaries entry: name the binary its model calls come from and the profiles that must allow it", rt)
75+
for id, globs := range profiles {
76+
t.Run(rt+"/"+id, func(t *testing.T) {
77+
have := scaffoldProfileBinaries(t, id)
78+
for _, g := range globs {
79+
assert.Contains(t, have, g, "profile %s must allowlist %s for the %s runtime", id, g, rt)
80+
}
81+
})
82+
}
83+
}
84+
}
85+
86+
// TestRuntimeEgressBinaries_OnlyKnownRuntimes keeps the table from drifting
87+
// into runtime names nothing can select.
88+
func TestRuntimeEgressBinaries_OnlyKnownRuntimes(t *testing.T) {
89+
known := map[string]bool{}
90+
for _, rt := range config.ValidRuntimes() {
91+
known[rt] = true
92+
}
93+
for rt := range runtimeEgressBinaries {
94+
assert.True(t, known[rt], "runtimeEgressBinaries names %q, which config.ValidRuntimes does not list", rt)
95+
}
96+
}

‎internal/runtime/opencode.go‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,15 @@ import (
1414
// interfaces for the OpenCode agent runtime. All methods are no-ops or return
1515
// not-implemented errors. Subsequent PRs will fill in stream parsing, bootstrap,
1616
// run execution, and transcript extraction.
17+
//
18+
// Egress note for whoever lands it: opencode is exec'd directly, like Claude
19+
// Code, not wrapped by node. The opencode-ai npm package ships bin/opencode.exe
20+
// as a shell stub that postinstall.mjs replaces (link or copy) with the
21+
// platform binary opencode-linux-x64/bin/opencode; with --ignore-scripts the
22+
// stub stays. Whichever file the Containerfile ends up exec'ing is the name
23+
// the inference profiles' binaries: globs must carry (**/opencode.exe or
24+
// **/opencode), and runtimeEgressBinaries in internal/cli must list it, or
25+
// every run dies on its first model call with policy_denied (fullsend#6971).
1726
type OpenCodeRuntime struct{}
1827

1928
func (OpenCodeRuntime) Name() string { return "opencode" }

0 commit comments

Comments
 (0)