Skip to content

Commit 17cf2fc

Browse files
envs/opencode_env: OpenCode coding-agent harness environment (#656)
* feat(opencode_env): OpenCode coding-agent harness environment Re-introduces the OpenCode env that was developed in PR #603 (merged into the stacked feature/harness-interface branch and dropped before that branch landed on main). With openenv-core 0.3.0 now published with the harness runtime, this lands cleanly on main without the stack dependency. Environment - envs/opencode_env/: deployable env wrapping the OpenCode CLI agent in an E2B sandbox against any OpenAI-compatible LLM endpoint. - Single MCP tool run_rollout (instruction + setup[] + verify[] bash; reward = passed_verify / total or override via reward.txt). - Endpoint shorthand catalog (vllm / openai / hf_router) resolves base_url / api_key / model from env vars. - transparent_proxy mode runs an in-sandbox FastAPI proxy that injects logprobs=true on forwarded /v1/chat/completions, captures per-turn logprobs to a JSON-lines trace (for GRPO), and strips logprobs from what opencode sees. - black_box mode skips the proxy for SFT / eval rollouts. - Pre-baked E2B template support drops sandbox cold start ~2min -> ~6s. - Streaming Gradio UI at /web with live phase log. - Implements ResourceSession / ResourceSessionFactory from openenv.core.harness. Pinned to openenv-core[core]>=0.3.0 (no more git-ref pin to a fork). Tests - tests/envs/test_opencode_env.py: 14 offline unit tests covering catalog resolution, model serialization, task coercion. No E2B / LLM / network. Plus one integration test gated on a deployed Space. Example - examples/opencode_env_simple.py: end-to-end binary_search rollout against the deployed HF Space, prints reward + per-turn logprobs. CI - .github/workflows/docker-build.yml: adds opencode-env to the matrix so the image is built and pushed to GHCR alongside other envs. * fix: address opencode env ci lint --------- Co-authored-by: burtenshaw <ben.burtenshaw@gmail.com>
1 parent 83711aa commit 17cf2fc

29 files changed

Lines changed: 7815 additions & 0 deletions

.github/workflows/docker-build.yml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -105,6 +105,9 @@ jobs:
105105
- name: unity-env
106106
dockerfile: envs/unity_env/server/Dockerfile
107107
context: envs/unity_env
108+
- name: opencode-env
109+
dockerfile: envs/opencode_env/server/Dockerfile
110+
context: envs/opencode_env
108111
- name: openapp-env
109112
dockerfile: envs/openapp_env/server/Dockerfile
110113
context: envs/openapp_env

docs/source/environments.md

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -549,6 +549,20 @@ AgentWorldModel-1K — 1,000 synthetic MCP tool-use environments with 10,000 tas
549549
```
550550
````
551551
552+
````{grid-item-card} Opencode
553+
:class-card: sd-border-1
554+
555+
`opencode_env` runs the OpenCode coding agent inside an isolated E2B sandbox against any OpenAI-compatible LLM endpoint, optionally capturing per-token logpr...
556+
557+
+++
558+
```{button-link} environments/opencode.html
559+
:color: primary
560+
:outline:
561+
562+
{octicon}`file;1em` Docs
563+
```
564+
````
565+
552566
`````
553567

554568
```{tip}
@@ -619,4 +633,5 @@ environments/tbench2
619633
environments/unity
620634
environments/wildfire
621635
environments/agent_world_model
636+
environments/opencode
622637
```
Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
```{include} ../../../envs/opencode_env/README.md
2+
```

envs/opencode_env/.dockerignore

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
# Don't bloat the build context. Copy only what the runtime needs.
2+
.venv
3+
__pycache__
4+
**/__pycache__
5+
*.pyc
6+
*.pyo
7+
.pytest_cache
8+
.mypy_cache
9+
.ruff_cache
10+
11+
# NEVER ship secrets into the image. The container reads E2B_API_KEY etc.
12+
# from runtime env vars (HF Space secrets, ``docker run -e``).
13+
.env
14+
.env.*
15+
16+
# Test artifacts and local debug output.
17+
tests/_artifacts/
18+
*.log
19+
.coverage
20+
htmlcov/
21+
22+
# Editor / IDE noise.
23+
.idea
24+
.vscode
25+
.DS_Store
26+
27+
# Git metadata is unused at runtime.
28+
.git
29+
.gitignore

envs/opencode_env/.gitignore

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
# Tests are dev-only — never ship into the deployable env.
2+
tests/
3+
4+
# Local debug artifacts (covered above, kept for clarity).
5+
*.log
6+
7+
# Local secrets / venv (also covered at repo root, kept here for safety).
8+
.env
9+
.env.*
10+
.venv
11+
__pycache__/
12+
*.pyc

envs/opencode_env/README.md

Lines changed: 294 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,294 @@
1+
---
2+
title: OpenCode Environment Server
3+
emoji: 🛠️
4+
colorFrom: indigo
5+
colorTo: purple
6+
sdk: docker
7+
pinned: false
8+
app_port: 8000
9+
base_path: /web
10+
tags:
11+
- openenv
12+
short_description: OpenCode coding agent in an E2B sandbox with logprob capture
13+
---
14+
15+
# OpenCode Environment for OpenEnv
16+
17+
`opencode_env` runs the [OpenCode](https://opencode.ai) coding agent inside
18+
an isolated [E2B](https://e2b.dev) sandbox against any OpenAI-compatible
19+
LLM endpoint, optionally capturing per-token logprobs for GRPO training.
20+
21+
**🚀 Try it live**: [`AdithyaSK/opencode-env`](https://huggingface.co/spaces/AdithyaSK/opencode-env)
22+
23+
The deployed Space exposes:
24+
25+
- **Web UI** at [`/web`](https://adithyask-opencode-env.hf.space/web) — pick endpoint, write task, hit Run, watch live phase log + reward + logprobs.
26+
- **MCP tool API** at [`/mcp`](https://adithyask-opencode-env.hf.space/mcp) — programmatic `run_rollout` calls.
27+
- **OpenAPI docs** at [`/docs`](https://adithyask-opencode-env.hf.space/docs).
28+
- **Health** at [`/health`](https://adithyask-opencode-env.hf.space/health).
29+
30+
The env is **task-agnostic** — every rollout is configured at call-time
31+
with a uniform Task shape:
32+
33+
- **`instruction`** — prompt for the agent
34+
- **`setup`** — list of bash commands run *before* the agent (pip
35+
install, git clone, file downloads — anything you need staged in the
36+
sandbox)
37+
- **`verify`** — list of bash commands run *after* the agent (asserts,
38+
pytest invocations, score-file writes)
39+
40+
Reward = `passed_verify / total_verify` unless any `verify` command writes
41+
a float to `/home/user/logs/verifier/reward.txt` (override).
42+
43+
## Quick Start
44+
45+
### Async (default — talk to the deployed Space)
46+
47+
```python
48+
import asyncio
49+
import os
50+
from opencode_env import OpenCodeEnv
51+
from opencode_env.client import _extract_text
52+
from opencode_env.models import RolloutResult
53+
54+
55+
async def main():
56+
SPACE = "https://adithyask-opencode-env.hf.space"
57+
58+
async with OpenCodeEnv(base_url=SPACE) as env:
59+
await env.reset()
60+
61+
# The MCP tool returns JSON; deserialize via the typed model.
62+
raw = await env.call_tool(
63+
"run_rollout",
64+
endpoint="openai", # vllm | openai | hf_router
65+
api_key=os.environ["OPENAI_API_KEY"], # or set as a Space secret
66+
instruction=(
67+
"Create binary_search.py exposing def binary_search(arr, target) -> int "
68+
"that returns the index of target in arr, or -1 if absent. Use a "
69+
"relative path."
70+
),
71+
setup=[],
72+
verify=[
73+
"test -f /home/user/workdir/binary_search.py",
74+
"python -c \"import sys; sys.path.insert(0, '/home/user/workdir'); "
75+
"import binary_search; "
76+
"assert binary_search.binary_search([1,2,3], 2) == 1; print('OK')\"",
77+
],
78+
template="opencode-rl", # prebaked E2B template
79+
task_id="binary_search_v1",
80+
)
81+
result = RolloutResult.model_validate_json(_extract_text(raw))
82+
83+
print("reward:", result.reward)
84+
print("turns:", len(result.proxy_turns))
85+
print("files:", list(result.files.keys()))
86+
print("wall:", result.wall_s, "s")
87+
88+
89+
asyncio.run(main())
90+
```
91+
92+
Expected output (~20s with the prebaked template):
93+
94+
```
95+
reward: 1.0
96+
turns: 3
97+
files: ['/home/user/workdir/binary_search.py', ...]
98+
wall: 19.8 s
99+
```
100+
101+
### Sync wrapper
102+
103+
```python
104+
import os
105+
from opencode_env import OpenCodeEnv
106+
107+
# .sync() returns a synchronous wrapper around the async client.
108+
with OpenCodeEnv(base_url="https://adithyask-opencode-env.hf.space").sync() as env:
109+
env.reset()
110+
# MCP tools are reachable via env.call_tool(...) / env.step(...) sync-wrapped.
111+
# See the async example above for the full run_rollout signature.
112+
```
113+
114+
Point `base_url` at `http://localhost:8000` to talk to a local container
115+
instead of the public Space.
116+
117+
### In-process primitive (no HTTP)
118+
119+
For trainers that want to drive a sandbox directly without an HTTP boundary:
120+
121+
```python
122+
import os
123+
from opencode_env import (
124+
OpenCodeConfig, OpenCodeSessionFactory, OpenCodeTask, E2BSandboxBackend,
125+
)
126+
127+
factory = OpenCodeSessionFactory(
128+
config=OpenCodeConfig(
129+
provider="openai_compatible",
130+
base_url="https://api.openai.com/v1",
131+
api_key=os.environ["OPENAI_API_KEY"],
132+
model="gpt-4o-mini",
133+
),
134+
sandbox_backend=E2BSandboxBackend(),
135+
mode="transparent_proxy", # captures per-token logprobs
136+
)
137+
session = factory.create(task=OpenCodeTask(instruction="..."))
138+
session.wait_for_completion()
139+
turns = session.fetch_proxy_trace() # per-turn (tokens, logprobs)
140+
session.close()
141+
```
142+
143+
## Building the Docker Image
144+
145+
The Dockerfile lives at `server/Dockerfile`. Use the `openenv` CLI from
146+
the env root:
147+
148+
```bash
149+
cd envs/opencode_env
150+
151+
openenv validate # check pyproject.toml + openenv.yaml + server/app.py + uv.lock
152+
openenv build -t opencode-env # builds the image (uses server/Dockerfile)
153+
154+
# run locally with E2B credentials
155+
docker run -p 8000:8000 -e E2B_API_KEY=e2b_... opencode-env
156+
157+
# push to HF Spaces (Docker variant)
158+
openenv push --repo-id <user>/opencode-env
159+
```
160+
161+
Or build directly without the CLI:
162+
163+
```bash
164+
docker build -t opencode-env -f envs/opencode_env/server/Dockerfile envs/opencode_env
165+
```
166+
167+
The image:
168+
169+
- Runs `uvicorn server.app:app --host 0.0.0.0 --port 8000`
170+
- Exposes the MCP API at `/mcp` and `/step`, the Gradio UI at `/web`,
171+
health at `/health`, and OpenAPI docs at `/docs`.
172+
- Reads `E2B_API_KEY` and (optionally) endpoint-specific env vars at
173+
runtime (see [Environment Variables](#environment-variables)).
174+
175+
## The MCP Tool: `run_rollout`
176+
177+
Single tool, two ways to specify the LLM endpoint:
178+
179+
**Option A — endpoint shorthand (recommended)**: pass
180+
`endpoint="vllm"` (or `"openai"` / `"hf_router"`). The server resolves
181+
`base_url`, `api_key`, and `model` from env vars + catalog defaults.
182+
Any explicit field overrides the catalog.
183+
184+
**Option B — fully explicit**: pass `base_url` + `api_key` + `model`
185+
directly.
186+
187+
| Arg | Type | Default | Notes |
188+
|---|---|---|---|
189+
| `endpoint` | `str` | `""` | One of `"vllm"` / `"openai"` / `"hf_router"`. |
190+
| `base_url` / `api_key` / `model` | `str` | `""` | Override / supply explicitly. |
191+
| `instruction` | `str` | required | Prompt passed to `opencode run`. |
192+
| `setup` | `list[str]` | `[]` | Bash commands run **before** the agent. |
193+
| `verify` | `list[str]` | `[]` | Bash commands run **after** the agent. |
194+
| `task_id` | `str` | `""` | Echoed back in result. |
195+
| `mode` | `str` | `"transparent_proxy"` | Or `"black_box"` (no logprobs). |
196+
| `disable_thinking` | `bool \| None` | `None` (catalog default) | Inject `chat_template_kwargs.enable_thinking=false`. |
197+
| `max_tokens_cap` | `int` | `4096` | Per-turn `max_tokens` clamp. |
198+
| `top_logprobs` | `int` | `5` | HF Router cap is 5; OpenAI 0–20; vLLM unbounded. |
199+
| `agent_timeout_s` | `float` | `600.0` | Hard wall budget for opencode. |
200+
| `template` | `str` | `""` | E2B template name; `"opencode-rl"` skips ~2 min of install per rollout. |
201+
202+
Returns `RolloutResult` JSON with: `reward`, `setup_results[]`,
203+
`verify_results[]`, `proxy_turns[]`, `files{}`, `agent_log_tail`,
204+
`proxy_log_tail`, `wall_s`, `agent_exit_code`, `sandbox_id`, `error`.
205+
206+
## Two Operating Modes
207+
208+
| Mode | What it does | Best for |
209+
|---|---|---|
210+
| **`transparent_proxy`** (default) | In-sandbox proxy at `localhost:7000` forwards opencode's LLM calls to `base_url`, injects `logprobs=true`, captures per-turn `(messages, completion_tokens, logprobs)` to `proxy_trace.jsonl`. | GRPO / RL training, observability, top-k distillation. |
211+
| **`black_box`** | No proxy. opencode talks straight to `base_url`. | Smoke tests, eval, SFT data collection. |
212+
213+
## Environment Variables
214+
215+
The server reads these at runtime. Local dev auto-loads them from a
216+
sibling `.env` file; on HF Spaces, set them as **Space secrets**.
217+
218+
| Variable | Required | Purpose |
219+
|---|---|---|
220+
| `E2B_API_KEY` | **yes** for any rollout | E2B sandbox credentials. |
221+
| `MAX_CONCURRENT_ENVS` | no | Env-instance pool size. Default `4`. |
222+
| `ENABLE_WEB_INTERFACE` | no | Set `false` to disable the `/web` Gradio mount. Default `true`. |
223+
| **vLLM endpoint** | | |
224+
| `VLLM_URL` | required for `endpoint="vllm"` | OAI-compatible base URL. |
225+
| `VLLM_API_KEY` | no | Defaults to `intercepted`. |
226+
| `VLLM_MODEL` | no | Defaults to `Qwen/Qwen3.5-4B`. |
227+
| **OpenAI endpoint** | | |
228+
| `OPENAI_API_KEY` | required for `endpoint="openai"` | Standard OpenAI key. |
229+
| `OPENAI_BASE_URL` | no | Defaults to `https://api.openai.com/v1`. |
230+
| `OPENAI_MODEL` | no | Defaults to `gpt-4o-mini` (gpt-5.x and o-series refuse logprobs). |
231+
| **HF Router endpoint** | | |
232+
| `HF_ROUTER_API_KEY` | required for `endpoint="hf_router"` | HF user token. |
233+
| `HF_ROUTER_BASE_URL` | no | Defaults to `https://router.huggingface.co/v1`. |
234+
| `HF_ROUTER_MODEL` | no | Defaults to `Qwen/Qwen3-4B-Instruct-2507:nscale`. |
235+
236+
Pick `provider:` suffixes that actually return logprobs:
237+
**Together / Nscale / Scaleway / SambaNova / Cerebras**. Avoid Novita /
238+
Hyperbolic / Featherless (silent drop) and Groq (HTTP 400).
239+
240+
## Pre-baked E2B Template
241+
242+
The first rollout in a fresh E2B sandbox spends ~2 min installing
243+
opencode and the proxy's Python deps. Build a one-time template that
244+
ships those pre-installed:
245+
246+
```bash
247+
.venv/bin/python envs/opencode_env/sandbox/build_template.py
248+
# → builds `opencode-rl` template in your E2B account (~1m20s, one-time)
249+
```
250+
251+
After this, pass `template="opencode-rl"` on every `run_rollout` call —
252+
each rollout drops to ~20–30s end-to-end.
253+
254+
## Project Structure
255+
256+
```
257+
opencode_env/
258+
├── README.md # this file
259+
├── openenv.yaml # OpenEnv space spec
260+
├── pyproject.toml # deps + ``server`` entrypoint
261+
├── uv.lock # frozen deps (required by ``openenv validate``)
262+
├── .gitignore / .dockerignore # excludes .env / __pycache__
263+
├── __init__.py # re-exports primitive + client + models
264+
265+
├── client.py # OpenCodeEnv(MCPToolClient)
266+
├── models.py # RolloutResult / RolloutTurn / OpenCodeState
267+
268+
├── config.py # OpenCodeConfig (primitive)
269+
├── harness.py # OpenCodeSession / OpenCodeSessionFactory (CLI-only)
270+
├── opencode_runtime.py # opencode.json builder + cmds
271+
├── task.py # OpenCodeTask
272+
273+
├── server/
274+
│ ├── __init__.py
275+
│ ├── app.py # FastAPI factory; mounts Gradio at /web
276+
│ ├── opencode_environment.py # MCPEnvironment with single ``run_rollout`` tool
277+
│ ├── gradio_ui.py # the /web Gradio Blocks UI
278+
│ ├── catalog.py # endpoint shorthand resolver
279+
│ └── Dockerfile # multi-stage uv build (used by ``openenv build``)
280+
281+
└── sandbox/
282+
├── __init__.py
283+
├── base.py # SandboxBackend / SandboxHandle Protocols
284+
├── e2b.py # E2B implementation
285+
├── interception.py # in-sandbox FastAPI proxy (logprob capture)
286+
└── build_template.py # one-time E2B template builder
287+
```
288+
289+
## References
290+
291+
- [OpenEnv docs](https://meta-pytorch.org/OpenEnv/)
292+
- [OpenCode CLI](https://opencode.ai/docs/cli/)
293+
- [E2B Python SDK](https://e2b.dev/docs)
294+
- [HF Inference Providers logprob matrix](../../../DOCS/HF/hf_inference_providers_logprobs.md)

0 commit comments

Comments
 (0)