From f40ea943e7a100160df0a53a06986bece6147487 Mon Sep 17 00:00:00 2001 From: Jonathan Bragg Date: Wed, 20 May 2026 14:32:47 -0700 Subject: [PATCH] workspace: launch the dev container after scaffolding MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Before, the dev container scaffold step ended with the devcontainer.json file and "tell the user how to open it." Now the agent also runs the launch command — `make dev` for local, `git push` + `gh codespace create` for Codespaces — so the user ends up in the running container. DEVELOPER.md also gains a `gh codespace create --web` CLI mention next to the green-button Codespaces flow. Artifacts description tightened to A2A-specific phrasings ("export the task results", "convert the task output to HTML") from broad ones ("show me the artifacts", "what did the agent produce"). At v0.17.2 the broad description routes "Show me what you've put together so far" to artifacts in 2/3 epochs (per-epoch Skill calls confirm); tightening restores 3/3 workspace routing. It's also a more honest description of what artifacts actually does. README's Benchmarking section gets a version-pinning convention (`-S version=` + `ASTA_IMAGE=…@sha256:…` on both arms, recorded in the PR description as `cc · · ghcr.io/allenai/asta:`) — discovered while validating this PR that the defaults (`auto`, `:latest`) drift, breaking paired comparisons. Paired with allenai/asta-bench-private's asta_skills extension that adds an avoid_host_installs case scoring the launch step, plus outcome-level scoring (sandbox_check) so routing-only metrics no longer pass under "scaffold and stop" behavior. --- README.md | 37 ++++++++++++++++++- .../asta-preview/skills/artifacts/SKILL.md | 2 +- .../asta-preview/skills/workspace/SKILL.md | 8 ++-- .../skills/workspace/assets/DEVELOPER.md | 2 +- plugins/asta/skills/workspace/SKILL.md | 8 ++-- .../asta/skills/workspace/assets/DEVELOPER.md | 2 +- skills/artifacts/SKILL.md | 2 +- skills/workspace/SKILL.md | 8 ++-- skills/workspace/assets/DEVELOPER.md | 2 +- 9 files changed, 56 insertions(+), 15 deletions(-) diff --git a/README.md b/README.md index 10c1996..723fbb2 100644 --- a/README.md +++ b/README.md @@ -96,7 +96,42 @@ The [`workspace`](skills/workspace/SKILL.md) skill lets users see and save the a To run a benchmark, see [Demo](https://github.com/allenai/agent-baselines/tree/main/solvers/inspect-swe#demo), which runs the [`astabench`](https://github.com/allenai/asta-bench) science-agent suite with default skills. For your own edits, use [Swapping in local skills](https://github.com/allenai/agent-baselines/tree/main/solvers/inspect-swe#swapping-in-local-skills) (run `make build-plugins` first, then point at the regenerated tree). -For measuring the effect of a skill change, run a paired comparison via [Comparing two configurations](https://github.com/allenai/agent-baselines/tree/main/solvers/inspect-swe#comparing-two-configurations). See [#60](https://github.com/allenai/asta-plugins/pull/60) for a worked example against existing cases, and [#63](https://github.com/allenai/asta-plugins/pull/63) for a worked example of adding new per-skill cases. +For measuring the effect of a skill change, run a paired comparison via [Comparing two configurations](https://github.com/allenai/agent-baselines/tree/main/solvers/inspect-swe#comparing-two-configurations). Both arms must end up with the same `-S version=` and same `ASTA_IMAGE=…@sha256:…`, so a typical flow is: run baseline with defaults, capture what resolved, pin the PR arm to match. + +```bash +# 1. Run baseline arm — defaults (`:latest`, `version=auto`) are fine. +# See the linked recipe above for the full `astabench eval` command; +# swap in `-S skills=…/asta-plugins-baseline/…` and a baseline log dir. + +# 2. Capture what resolved (these get reused in step 3): +eval "$(inspect log dump logs/baseline/*.eval | jq -er '.samples[0].metadata + | "ASTA_IMAGE=\(.asta_image)\nAGENT_VERSION=\(.agent_version)"')" +export ASTA_IMAGE AGENT_VERSION + +# 3. Run the PR arm. ASTA_IMAGE is read from the env (already exported), so +# no flag is needed for the image. Add `-S version="$AGENT_VERSION"` and +# the PR branch's `-S skills=…/asta-plugins/…`, into a different log dir. + +# 4. Map the @sha256:… digest in $ASTA_IMAGE to a readable release tag +# by matching the eval timestamp against this repo's release tags: +git tag --sort=-creatordate -l 'v*' | head +``` + +Record the pins in the PR description like `claude_code 2.1.142 · sonnet-4-6 · ghcr.io/allenai/asta:v0.17.2` (`@sha256:bf92d6a2…`) — tag for readability, digest for strict reproducibility. + +See [#60](https://github.com/allenai/asta-plugins/pull/60) for a worked example against existing cases, and [#63](https://github.com/allenai/asta-plugins/pull/63) for a worked example of adding new per-skill cases. + +When a comparison includes a configuration that isn't a regular commit on a PR branch (an ablation, an A/B variant, etc.), preserve it as an annotated git tag under `experiments/PR-/` so reviewers can check it out and reproduce. Tag after the PR is open so the number is known: + +```bash +git tag -a experiments/PR-123/workspace-ablate-artifacts-tightening \ + -m "PR #123's workspace branch with skills/artifacts/SKILL.md reverted to main. Used to measure view_agent_output routing dependency on the artifacts tightening." +git push origin experiments/PR-123/workspace-ablate-artifacts-tightening +``` + +Tags survive branch deletion. Listable per-PR with `git tag -l 'experiments/PR-123/*'`. Link the tag from the PR description. + +External contributors push the tag to their fork (no write access here) and link to the fork's tag URL — same convention, different remote. ## Development diff --git a/plugins/asta-preview/skills/artifacts/SKILL.md b/plugins/asta-preview/skills/artifacts/SKILL.md index 3b4d72f..6c98af0 100644 --- a/plugins/asta-preview/skills/artifacts/SKILL.md +++ b/plugins/asta-preview/skills/artifacts/SKILL.md @@ -1,6 +1,6 @@ --- name: artifacts -description: Use this skill when the user asks to view, export, or open agent output artifacts — "show me the artifacts", "export the results", "open the report", "convert the task output to HTML", "what did the agent produce", or wants to work with structured outputs from an A2A agent task. +description: Use this skill to export and open A2A agent task outputs — "export the task results", "convert the task output to HTML", or work with the structured artifacts produced by an A2A agent task run. metadata: internal: true allowed-tools: Bash(asta * task *) Bash(asta artifacts *) Bash(asta documents *) Bash(open *) diff --git a/plugins/asta-preview/skills/workspace/SKILL.md b/plugins/asta-preview/skills/workspace/SKILL.md index 3831d13..ad4fcdb 100644 --- a/plugins/asta-preview/skills/workspace/SKILL.md +++ b/plugins/asta-preview/skills/workspace/SKILL.md @@ -37,7 +37,7 @@ Add components only when needed; don't proactively offer. |---|---| | **Quarto build tool** | Always — it's the project structure. | | **GitHub Pages deploy** | When you have no user-reachable port, or the user asks for a deployed URL. | -| **Dev container** | When the user wants to avoid installing Quarto/dependencies on the host — their own machine or anyone else's (e.g., for collaborator access without local setup). | +| **Dev container** | User wants to avoid installing host dependencies, or wants browser-only access from another machine. See subsection for the two flows. | Before writing any file in the steps below, check whether the target path already exists. If it does, ask the user before overwriting, or merge the asset's contents into the existing file. @@ -63,5 +63,7 @@ Before writing any file in the steps below, check whether the target path alread ### Dev container -1. Copy `assets/devcontainer.json` to `.devcontainer/devcontainer.json`. -2. Tell the user how to open it (Codespaces or VS Code "Reopen in Container") and authenticate Asta tools — see `DEVELOPER.md`. +Copy `assets/devcontainer.json` to `.devcontainer/devcontainer.json`, then pick the flow that matches the user's intent: + +- **Local container** (working on their machine without installs): run `make dev` to open VS Code attached to the local container. +- **Codespaces** (browser-based access from anywhere): commit, push to a GitHub remote (creating one if needed), then `gh codespace create` and give the user the URL. diff --git a/plugins/asta-preview/skills/workspace/assets/DEVELOPER.md b/plugins/asta-preview/skills/workspace/assets/DEVELOPER.md index 4f45f56..c95dc8d 100644 --- a/plugins/asta-preview/skills/workspace/assets/DEVELOPER.md +++ b/plugins/asta-preview/skills/workspace/assets/DEVELOPER.md @@ -11,7 +11,7 @@ With [Quarto](https://quarto.org) on the host: `make preview`, open `http://loca Based on `ghcr.io/allenai/asta:latest` — Quarto and [Asta](https://asta.allen.ai) pre-installed. - **VS Code locally:** `make dev`, or open folder → Command Palette → **Reopen in Container**. Needs [Docker Desktop](https://www.docker.com/products/docker-desktop/) and the [Dev Containers extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers). -- **Codespaces:** green **`<> Code`** button → **Codespaces** tab → **Create codespace on main**. Port 4848 auto-forwards. +- **Codespaces:** green **`<> Code`** button → **Codespaces** tab → **Create codespace on main**, or `gh codespace create --web` from the CLI. Port 4848 auto-forwards. Asta auth: set `ASTA_TOKEN` env var locally before launching, or `asta auth login` in the container. For Codespaces: add `ASTA_TOKEN` as a secret. diff --git a/plugins/asta/skills/workspace/SKILL.md b/plugins/asta/skills/workspace/SKILL.md index 3831d13..ad4fcdb 100644 --- a/plugins/asta/skills/workspace/SKILL.md +++ b/plugins/asta/skills/workspace/SKILL.md @@ -37,7 +37,7 @@ Add components only when needed; don't proactively offer. |---|---| | **Quarto build tool** | Always — it's the project structure. | | **GitHub Pages deploy** | When you have no user-reachable port, or the user asks for a deployed URL. | -| **Dev container** | When the user wants to avoid installing Quarto/dependencies on the host — their own machine or anyone else's (e.g., for collaborator access without local setup). | +| **Dev container** | User wants to avoid installing host dependencies, or wants browser-only access from another machine. See subsection for the two flows. | Before writing any file in the steps below, check whether the target path already exists. If it does, ask the user before overwriting, or merge the asset's contents into the existing file. @@ -63,5 +63,7 @@ Before writing any file in the steps below, check whether the target path alread ### Dev container -1. Copy `assets/devcontainer.json` to `.devcontainer/devcontainer.json`. -2. Tell the user how to open it (Codespaces or VS Code "Reopen in Container") and authenticate Asta tools — see `DEVELOPER.md`. +Copy `assets/devcontainer.json` to `.devcontainer/devcontainer.json`, then pick the flow that matches the user's intent: + +- **Local container** (working on their machine without installs): run `make dev` to open VS Code attached to the local container. +- **Codespaces** (browser-based access from anywhere): commit, push to a GitHub remote (creating one if needed), then `gh codespace create` and give the user the URL. diff --git a/plugins/asta/skills/workspace/assets/DEVELOPER.md b/plugins/asta/skills/workspace/assets/DEVELOPER.md index 4f45f56..c95dc8d 100644 --- a/plugins/asta/skills/workspace/assets/DEVELOPER.md +++ b/plugins/asta/skills/workspace/assets/DEVELOPER.md @@ -11,7 +11,7 @@ With [Quarto](https://quarto.org) on the host: `make preview`, open `http://loca Based on `ghcr.io/allenai/asta:latest` — Quarto and [Asta](https://asta.allen.ai) pre-installed. - **VS Code locally:** `make dev`, or open folder → Command Palette → **Reopen in Container**. Needs [Docker Desktop](https://www.docker.com/products/docker-desktop/) and the [Dev Containers extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers). -- **Codespaces:** green **`<> Code`** button → **Codespaces** tab → **Create codespace on main**. Port 4848 auto-forwards. +- **Codespaces:** green **`<> Code`** button → **Codespaces** tab → **Create codespace on main**, or `gh codespace create --web` from the CLI. Port 4848 auto-forwards. Asta auth: set `ASTA_TOKEN` env var locally before launching, or `asta auth login` in the container. For Codespaces: add `ASTA_TOKEN` as a secret. diff --git a/skills/artifacts/SKILL.md b/skills/artifacts/SKILL.md index 3b4d72f..6c98af0 100644 --- a/skills/artifacts/SKILL.md +++ b/skills/artifacts/SKILL.md @@ -1,6 +1,6 @@ --- name: artifacts -description: Use this skill when the user asks to view, export, or open agent output artifacts — "show me the artifacts", "export the results", "open the report", "convert the task output to HTML", "what did the agent produce", or wants to work with structured outputs from an A2A agent task. +description: Use this skill to export and open A2A agent task outputs — "export the task results", "convert the task output to HTML", or work with the structured artifacts produced by an A2A agent task run. metadata: internal: true allowed-tools: Bash(asta * task *) Bash(asta artifacts *) Bash(asta documents *) Bash(open *) diff --git a/skills/workspace/SKILL.md b/skills/workspace/SKILL.md index 3831d13..ad4fcdb 100644 --- a/skills/workspace/SKILL.md +++ b/skills/workspace/SKILL.md @@ -37,7 +37,7 @@ Add components only when needed; don't proactively offer. |---|---| | **Quarto build tool** | Always — it's the project structure. | | **GitHub Pages deploy** | When you have no user-reachable port, or the user asks for a deployed URL. | -| **Dev container** | When the user wants to avoid installing Quarto/dependencies on the host — their own machine or anyone else's (e.g., for collaborator access without local setup). | +| **Dev container** | User wants to avoid installing host dependencies, or wants browser-only access from another machine. See subsection for the two flows. | Before writing any file in the steps below, check whether the target path already exists. If it does, ask the user before overwriting, or merge the asset's contents into the existing file. @@ -63,5 +63,7 @@ Before writing any file in the steps below, check whether the target path alread ### Dev container -1. Copy `assets/devcontainer.json` to `.devcontainer/devcontainer.json`. -2. Tell the user how to open it (Codespaces or VS Code "Reopen in Container") and authenticate Asta tools — see `DEVELOPER.md`. +Copy `assets/devcontainer.json` to `.devcontainer/devcontainer.json`, then pick the flow that matches the user's intent: + +- **Local container** (working on their machine without installs): run `make dev` to open VS Code attached to the local container. +- **Codespaces** (browser-based access from anywhere): commit, push to a GitHub remote (creating one if needed), then `gh codespace create` and give the user the URL. diff --git a/skills/workspace/assets/DEVELOPER.md b/skills/workspace/assets/DEVELOPER.md index 4f45f56..c95dc8d 100644 --- a/skills/workspace/assets/DEVELOPER.md +++ b/skills/workspace/assets/DEVELOPER.md @@ -11,7 +11,7 @@ With [Quarto](https://quarto.org) on the host: `make preview`, open `http://loca Based on `ghcr.io/allenai/asta:latest` — Quarto and [Asta](https://asta.allen.ai) pre-installed. - **VS Code locally:** `make dev`, or open folder → Command Palette → **Reopen in Container**. Needs [Docker Desktop](https://www.docker.com/products/docker-desktop/) and the [Dev Containers extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers). -- **Codespaces:** green **`<> Code`** button → **Codespaces** tab → **Create codespace on main**. Port 4848 auto-forwards. +- **Codespaces:** green **`<> Code`** button → **Codespaces** tab → **Create codespace on main**, or `gh codespace create --web` from the CLI. Port 4848 auto-forwards. Asta auth: set `ASTA_TOKEN` env var locally before launching, or `asta auth login` in the container. For Codespaces: add `ASTA_TOKEN` as a secret.