Skip to content

Commit aa37fcf

Browse files
committed
improve-skills: skill for fixing or reporting undesirable skill behavior
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
1 parent f40ea94 commit aa37fcf

13 files changed

Lines changed: 299 additions & 36 deletions

File tree

README.md

Lines changed: 1 addition & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -96,42 +96,7 @@ The [`workspace`](skills/workspace/SKILL.md) skill lets users see and save the a
9696

9797
To run a benchmark, see [Demo](https://github.com/allenai/agent-baselines/tree/main/solvers/inspect-swe#demo), which runs the [`astabench`](https://github.com/allenai/asta-bench) science-agent suite with default skills. For your own edits, use [Swapping in local skills](https://github.com/allenai/agent-baselines/tree/main/solvers/inspect-swe#swapping-in-local-skills) (run `make build-plugins` first, then point at the regenerated tree).
9898

99-
For measuring the effect of a skill change, run a paired comparison via [Comparing two configurations](https://github.com/allenai/agent-baselines/tree/main/solvers/inspect-swe#comparing-two-configurations). Both arms must end up with the same `-S version=<cc-ver>` and same `ASTA_IMAGE=…@sha256:…`, so a typical flow is: run baseline with defaults, capture what resolved, pin the PR arm to match.
100-
101-
```bash
102-
# 1. Run baseline arm — defaults (`:latest`, `version=auto`) are fine.
103-
# See the linked recipe above for the full `astabench eval` command;
104-
# swap in `-S skills=…/asta-plugins-baseline/…` and a baseline log dir.
105-
106-
# 2. Capture what resolved (these get reused in step 3):
107-
eval "$(inspect log dump logs/baseline/*.eval | jq -er '.samples[0].metadata
108-
| "ASTA_IMAGE=\(.asta_image)\nAGENT_VERSION=\(.agent_version)"')"
109-
export ASTA_IMAGE AGENT_VERSION
110-
111-
# 3. Run the PR arm. ASTA_IMAGE is read from the env (already exported), so
112-
# no flag is needed for the image. Add `-S version="$AGENT_VERSION"` and
113-
# the PR branch's `-S skills=…/asta-plugins/…`, into a different log dir.
114-
115-
# 4. Map the @sha256:… digest in $ASTA_IMAGE to a readable release tag
116-
# by matching the eval timestamp against this repo's release tags:
117-
git tag --sort=-creatordate -l 'v*' | head
118-
```
119-
120-
Record the pins in the PR description like `claude_code 2.1.142 · sonnet-4-6 · ghcr.io/allenai/asta:v0.17.2` (`@sha256:bf92d6a2…`) — tag for readability, digest for strict reproducibility.
121-
122-
See [#60](https://github.com/allenai/asta-plugins/pull/60) for a worked example against existing cases, and [#63](https://github.com/allenai/asta-plugins/pull/63) for a worked example of adding new per-skill cases.
123-
124-
When a comparison includes a configuration that isn't a regular commit on a PR branch (an ablation, an A/B variant, etc.), preserve it as an annotated git tag under `experiments/PR-<num>/<description>` so reviewers can check it out and reproduce. Tag after the PR is open so the number is known:
125-
126-
```bash
127-
git tag -a experiments/PR-123/workspace-ablate-artifacts-tightening \
128-
-m "PR #123's workspace branch with skills/artifacts/SKILL.md reverted to main. Used to measure view_agent_output routing dependency on the artifacts tightening."
129-
git push origin experiments/PR-123/workspace-ablate-artifacts-tightening
130-
```
131-
132-
Tags survive branch deletion. Listable per-PR with `git tag -l 'experiments/PR-123/*'`. Link the tag from the PR description.
133-
134-
External contributors push the tag to their fork (no write access here) and link to the fork's tag URL — same convention, different remote.
99+
For measuring the effect of a skill change (paired PRs + evaluation against `asta_skills`), invoke the [`improve-skills`](skills/improve-skills/SKILL.md) skill or follow [`assets/benchmarking.md`](skills/improve-skills/assets/benchmarking.md) by hand. See [`assets/worked-examples.md`](skills/improve-skills/assets/worked-examples.md) for past PRs as reference.
135100

136101
## Development
137102

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
---
2+
name: improve-skills
3+
description: Improve Asta skills. Always start by capturing the problem and desired behavior. Then either report (open an issue at allenai/asta-plugins) or continue to a fix (paired-PR workflow with paired evaluation). Use when the user observes undesirable agent behavior, e.g. "the agent does X instead of Y", "fix this routing", "this skill doesn't follow through", "report a bug in skill Z".
4+
metadata:
5+
internal: true
6+
allowed-tools: Bash(git *) Bash(gh *) Bash(make *) Bash(inspect log *) Bash(jq *) Bash(cp *) Bash(uv run *) Bash(asta *) Read(assets/**) Read Write Edit
7+
---
8+
9+
# Improve Asta skills
10+
11+
Linear walkthrough; each step's output is shareable (sidebars below mark the natural stopping points, progressively stronger handoffs). See [`assets/worked-examples.md`](assets/worked-examples.md) for past paired PRs.
12+
13+
For SKILL.md authoring or quality conventions outside of Asta-specific ones below, use whatever tooling the user has (Anthropic's `plugin-dev:skill-development` and `plugin-dev:skill-reviewer` are one example).
14+
15+
## 1. Capture
16+
17+
`cp assets/issue.md .asta/improve-skills/<YYYY-MM-DD>-<short-slug>.md` and fill in. Path matches the `.asta/<invoking-skill>/` artifact convention; format is a GitHub issue body so reporting at any point is a direct `gh issue create --body-file`.
18+
19+
> **Stop here (report n=1 observation)**: `gh issue create --repo allenai/asta-plugins --title "<skill>: <one-line>" --body-file .asta/improve-skills/<slug>.md`. Re-invoke later (with the issue linked) to continue. If continuing to a patch PR, skip the issue — the PR description covers the bug.
20+
21+
## 2. Encode the capture as an asta_skills case
22+
23+
One entry in [`astabench/ai2/evals/asta_skills/data.json`](https://github.com/allenai/asta-bench-private/blob/main/astabench/ai2/evals/asta_skills/data.json):
24+
25+
- `initial_prompt`: copy from the capture verbatim.
26+
- `setup`: if non-trivial, write a script under [`setups/`](https://github.com/allenai/asta-bench-private/tree/main/astabench/ai2/evals/asta_skills/setups) inlining the starting state.
27+
- `tool_metric_configs`: one or more handlers (see [`scoring.py`](https://github.com/allenai/asta-bench-private/blob/main/astabench/ai2/evals/asta_skills/scoring.py)):
28+
- `tool_call_match` — agent's tool calls (intent). CLI-specific.
29+
- `sandbox_check` — shell command post-run, exit 0 = fire (outcome). Agent-agnostic.
30+
31+
Prefer `sandbox_check` for "did the right thing land on disk"; `tool_call_match` for "did the agent attempt X" (no persistent outcome).
32+
33+
Commit to a branch in `asta-bench-private`. Don't open the PR yet.
34+
35+
> **Stop here (share draft case)**: push the case branch and open the asta-bench-private PR alone (marked draft if unverified). Append to the capture a `## Reproducible test (draft, not yet measured)` section linking the case PR, then `gh issue create` per step 1.
36+
37+
## 3. Run baseline (new case + regression guards)
38+
39+
Agent, model, pinning, and recording rules in [`assets/benchmarking.md`](assets/benchmarking.md). Run baseline with defaults; capture resolved pins in step 4.
40+
41+
Eval the new case alone first (`--sample-id <case_id>`) — should score 0 on the bad-behavior metric. If not, the case isn't capturing what you think; iterate.
42+
43+
Then run the full case set (new case + cases involving the same skill(s) as regression guards) for the baseline arm.
44+
45+
> **Stop here (share case + baseline, TDD red)**: push the case branch and open the asta-bench-private PR alone. Append a `## Reproducible test` section to the capture with the case PR link and baseline numbers, then `gh issue create` per step 1.
46+
>
47+
> External contributors without write access to `asta-bench-private`: fork it, or host the case files in any public repo you own. Link from the issue. Same for the experiments tag (step 7).
48+
49+
## 4. Capture resolved pins
50+
51+
```bash
52+
eval "$(inspect log dump logs/baseline/*.eval | jq -er '.samples[0].metadata
53+
| "ASTA_IMAGE=\(.asta_image)\nAGENT_VERSION=\(.agent_version)"')"
54+
export ASTA_IMAGE AGENT_VERSION
55+
```
56+
57+
`ASTA_IMAGE` is read by the docker-compose sandbox from the env (must `export`); `AGENT_VERSION` interpolates into `-S version=` in step 6.
58+
59+
## 5. Make the skill change(s)
60+
61+
Edit on a branch in `asta-plugins`. Often spans multiple skills (one's description winning routing against another's). Keep changes minimal.
62+
63+
## 6. Run the PR arm (same case set)
64+
65+
Add `-S version="$AGENT_VERSION"` and `-S skills=<your-PR-branch's-plugins/asta-preview/skills>` to the same command shape as step 3, with a different `--log-dir`. The new case's metrics should fire; regression-guard cases should hold at baseline level — if they drop, investigate before opening.
66+
67+
## 7. (Optional) Ablation if the change has multiple parts
68+
69+
Single revert commit on top of the PR branch + annotated tag:
70+
71+
```bash
72+
# After the PR is open and you know its number:
73+
git checkout -b experiments/PR-<num>/<short-desc> <pr-branch>
74+
git checkout main -- skills/<reverted-skill>/SKILL.md
75+
make build-plugins
76+
git add -A && git commit -m "ablate: revert <description>"
77+
git tag -a experiments/PR-<num>/<short-desc> \
78+
-m "PR #<num> with <reverted-thing> at main. Middle column of PR's validation table."
79+
git push origin refs/heads/experiments/PR-<num>/<short-desc> \
80+
refs/tags/experiments/PR-<num>/<short-desc>
81+
```
82+
83+
Branch + tag share the name.
84+
85+
## 8. Open paired PRs
86+
87+
For each, `cp assets/<...>-pr.md PR.md` and fill in:
88+
89+
- `assets/asta-plugins-pr.md` — the asta-plugins side
90+
- `assets/asta-bench-private-pr.md` — the asta-bench-private side
91+
92+
Cross-link the companion PR after both numbers are assigned (`gh pr edit --body-file PR.md`).
Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
# asta_skills: <what the new case(s) measure>
2+
3+
Companion PR: [allenai/asta-plugins#<n>](URL) <describes the skill change>.
4+
5+
- New `<case_id>` case — <scenario, prompt, what it scores>.
6+
- (Optional) `<existing_case>` extended — <what was added>.
7+
8+
<!-- Supporting machinery (only if you added scorer handlers, setup scripts, or
9+
package-data wiring): one-line bullets each. -->
Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
# <skill>: <verb-first headline>
2+
3+
<1-paragraph or 2-bullet before/after, framed around the user-visible change.>
4+
5+
Companion PR: [allenai/asta-bench-private#<n>](URL) adds the eval case (`<case_id>`) that catches this.
6+
7+
## Validation
8+
9+
Setup: `claude_code <ver>` · `<model>` · `ghcr.io/allenai/asta:<tag>` (`@sha256:<short>…`), <n> epochs.
10+
11+
| case | metric | main | this PR (skill A only) | this PR (full) |
12+
|---|---|---|---|---|
13+
| `<new_case>` | `<metric>` | <m>/<n> | <m>/<n> | <m>/<n> |
14+
| `<regression_guard_case>` | `<metric>` | <m>/<n> | <m>/<n> | <m>/<n> |
15+
16+
<Bullet per metric explaining the lift. Link the ablation tag if any.>
17+
18+
<!-- Drop the middle column if no ablation. -->
Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
# Benchmarking Asta skills
2+
3+
For any eval run against `astabench/asta_skills` (or related); the last two bullets are specific to paired comparisons.
4+
5+
- **Agent and model**: `-S agent=claude_code` and `--model anthropic/claude-sonnet-4-6` (Sonnet 4-class). Routing and skill-following behavior is what this repo measures; that's what these defaults reflect.
6+
- **Pin both arms** to the same `-S version=<cc-ver>` (agent CLI) and `ASTA_IMAGE=…@sha256:…` (sandbox image). The defaults (`auto`, `:latest`) drift between runs and silently break paired comparisons.
7+
- **Record the pins in the PR description** as `claude_code <ver> · <model> · ghcr.io/allenai/asta:<tag>` (`@sha256:<short>…`) — tag for readability, digest for strict reproducibility.
8+
- **Preserve non-PR-branch configurations** (ablations, A/B variants) as annotated git tags under `experiments/PR-<num>/<description>`, so reviewers can check out and reproduce middle columns of validation tables. Tag after the PR is open so the number is known. External contributors push to their fork and link to the fork's tag URL.
9+
10+
The actual paired-comparison recipe (run baseline, capture resolved pins, run PR arm with pins) is in [`agent-baselines`' inspect-swe README](https://github.com/allenai/agent-baselines/tree/main/solvers/inspect-swe#comparing-two-configurations); the `improve-skills` SKILL.md walks through it.
Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
## What happened
2+
<one paragraph: user's prompt + observed agent behavior>
3+
4+
## Expected
5+
<one sentence>
6+
7+
## Repro
8+
- Skill(s): [`<name>`](skills/<name>/SKILL.md)
9+
- Prompt: `<verbatim prompt>`
10+
- Setup: <if non-trivial; otherwise "fresh `/app`">
11+
12+
## Notes
13+
<optional: mechanism guess if the user has one — don't speculate>
Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
# Worked examples
2+
3+
Past paired-PR fixes — useful as concrete priors for what scope, framing, and validation look like.
4+
5+
- [**#60**](https://github.com/allenai/asta-plugins/pull/60) (description rewrite). Single SKILL.md description change to fix routing; paired comparison on four `asta_multitool` cases. Demonstrates the simplest fix shape: one file, one metric moved.
6+
- [**#63**](https://github.com/allenai/asta-plugins/pull/63) (new skill + new per-skill cases). Adds a whole skill alongside the eval cases that exercise it. Demonstrates how to introduce a fresh capability and validate it from day one.
7+
- [**#67**](https://github.com/allenai/asta-plugins/pull/67) (skill body + tightened description across two skills, with ablation). Multi-skill fix (workspace + artifacts) for one user-visible bug; ablation arm proves the artifacts tightening is load-bearing (`view_agent_output` routing collision). Demonstrates regression-guard discovery — `view_agent_output` was in the eval *as a regression guard*, and that's how the collision surfaced.

skills/improve-skills/SKILL.md

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
---
2+
name: improve-skills
3+
description: Improve Asta skills. Always start by capturing the problem and desired behavior. Then either report (open an issue at allenai/asta-plugins) or continue to a fix (paired-PR workflow with paired evaluation). Use when the user observes undesirable agent behavior, e.g. "the agent does X instead of Y", "fix this routing", "this skill doesn't follow through", "report a bug in skill Z".
4+
metadata:
5+
internal: true
6+
allowed-tools: Bash(git *) Bash(gh *) Bash(make *) Bash(inspect log *) Bash(jq *) Bash(cp *) Bash(uv run *) Bash(asta *) Read(assets/**) Read Write Edit
7+
---
8+
9+
# Improve Asta skills
10+
11+
Linear walkthrough; each step's output is shareable (sidebars below mark the natural stopping points, progressively stronger handoffs). See [`assets/worked-examples.md`](assets/worked-examples.md) for past paired PRs.
12+
13+
For SKILL.md authoring or quality conventions outside of Asta-specific ones below, use whatever tooling the user has (Anthropic's `plugin-dev:skill-development` and `plugin-dev:skill-reviewer` are one example).
14+
15+
## 1. Capture
16+
17+
`cp assets/issue.md .asta/improve-skills/<YYYY-MM-DD>-<short-slug>.md` and fill in. Path matches the `.asta/<invoking-skill>/` artifact convention; format is a GitHub issue body so reporting at any point is a direct `gh issue create --body-file`.
18+
19+
> **Stop here (report n=1 observation)**: `gh issue create --repo allenai/asta-plugins --title "<skill>: <one-line>" --body-file .asta/improve-skills/<slug>.md`. Re-invoke later (with the issue linked) to continue. If continuing to a patch PR, skip the issue — the PR description covers the bug.
20+
21+
## 2. Encode the capture as an asta_skills case
22+
23+
One entry in [`astabench/ai2/evals/asta_skills/data.json`](https://github.com/allenai/asta-bench-private/blob/main/astabench/ai2/evals/asta_skills/data.json):
24+
25+
- `initial_prompt`: copy from the capture verbatim.
26+
- `setup`: if non-trivial, write a script under [`setups/`](https://github.com/allenai/asta-bench-private/tree/main/astabench/ai2/evals/asta_skills/setups) inlining the starting state.
27+
- `tool_metric_configs`: one or more handlers (see [`scoring.py`](https://github.com/allenai/asta-bench-private/blob/main/astabench/ai2/evals/asta_skills/scoring.py)):
28+
- `tool_call_match` — agent's tool calls (intent). CLI-specific.
29+
- `sandbox_check` — shell command post-run, exit 0 = fire (outcome). Agent-agnostic.
30+
31+
Prefer `sandbox_check` for "did the right thing land on disk"; `tool_call_match` for "did the agent attempt X" (no persistent outcome).
32+
33+
Commit to a branch in `asta-bench-private`. Don't open the PR yet.
34+
35+
> **Stop here (share draft case)**: push the case branch and open the asta-bench-private PR alone (marked draft if unverified). Append to the capture a `## Reproducible test (draft, not yet measured)` section linking the case PR, then `gh issue create` per step 1.
36+
37+
## 3. Run baseline (new case + regression guards)
38+
39+
Agent, model, pinning, and recording rules in [`assets/benchmarking.md`](assets/benchmarking.md). Run baseline with defaults; capture resolved pins in step 4.
40+
41+
Eval the new case alone first (`--sample-id <case_id>`) — should score 0 on the bad-behavior metric. If not, the case isn't capturing what you think; iterate.
42+
43+
Then run the full case set (new case + cases involving the same skill(s) as regression guards) for the baseline arm.
44+
45+
> **Stop here (share case + baseline, TDD red)**: push the case branch and open the asta-bench-private PR alone. Append a `## Reproducible test` section to the capture with the case PR link and baseline numbers, then `gh issue create` per step 1.
46+
>
47+
> External contributors without write access to `asta-bench-private`: fork it, or host the case files in any public repo you own. Link from the issue. Same for the experiments tag (step 7).
48+
49+
## 4. Capture resolved pins
50+
51+
```bash
52+
eval "$(inspect log dump logs/baseline/*.eval | jq -er '.samples[0].metadata
53+
| "ASTA_IMAGE=\(.asta_image)\nAGENT_VERSION=\(.agent_version)"')"
54+
export ASTA_IMAGE AGENT_VERSION
55+
```
56+
57+
`ASTA_IMAGE` is read by the docker-compose sandbox from the env (must `export`); `AGENT_VERSION` interpolates into `-S version=` in step 6.
58+
59+
## 5. Make the skill change(s)
60+
61+
Edit on a branch in `asta-plugins`. Often spans multiple skills (one's description winning routing against another's). Keep changes minimal.
62+
63+
## 6. Run the PR arm (same case set)
64+
65+
Add `-S version="$AGENT_VERSION"` and `-S skills=<your-PR-branch's-plugins/asta-preview/skills>` to the same command shape as step 3, with a different `--log-dir`. The new case's metrics should fire; regression-guard cases should hold at baseline level — if they drop, investigate before opening.
66+
67+
## 7. (Optional) Ablation if the change has multiple parts
68+
69+
Single revert commit on top of the PR branch + annotated tag:
70+
71+
```bash
72+
# After the PR is open and you know its number:
73+
git checkout -b experiments/PR-<num>/<short-desc> <pr-branch>
74+
git checkout main -- skills/<reverted-skill>/SKILL.md
75+
make build-plugins
76+
git add -A && git commit -m "ablate: revert <description>"
77+
git tag -a experiments/PR-<num>/<short-desc> \
78+
-m "PR #<num> with <reverted-thing> at main. Middle column of PR's validation table."
79+
git push origin refs/heads/experiments/PR-<num>/<short-desc> \
80+
refs/tags/experiments/PR-<num>/<short-desc>
81+
```
82+
83+
Branch + tag share the name.
84+
85+
## 8. Open paired PRs
86+
87+
For each, `cp assets/<...>-pr.md PR.md` and fill in:
88+
89+
- `assets/asta-plugins-pr.md` — the asta-plugins side
90+
- `assets/asta-bench-private-pr.md` — the asta-bench-private side
91+
92+
Cross-link the companion PR after both numbers are assigned (`gh pr edit --body-file PR.md`).
Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
# asta_skills: <what the new case(s) measure>
2+
3+
Companion PR: [allenai/asta-plugins#<n>](URL) <describes the skill change>.
4+
5+
- New `<case_id>` case — <scenario, prompt, what it scores>.
6+
- (Optional) `<existing_case>` extended — <what was added>.
7+
8+
<!-- Supporting machinery (only if you added scorer handlers, setup scripts, or
9+
package-data wiring): one-line bullets each. -->
Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
# <skill>: <verb-first headline>
2+
3+
<1-paragraph or 2-bullet before/after, framed around the user-visible change.>
4+
5+
Companion PR: [allenai/asta-bench-private#<n>](URL) adds the eval case (`<case_id>`) that catches this.
6+
7+
## Validation
8+
9+
Setup: `claude_code <ver>` · `<model>` · `ghcr.io/allenai/asta:<tag>` (`@sha256:<short>…`), <n> epochs.
10+
11+
| case | metric | main | this PR (skill A only) | this PR (full) |
12+
|---|---|---|---|---|
13+
| `<new_case>` | `<metric>` | <m>/<n> | <m>/<n> | <m>/<n> |
14+
| `<regression_guard_case>` | `<metric>` | <m>/<n> | <m>/<n> | <m>/<n> |
15+
16+
<Bullet per metric explaining the lift. Link the ablation tag if any.>
17+
18+
<!-- Drop the middle column if no ablation. -->

0 commit comments

Comments
 (0)