|
| 1 | +--- |
| 2 | +name: improve-skills |
| 3 | +description: Improve Asta skills. Always start by capturing the problem and desired behavior. Then either report (open an issue at allenai/asta-plugins) or continue to a fix (paired-PR workflow with paired evaluation). Use when the user observes undesirable agent behavior, e.g. "the agent does X instead of Y", "fix this routing", "this skill doesn't follow through", "report a bug in skill Z". |
| 4 | +metadata: |
| 5 | + internal: true |
| 6 | +allowed-tools: Bash(git *) Bash(gh *) Bash(make *) Bash(inspect log *) Bash(jq *) Bash(cp *) Bash(uv run *) Bash(asta *) Read(assets/**) Read Write Edit |
| 7 | +--- |
| 8 | + |
| 9 | +# Improve Asta skills |
| 10 | + |
| 11 | +Linear walkthrough; each step's output is shareable (sidebars below mark the natural stopping points, progressively stronger handoffs). See [`assets/worked-examples.md`](assets/worked-examples.md) for past paired PRs. |
| 12 | + |
| 13 | +For SKILL.md authoring or quality conventions outside of Asta-specific ones below, use whatever tooling the user has (Anthropic's `plugin-dev:skill-development` and `plugin-dev:skill-reviewer` are one example). |
| 14 | + |
| 15 | +## 1. Capture |
| 16 | + |
| 17 | +`cp assets/issue.md .asta/improve-skills/<YYYY-MM-DD>-<short-slug>.md` and fill in. Path matches the `.asta/<invoking-skill>/` artifact convention; format is a GitHub issue body so reporting at any point is a direct `gh issue create --body-file`. |
| 18 | + |
| 19 | +> **Stop here (report n=1 observation)**: `gh issue create --repo allenai/asta-plugins --title "<skill>: <one-line>" --body-file .asta/improve-skills/<slug>.md`. Re-invoke later (with the issue linked) to continue. If continuing to a patch PR, skip the issue — the PR description covers the bug. |
| 20 | +
|
| 21 | +## 2. Encode the capture as an asta_skills case |
| 22 | + |
| 23 | +One entry in [`astabench/ai2/evals/asta_skills/data.json`](https://github.com/allenai/asta-bench-private/blob/main/astabench/ai2/evals/asta_skills/data.json): |
| 24 | + |
| 25 | +- `initial_prompt`: copy from the capture verbatim. |
| 26 | +- `setup`: if non-trivial, write a script under [`setups/`](https://github.com/allenai/asta-bench-private/tree/main/astabench/ai2/evals/asta_skills/setups) inlining the starting state. |
| 27 | +- `tool_metric_configs`: one or more handlers (see [`scoring.py`](https://github.com/allenai/asta-bench-private/blob/main/astabench/ai2/evals/asta_skills/scoring.py)): |
| 28 | + - `tool_call_match` — agent's tool calls (intent). CLI-specific. |
| 29 | + - `sandbox_check` — shell command post-run, exit 0 = fire (outcome). Agent-agnostic. |
| 30 | + |
| 31 | +Prefer `sandbox_check` for "did the right thing land on disk"; `tool_call_match` for "did the agent attempt X" (no persistent outcome). |
| 32 | + |
| 33 | +Commit to a branch in `asta-bench-private`. Don't open the PR yet. |
| 34 | + |
| 35 | +> **Stop here (share draft case)**: push the case branch and open the asta-bench-private PR alone (marked draft if unverified). Append to the capture a `## Reproducible test (draft, not yet measured)` section linking the case PR, then `gh issue create` per step 1. |
| 36 | +
|
| 37 | +## 3. Run baseline (new case + regression guards) |
| 38 | + |
| 39 | +Agent, model, pinning, and recording rules in [`assets/benchmarking.md`](assets/benchmarking.md). Run baseline with defaults; capture resolved pins in step 4. |
| 40 | + |
| 41 | +Eval the new case alone first (`--sample-id <case_id>`) — should score 0 on the bad-behavior metric. If not, the case isn't capturing what you think; iterate. |
| 42 | + |
| 43 | +Then run the full case set (new case + cases involving the same skill(s) as regression guards) for the baseline arm. |
| 44 | + |
| 45 | +> **Stop here (share case + baseline, TDD red)**: push the case branch and open the asta-bench-private PR alone. Append a `## Reproducible test` section to the capture with the case PR link and baseline numbers, then `gh issue create` per step 1. |
| 46 | +> |
| 47 | +> External contributors without write access to `asta-bench-private`: fork it, or host the case files in any public repo you own. Link from the issue. Same for the experiments tag (step 7). |
| 48 | +
|
| 49 | +## 4. Capture resolved pins |
| 50 | + |
| 51 | +```bash |
| 52 | +eval "$(inspect log dump logs/baseline/*.eval | jq -er '.samples[0].metadata |
| 53 | + | "ASTA_IMAGE=\(.asta_image)\nAGENT_VERSION=\(.agent_version)"')" |
| 54 | +export ASTA_IMAGE AGENT_VERSION |
| 55 | +``` |
| 56 | + |
| 57 | +`ASTA_IMAGE` is read by the docker-compose sandbox from the env (must `export`); `AGENT_VERSION` interpolates into `-S version=` in step 6. |
| 58 | + |
| 59 | +## 5. Make the skill change(s) |
| 60 | + |
| 61 | +Edit on a branch in `asta-plugins`. Often spans multiple skills (one's description winning routing against another's). Keep changes minimal. |
| 62 | + |
| 63 | +## 6. Run the PR arm (same case set) |
| 64 | + |
| 65 | +Add `-S version="$AGENT_VERSION"` and `-S skills=<your-PR-branch's-plugins/asta-preview/skills>` to the same command shape as step 3, with a different `--log-dir`. The new case's metrics should fire; regression-guard cases should hold at baseline level — if they drop, investigate before opening. |
| 66 | + |
| 67 | +## 7. (Optional) Ablation if the change has multiple parts |
| 68 | + |
| 69 | +Single revert commit on top of the PR branch + annotated tag: |
| 70 | + |
| 71 | +```bash |
| 72 | +# After the PR is open and you know its number: |
| 73 | +git checkout -b experiments/PR-<num>/<short-desc> <pr-branch> |
| 74 | +git checkout main -- skills/<reverted-skill>/SKILL.md |
| 75 | +make build-plugins |
| 76 | +git add -A && git commit -m "ablate: revert <description>" |
| 77 | +git tag -a experiments/PR-<num>/<short-desc> \ |
| 78 | + -m "PR #<num> with <reverted-thing> at main. Middle column of PR's validation table." |
| 79 | +git push origin refs/heads/experiments/PR-<num>/<short-desc> \ |
| 80 | + refs/tags/experiments/PR-<num>/<short-desc> |
| 81 | +``` |
| 82 | + |
| 83 | +Branch + tag share the name. |
| 84 | + |
| 85 | +## 8. Open paired PRs |
| 86 | + |
| 87 | +For each, `cp assets/<...>-pr.md PR.md` and fill in: |
| 88 | + |
| 89 | +- `assets/asta-plugins-pr.md` — the asta-plugins side |
| 90 | +- `assets/asta-bench-private-pr.md` — the asta-bench-private side |
| 91 | + |
| 92 | +Cross-link the companion PR after both numbers are assigned (`gh pr edit --body-file PR.md`). |
0 commit comments