workspace: launch dev containers, route "show me my work" reliably#67
Merged
Conversation
Before, the dev container scaffold step ended with the devcontainer.json
file and "tell the user how to open it." Now the agent also runs the
launch command — `make dev` for local, `git push` + `gh codespace create`
for Codespaces — so the user ends up in the running container.
DEVELOPER.md also gains a `gh codespace create --web` CLI mention next
to the green-button Codespaces flow.
Artifacts description tightened to A2A-specific phrasings ("export the
task results", "convert the task output to HTML") from broad ones ("show
me the artifacts", "what did the agent produce"). At v0.17.2 the broad
description routes "Show me what you've put together so far" to artifacts
in 2/3 epochs (per-epoch Skill calls confirm); tightening restores 3/3
workspace routing. It's also a more honest description of what artifacts
actually does.
README's Benchmarking section gets a version-pinning convention
(`-S version=` + `ASTA_IMAGE=…@sha256:…` on both arms, recorded in the
PR description as `cc <ver> · <model> · ghcr.io/allenai/asta:<tag>`) —
discovered while validating this PR that the defaults (`auto`, `:latest`)
drift, breaking paired comparisons.
Paired with allenai/asta-bench-private's asta_skills extension that adds
an avoid_host_installs case scoring the launch step, plus outcome-level
scoring (sandbox_check) so routing-only metrics no longer pass under
"scaffold and stop" behavior.
rodneykinney
approved these changes
Jun 2, 2026
rodneykinney
left a comment
rodneykinney
left a comment
Member
There was a problem hiding this comment.
Tightening artifacts to A2A-specific phrasings ("export the task results", "convert the task output to HTML") restores reliable workspace routing.
🥇
@charliemcgrady FYI
jbragg
added a commit
that referenced
this pull request
Jun 2, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
jbragg
added a commit
that referenced
this pull request
Jun 2, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
jbragg
added a commit
that referenced
this pull request
Jun 2, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
jbragg
added a commit
that referenced
this pull request
Jun 3, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
jbragg
added a commit
that referenced
this pull request
Jun 3, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
jbragg
added a commit
that referenced
this pull request
Jun 3, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
jbragg
added a commit
that referenced
this pull request
Jun 3, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
jbragg
added a commit
that referenced
this pull request
Jun 3, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
jbragg
added a commit
that referenced
this pull request
Jun 3, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
jbragg
added a commit
that referenced
this pull request
Jun 3, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
jbragg
added a commit
that referenced
this pull request
Jun 3, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
jbragg
added a commit
that referenced
this pull request
Jun 3, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
jbragg
added a commit
that referenced
this pull request
Jun 3, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
jbragg
added a commit
that referenced
this pull request
Jun 3, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
jbragg
added a commit
that referenced
this pull request
Jun 3, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
jbragg
added a commit
that referenced
this pull request
Jun 3, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
jbragg
added a commit
that referenced
this pull request
Jun 3, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
jbragg
added a commit
that referenced
this pull request
Jun 3, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
jbragg
added a commit
that referenced
this pull request
Jun 3, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
jbragg
added a commit
that referenced
this pull request
Jun 3, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
jbragg
added a commit
that referenced
this pull request
Jun 3, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
jbragg
added a commit
that referenced
this pull request
Jun 3, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
jbragg
added a commit
that referenced
this pull request
Jun 3, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
jbragg
added a commit
that referenced
this pull request
Jun 3, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
jbragg
added a commit
that referenced
this pull request
Jun 3, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
jbragg
added a commit
that referenced
this pull request
Jun 3, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
jbragg
added a commit
that referenced
this pull request
Jun 3, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
jbragg
added a commit
that referenced
this pull request
Jun 3, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
jbragg
added a commit
that referenced
this pull request
Jun 3, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
jbragg
added a commit
that referenced
this pull request
Jun 3, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
jbragg
added a commit
that referenced
this pull request
Jun 3, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
jbragg
added a commit
that referenced
this pull request
Jun 3, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
jbragg
added a commit
that referenced
this pull request
Jun 3, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
jbragg
added a commit
that referenced
this pull request
Jun 3, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
charliemcgrady
approved these changes
Jun 3, 2026
jbragg
added a commit
that referenced
this pull request
Jun 3, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
jbragg
added a commit
that referenced
this pull request
Jun 3, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
jbragg
added a commit
that referenced
this pull request
Jun 3, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
jbragg
added a commit
that referenced
this pull request
Jun 3, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
jbragg
added a commit
that referenced
this pull request
Jun 3, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
jbragg
added a commit
that referenced
this pull request
Jun 5, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
jbragg
added a commit
that referenced
this pull request
Jun 5, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
jbragg
added a commit
that referenced
this pull request
Jun 5, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
jbragg
added a commit
that referenced
this pull request
Jun 5, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
jbragg
added a commit
that referenced
this pull request
Jun 5, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
jbragg
added a commit
that referenced
this pull request
Jun 5, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
jbragg
added a commit
that referenced
this pull request
Jun 5, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
jbragg
added a commit
that referenced
this pull request
Jun 5, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
jbragg
added a commit
that referenced
this pull request
Jun 5, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
jbragg
added a commit
that referenced
this pull request
Jun 5, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
jbragg
added a commit
that referenced
this pull request
Jun 5, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
jbragg
added a commit
that referenced
this pull request
Jun 5, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
jbragg
added a commit
that referenced
this pull request
Jun 5, 2026
Two paths in the skill: - Fix: paired-PR workflow — asta_skills case scaffold, baseline run, pin capture, the skill edit (often multi-skill), PR-arm run, optional ablation + experiment tag, cross-linked PR creation. - Report: opens a GitHub issue at allenai/asta-plugins with the observed prompt, behavior, and expected outcome, for someone else to pick up. Re-invokable later via the fix path. Scope is the skills system, not a single skill — many useful fixes span multiple SKILL.md files (e.g. #67 touched workspace + artifacts for one user-visible change). The skill prescribes only the Asta- specific conventions (case scaffolding, paired-comparison flow, PR.md templates, experiment-tag namespace). For generic skill-quality concerns (description format, body conventions, review checklists), defer to whatever tooling the user has — Anthropic's plugin-dev skills are mentioned as one example, not required. README's Benchmarking section shrinks from the full walkthrough to the conventions stated as rules, pointing at the skill for the procedure. Worked-example PR refs stay (#60, #63, #67).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two related fixes to the workspace skill's UX:
.devcontainer/devcontainer.jsonwith "tell the user how to open it." Now the agent runs the launch command —make devfor local,git push+gh codespace createfor Codespaces — so the user ends up in the running container, not at a config file."show me the artifacts","what did the agent produce","open the report") was winning routing against the workspace skill for prompts like "Show me what you've put together so far" — agent went to A2A-task artifacts export instead of showing the user their research project (2/3 of epochs at v0.17.2). Tightening artifacts to A2A-specific phrasings ("export the task results","convert the task output to HTML") restores reliable workspace routing.Companion PR: allenai/asta-bench-private#228 adds the eval case (
avoid_host_installs) that catches the scaffold-and-stop behavior, plus outcome-level scoring so routing-only metrics can't paper over it.Validation
Setup:
claude_code 2.1.142·sonnet-4-6·ghcr.io/allenai/asta:v0.17.2(@sha256:bf92d6a2029b...) viainspect_swe, Inspect docker sandbox, working-limit 600s, 3 epochs per cell.Cells are
<epochs fired>/<n>for each per-case metric. Metric names matchdata.json.start_workspaceandsave_iterationare regression guards (at ceiling in all arms — included for completeness).avoid_host_installs(new)workspace_skill_activateddevcontainer_scaffoldedlocal_launch_attemptedview_agent_outputworkspace_skill_activatedstart_workspaceworkspace_skill_activatedquarto_scaffoldedsave_iterationworkspace_skill_activatedThe middle column ablates the artifacts description tightening (workspace body change applied, artifacts left at main).
avoid_host_installs: the workspace body change adds the launch step (local_launch_attemptedgoes 0/3 → 3/3 — the headline).view_agent_output: the routing collision is described in the lead. Per-epochSkillcalls in baseline confirm the agent invokesartifacts(notworkspace) on the failing epochs. Workspace SKILL body change alone moves the needle slightly (1/3 → 2/3); the artifacts tightening is what reaches 3/3.The Codespaces path ships in SKILL.md but isn't eval-validated — sandbox
ghcan't authenticate.DEVELOPER.mdgains agh codespace create --webCLI mention next to the green-button Codespaces flow, symmetric withmake devalready being the CLI alternative for VS Code's "Reopen in Container."README.md(Benchmarking section) gains two conventions, both surfaced while validating this PR:-S version=+ASTA_IMAGE=…@sha256:…pinned on both arms, recorded in the PR description ascc <ver> · <model> · ghcr.io/allenai/asta:<tag>(@sha256:…). The defaults (auto,:latest) drift between runs and break paired comparisons.experiments/PR-<num>/<description>, so reviewers can check out and reproduce the middle columns of validation tables. Applied to this PR's ablation arm atexperiments/PR-67/workspace-ablate-artifacts-tightening.