Reproduction: the regression tax in office-artifact agents

This repository contains a public, claim-by-claim reproduction of The Regression Tax: Decomposing Why Skills Help—and Hurt—LLM Agents (arXiv:2607.22520). We tested whether paired evaluation exposes regressions, whether skill descriptions change outcomes without body invocation, and whether explicit grounding plus output review recovers failures better than procedure alone.

Assessment: partially reproduced. The paper reports 553 gains and 324 regressions across 5,832 task conditions, with regressions offsetting 59% of gross gains. On our 96 paired task-seed units, descriptions alone produced 3 gains and 12 regressions, while procedure produced 2 gains and 10 regressions. Twelve description-only regressions occurred with no body loaded. A concise grounding-plus-review prompt recovered 5/28 baseline failures versus 2/28 for procedure, but both passed 61/96 overall; verbose and executable checking variants were mixed or negative.

The original 486-task OfficeQA-Pro/SpreadsheetBench splits and trajectories are not released. We therefore use a bounded public substitute: 12 deterministic .xlsx, .docx, and .pptx fixtures, eight paired stochastic seeds, Qwen3-Coder-30B-A3B-Instruct, real OOXML artifacts, and deterministic validators. This tests the mechanisms rather than the paper’s absolute rates. All formal runs used OpenResearch Kubernetes on NVIDIA RTX PRO 6000 Blackwell GPUs, with a peak of 16 concurrent GPUs and 0.38 hours elapsed wall time.

Read the illustrated report · Inspect the frozen results · Open the self-contained notebook

Experiment log

Every experiment inherited the same exact run command. Branches vary only committed prompts or paired-seed blocks.

Branch / experiment	Purpose or change	Exact run command	Assessment / outcome	Compute
main	Public README, report, figures, notebook, and polished winning harness	Not run as an experiment (publication surface)	Publication only	—
Primary seeds 0–3	Four contexts × 12 tasks × seeds 0–3	python -m pip install --quiet -r requirements-reproduction.txt && python run_reproduction.py	No skill 36/48; descriptions 33/48; procedure 31/48; verbose G+V 27/48	Kubernetes, 8× RTX PRO 6000 Blackwell
Primary seeds 4–7	Independent paired seeds 4–7	python -m pip install --quiet -r requirements-reproduction.txt && python run_reproduction.py	No skill 34/48; descriptions 28/48; procedure 31/48; verbose G+V 27/48	Kubernetes, 8× RTX PRO 6000 Blackwell
Concise G+V seeds 0–3	Task-authority checklist plus non-asserting review	python -m pip install --quiet -r requirements-reproduction.txt && python run_reproduction.py	G+V 30/48 vs procedure 29/48; baseline recovery 2 vs 0	Kubernetes, 8× RTX PRO 6000 Blackwell
Concise G+V seeds 4–7	Independent replication of promoted prompt	python -m pip install --quiet -r requirements-reproduction.txt && python run_reproduction.py	G+V 31/48 vs procedure 32/48; baseline recovery 3 vs 2	Kubernetes, 8× RTX PRO 6000 Blackwell
Nonblocking executable checks	Reopen/check/repair without assertions	python -m pip install --quiet -r requirements-reproduction.txt && python run_reproduction.py	Did not improve: G+V 29/48 vs procedure 30/48	Kubernetes, 8× RTX PRO 6000 Blackwell
Grounding-only ablation	Remove final review instruction	python -m pip install --quiet -r requirements-reproduction.txt && python run_reproduction.py	Did not improve: 29/48 vs procedure 31/48	Kubernetes, 8× RTX PRO 6000 Blackwell

Two setup generations failed before experiment code ran (shell expansion, then a missing python alias); their descendants preserve the manifest-only repairs. Scientific results above come only from successful Kubernetes runs with nonempty terminal logs.

---

meta-skill-creator

The Skill • Creating Skills • Use With Your Agent

A skill that helps you create skills.

This repo hosts skill-creator — a harness-agnostic meta-skill that guides an agent through authoring, updating, and optimizing other Agent Skills. It relies on no specific runtime, CLI, or vendor tooling: the helper scripts in skill-creator/scripts/ use only the Python standard library, and every step has a plain "do it by hand" fallback, so any agent in any harness can follow it.

The layout follows the Agent Skills convention — a skill is a self-contained folder holding a SKILL.md plus optional scripts/, references/, and assets/.

Layout

meta-skill-creator/
├── LICENSE
├── README.md          ← you are here
└── skill-creator/
    ├── SKILL.md        (the meta-skill: guidance for creating skills)
    └── scripts/        (stdlib-only helpers, each with a manual fallback)
        ├── scaffold_skill.py   scaffold a new skill folder + SKILL.md skeleton
        ├── validate_skill.py   check frontmatter format, required fields, naming
        └── discover_skills.py  discover existing skills before creating a duplicate

What is a skill?

A skill is a modular, self-contained folder that extends an agent's capabilities with specialized knowledge, workflows, or tools — an "onboarding guide" for a task or domain. Skills load in three levels (progressive disclosure), so each step should cost only what it needs:

1. Metadata (name + description) — always in context. This is the trigger.
2. SKILL.md body — loaded only once the skill triggers.
3. Bundled resources (scripts/, references/, assets/) — loaded or executed only when a step needs them.

The flow of creating a skill

The full guidance lives in skill-creator/SKILL.md. In short, the meta-skill walks through these steps in order (skip one only when it clearly doesn't apply):

1. Understand with concrete examples — gather (or propose, then confirm) real trigger requests and expected outputs before writing anything. "What should this enable? What would a user say to trigger it? What's the expected output?"

2. Discover existing skills — update over create — before scaffolding, check whether a similar skill already exists; a near-duplicate dilutes triggering and drifts out of sync.

   python skill-creator/scripts/discover_skills.py "<proposed name or one-line intent>" \
       --skills-dir <skills-dir>

   Same capability → update it in place (and broaden its description to keep triggering on everything the merged parts covered). Merely adjacent → create a new one.

3. Plan reusable contents — for each example, ask what would help if the agent did this repeatedly. Bundle resources generously: scripts/ for anything fragile, exact, or multi-step (calculations, conversions, parsing); references/ for the precise specs and conventions a task hinges on; assets/ for templates used verbatim in the output.

4. Scaffold (new skills only) — create the folder and a SKILL.md skeleton.

   python skill-creator/scripts/scaffold_skill.py <skill-name> --path <dir> \
       [--resources scripts,references,assets]

5. Write the SKILL.md and resources — build (and test) the resources first, then write the body that ties them together.

   • name — hyphen-case, lowercase letters/digits/hyphens, ≤64 chars; name the folder to match.
   • description — the primary trigger. Enumerate the literal phrases a matching request would use, cover every capability, and add a short "Do NOT use for …" boundary.
   • Body — imperative instructions addressed to the agent; explain the why, stay concise, and write for reuse rather than one example.

6. Validate — check frontmatter format, required fields, and naming rules.

   python skill-creator/scripts/validate_skill.py <path/to/skill-folder>

7. Self-reflect and optimize — re-read the draft with fresh eyes and revise across these lenses: triggering (do the literal words match?), self-containment, concision and altitude, progressive disclosure, degrees of freedom, and a dry run (actually compute anything numeric). Apply the revisions; repeat if the pass turned up substantial issues.

8. Iterate on real usage — use the skill on real tasks; when several runs reinvent the same helper or take the same detour, bundle it into the skill so future runs don't.

Use it with your agent

skill-creator is a standard Agent Skill — a folder with a SKILL.md. Install it wherever your agent looks for skills; once it's there, the agent loads it automatically when you ask to build or improve a skill (the description is the trigger), or you can invoke it by name. The helper scripts are stdlib-only Python, so they run anywhere python3 does — and each has a by-hand fallback in SKILL.md for agents that can't run scripts at all.

Claude Code

Skills live in a skills/ folder under .claude/. Copy (or symlink) this repo's skill-creator/ directory into one of them — the folder name becomes the command:

# personal — available in all your projects
cp -r skill-creator ~/.claude/skills/skill-creator

# or project-local — checked in, shared with the repo
cp -r skill-creator .claude/skills/skill-creator

Level	Path	Scope
Personal	~/.claude/skills/skill-creator/SKILL.md	all your projects
Project	.claude/skills/skill-creator/SKILL.md	this repo only

Then just ask ("help me turn this workflow into a skill") and Claude loads it when relevant, or run /skill-creator to invoke it directly. Claude Code picks up new skills in an existing .claude/skills/ without a restart.

OpenAI Codex

Install with Codex's built-in $skill-installer, pointing it at this skill's folder (a Git URL or a local path), then restart Codex to pick it up:

$skill-installer install <git-url-or-path-to>/meta-skill-creator/skill-creator

After restart, ask Codex to create or improve a skill and it will use skill-creator.

Any other agent

The skill is deliberately harness-agnostic — no vendor tooling, no runtime assumptions. Two ways to use it with any agent:

1. Load the instructions — point the agent at skill-creator/SKILL.md (paste it, add it to the system prompt, or drop the folder wherever that agent discovers skills) and follow the 8-step flow above.

2. Run the helpers directly — the three scripts work standalone, no install needed:

   # discover before creating
   python skill-creator/scripts/discover_skills.py "rotate pdf pages" --skills-dir ./skills
   
   # scaffold a new skill
   python skill-creator/scripts/scaffold_skill.py rotate-pdf --path ./skills --resources scripts
   
   # validate it
   python skill-creator/scripts/validate_skill.py ./skills/rotate-pdf