fix(spec-workflow): migrate skills for Opus 5 behavior - #567
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
● Reviewed · 2026-08-05 23:28 UTC · view run ↗
Applies the Opus 5 migration deltas to AssessmentThe core fix is sound: loop exits now key off test/lint/typecheck results and applied-or-rejected reviewer findings instead of a number the model would invent, and every remaining agent name resolves to a real agent. The version bump (2.0.1 → 2.1.0), root table, plugin Suggestions
|
📚 Documentation Check ✅Verdict: Passed The spec-workflow plugin version was bumped (2.0.1 → 2.1.0) in both plugin.json and the root CLAUDE.md version table, satisfying the mandatory version-bump rule. The plugin's own CLAUDE.md was updated with a new "Agent Staffing Conventions" section that accurately reflects the behavioral changes made to both SKILL.md files. No blocking issues found. SummaryPlugin version bump (blocking check): satisfied. Bump size: Minor is defensible. The change is a behavioral refinement across
Changelog: none added; per the validator config this is informational, not blocking. Naming conventions: no new skills/agents/commands were introduced, so the skill-vs-agent naming convention rule doesn't apply here. Missing Updates
Suggestions (1)
🤖 Generated by Claude Documentation Validator | Mode: |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5d747e6468
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| - **Documentation**: documentation-agent, review-plan | ||
| - **Documentation**: documentation-agent | ||
|
|
||
| Prefer running the project's own test, lint, and typecheck commands over spawning an agent to judge quality - a tool result is evidence, an agent's opinion of code it just read is not. |
There was a problem hiding this comment.
When implement-spec runs with quality gates enabled, this new instruction tells the skill to prefer running the project's test/lint/typecheck commands directly, but the skill's frontmatter still only allows Read, Task, and the spec-workflow MCP tools, with no Bash permission. In that context the main skill cannot actually collect the tool-result evidence it now prefers and must either skip these checks or delegate to agents anyway; add the necessary Bash(...) permissions or keep the checks explicitly delegated.
Useful? React with 👍 / 👎.
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
|
|
||
| ### Phase 3: Task Execution | ||
|
|
||
| Each list below is a menu of agents that fit that task type, not a team to staff in full. Pick only the ones the task actually needs, and do the work directly rather than spawning an agent when it would finish in a handful of tool calls. **Ceiling: at most 4 concurrent agents at any moment** - a ceiling, not a quota. |
There was a problem hiding this comment.
agent-orchestrator-agent, whose context list (lines 54-58) doesn't include the ceiling, and whose own definition has no concurrency bound (agent-orchestrator.md: "Launch all agents simultaneously", "Maximize parallel execution opportunities"). Whenever the orchestrator is available, the ceiling is inert — pass it through as part of the Phase 2 execution configuration.
| Each list below is a menu of agents that fit that task type, not a team to staff in full. Pick only the ones the task actually needs, and do the work directly rather than spawning an agent when it would finish in a handful of tool calls. **Ceiling: at most 4 concurrent agents at any moment** - a ceiling, not a quota. | |
| Each list below is a menu of agents that fit that task type, not a team to staff in full. Pick only the ones the task actually needs, and do the work directly rather than spawning an agent when it would finish in a handful of tool calls. **Ceiling: at most 4 concurrent agents at any moment** - a ceiling, not a quota. Pass this ceiling to `agent-orchestrator-agent` as part of its execution configuration in Phase 2; it does not apply one by default. |
| ### CRITICAL INSTRUCTIONS FOR AUTONOMOUS EXECUTION | ||
| ### Autonomous execution contract | ||
|
|
||
| 1. Never prompt the user for review or approval at any point in the workflow. |
There was a problem hiding this comment.
🔵 info · correctness — This contract line contradicts the retained skip_final_review input at line 24 ("skip final user review (default: false)"), which no phase consumes. Read literally, the input spec says a final user review happens by default — the same invented-gate class this PR removed elsewhere.
| @@ -1,29 +1,20 @@ | |||
| --- | |||
| description: Autonomously create and implement a complete spec workflow with multi-agent collaboration, bypassing manual review steps through intelligent consensus-building | |||
| description: Autonomously create and implement a complete spec workflow with multi-agent collaboration, replacing manual review steps with agent review | |||
There was a problem hiding this comment.
🔵 info · docs — The skill's description and behavior contract changed (consensus-based review → agent review with observable exit criteria); component counts are unchanged, but the Notion Plugin Marketplace description for auto-spec is now stale — update the spec-workflow section there.
There was a problem hiding this comment.
Note
✅ Approved — see full review in the sticky comment ↑
Replace the invented quality_threshold loop gate with observable exit criteria, cap agent fan-out, remove the defunct MultiEdit tool, drop a fabricated agent reference, and add output length guidance. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
5d747e6 to
bc18144
Compare
1 participant
Applies the Opus 5 migration deltas to
packages/plugins/spec-workflow. Both.mdskill files were read in full; no other plugin is touched.Headline fix: the invented
quality_thresholdloop gateauto-specdeclared an inputquality_threshold: 0.8with no rubric defined anywhere in the plugin, and Phase 5 then saidIterate until quality threshold met. The model has no data from which to compute a quality score, so it fabricates one — and that fabricated number was the loop's termination condition. A made-up number decided when implementation stopped.Replaced with exit criteria that are observable from tool output:
The
quality_thresholdinput and the--quality-threshold=0.9usage example are gone.Changes by delta
Invented numbers
quality_thresholdinput, the loop gate, and the usage example (above).Quality Score: [X/10]from the output template — nothing measures it.Unit Tests: [X% coverage]→ coverage is reported only if a coverage tool actually ran, quoting its output; the line is omitted otherwise. Test result must be quoted from the real run.implement-specoutput: replaced undefinedquality metricsanddurationwith the observed result of the test/lint/typecheck run, plus an explicit "do not report a score you did not measure".Iterate until consensus reached(unbounded, undefined "consensus") → revise until each finding is applied or explicitly rejected, then stop.Delta 2 — delegation ceilings
auto-spec: rosters are a menu, not a checklist; ceiling of 4 concurrent agents, stated as a ceiling not a quota; and a "when NOT to delegate" rule (don't spawn an agent for work that finishes in a handful of direct tool calls).implement-specPhase 3, which previously read as a prescribed team per task type over an unbounded task list withparallel_execution: true.Delta 1 — verification ceremony
implement-specPhase 4 now prefers running the project's own test/lint/typecheck commands over spawning an agent to judge quality: "a tool result is evidence, an agent's opinion of code it just read is not." Grounding instructions were kept, not removed.Delta 3 — literal instruction-following / decorative emphasis
auto-specstated its autonomy contract three times (header block, a**CRITICAL**bullet list, and a numberedCRITICAL INSTRUCTIONSsection). Kept one operative statement — the numbered contract, which is the only one naming the specific tool never to call. Removed the duplicate**CRITICAL**block and the(NOT user review)/(INSTEAD of user review)parentheticals that restated it on six more lines.CRITICAL/NEVER/ALWAYS/ANYdecoration in that block while keeping the constraints themselves intact.Delta 4 — output length
auto-spec: the four generated documents feed a later agent's context, so ~600 words each unless the feature needs more. The skill previously said "comprehensive" with no length guidance at all.Broken references
implement-specreferencedagent-capability-analyst, which does not exist as an agent anywhere in the repo (verified against everyagents/*.mdname:frontmatter value). Replaced with a plain description of the behavior.implement-specPhase 4 listedreview-planamong agents; it is a skill indevelopment-planning, andTask(subagent_type:)takes agent names. Removed.planner-agent,plan-reviewer-agent,security-analyzer-agent,performance-analyzer-agent,refactorer-agent,test-writer-agent,code-explainer-agent,agent-orchestrator-agent,code-generator-agent,documentation-agent,style-enforcer-agent,agent-tester-agent,infrastructure-agent,cicd-agent,migration-assistant-agent.Stale facts
MultiEdit(*)fromauto-spec'sallowed-tools; the tool no longer exists andEditcovers its cases.budget_tokens, or OpenAI-only params appear anywhere in this plugin.Repo conventions
allowed-toolschange and a changed loop-exit condition are behavior changes), in this commit.CLAUDE.mdversion table row updated in the same commit.CLAUDE.mdgained a short "Agent Staffing Conventions" section documenting the ceiling and the observable-exit-criteria rule.README.mdneeded no change: no component was added, removed, or renamed.Test plan
bunx nx format:write --uncommitted— clean, no diff produced.bunx markdownlint-cli2 --fix "packages/plugins/spec-workflow/**/*.md"—Summary: 0 error(s), exit 0.node scripts/validate-plugin.cjs packages/plugins/spec-workflow—Validation PASSED(plugin.json: spec-workflow v2.1.0, 2 skills, 1 MCP server).name:frontmatter from everypackages/plugins/**/agents/*.mdand diffed it against every agent named in both skills.agent-capability-analystwas the only miss (also referenced in prose by two files outside this plugin, left untouched as out of scope).grep -rn -i 'quality_threshold|quality threshold|consensus|MultiEdit' packages/plugins/spec-workflow/returns nothing.🤖 Generated with Claude Code