Skip to content

feat: improve skill score for generate-agent-skills #45

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
yogesh-tessl wants to merge 2 commits into
base: main
Choose a base branch
from
+51 −255
Open

feat: improve skill score for generate-agent-skills #45

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6a0314c
feat: improve skill score for generate-agent-skills
yogesh-tessl May 29, 2026
95459fa
feat: restore LLM behavioral anchors per review feedback
yogesh-tessl Jun 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
306 changes: 51 additions & 255 deletions skills/generate-agent-skills/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
name: generate-agent-skills
description: Architects, generates, and validates Agent Skills. Enforces specification and best practices. Used any time an agent skill must be created or updated.
description: "Creates and updates SKILL.md files with YAML frontmatter, workflow steps, and bundle resources. Scaffolds directories via scaffold_skill.py and validates via validate_skill.py. Use when creating a new agent skill, writing a skill.md, updating skill definitions, or generating skill templates."
compatibility: python3
allowed-tools: python3 ls grep cat mkdir
metadata:
Expand All @@ -10,27 +10,13 @@ metadata:

# Agent Skill Architect Workflow

This skill guides you through creating high-quality Agent Skills following a proven 6-step process.
Create high-quality Agent Skills following a 6-step process: understand → plan → scaffold → write → validate → test.

---

## 🚨 **CRITICAL WORKFLOW REQUIREMENTS** 🚨

**Before you begin, understand these NON-NEGOTIABLE rules:**

1. **You MUST run `scripts/scaffold_skill.py` in Step 3.**
Manual file creation is PROHIBITED. The scaffolding script ensures consistency.

2. **You MUST use the generated templates.**
After scaffolding, templates exist in `references/`. Use them as your foundation.

3. **You MUST run `scripts/validate_skill.py` in Step 5.**
Validation catches errors before they propagate.

4. **You MUST follow all 6 steps in order.**
Skipping steps leads to non-compliant or broken skills.

**If you bypass scaffolding scripts, you have FAILED this workflow.**
**Mandatory constraints:**
1. Run `scripts/scaffold_skill.py` in Step 3 — manual file creation is prohibited.
2. Use the generated templates from `references/` as your foundation.
3. Run `scripts/validate_skill.py` in Step 5 before finalizing.
4. Follow all 6 steps in order.

---

Expand Down Expand Up @@ -62,157 +48,54 @@ If working with an existing skill, analyze:

## Step 2: Planning Reusable Contents

Analyze the concrete examples from Step 1 to identify what **reusable resources** would help.

### ⚠️ Critical Decision: Script vs. Checklist

**Before planning scripts, ask:** "Is this task primarily analysis or computation?"

**Analysis tasks** (reading, synthesizing, pattern recognition):
→ Use **checklists** or **reference docs** for LLM to follow
→ Examples: Repository analysis, code review, documentation synthesis

**Computation tasks** (math, APIs, precise transformations):
→ Use **scripts** for deterministic execution
→ Examples: Schema validation, API calls, file format conversion

**Real example from this session:**

@srbouffard srbouffard May 29, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The analyze_repo.py → analysis_checklist.md real-world example is a great anchor for LLMs that tend to over-script analysis tasks. Would you be open to keeping that inline example and pointing to BEST_PRACTICES.md §6 after it rather than replacing it? Keeps the token saving on the taxonomy lists while preserving the reasoning anchor.

- ❌ Initially planned `analyze_repo.py` script
- ✅ Corrected to `analysis_checklist.md` reference
- **Why:** Repository analysis = LLM strength (reading, pattern detection, synthesis)

**See `references/BEST_PRACTICES.md` §6 for detailed decision flowchart.**

---

### Ask for each example:
1. How would I execute this task from scratch?
2. What scripts, references, or assets would make this repeatable?
3. **Is this analysis (LLM) or computation (script)?**
Analyze examples from Step 1 to identify reusable resources. For each, decide: analysis tasks → `references/` (checklists, patterns, domain knowledge); computation tasks → `scripts/` (math, APIs, validation); output artifacts → `assets/` (templates, images, seed data).

### Resource Types:
**Real example:** A repository-analysis task was initially planned as an `analyze_repo.py` script, then corrected to an `analysis_checklist.md` reference — repository analysis is an LLM strength (reading, pattern detection, synthesis), not deterministic computation. When in doubt, prefer a checklist over a script for analysis work. See `references/BEST_PRACTICES.md` §6 for the full decision flowchart.

**scripts/** - For deterministic operations only:
- ✅ Math/computation (calculations, aggregations)
- ✅ External interactions (API calls, database queries)
- ✅ Precise transformations (file format conversion, schema validation)
- ✅ Repetitive generation (boilerplate rendering)
- ❌ Analysis tasks (use checklists instead)
- ❌ Pattern recognition (LLM excels at this)

**references/** - For LLM-driven analysis and knowledge:
- ✅ Checklists for systematic analysis (e.g., repository discovery)
- ✅ Pattern libraries (e.g., positive constraint conversions)
- ✅ API documentation (endpoints, parameters)
- ✅ Domain knowledge (company policies, industry standards)
- ✅ Decision trees and workflows

**assets/** - For files used in output:
- ✅ Templates (documents, slides, boilerplate code)
- ✅ Images (logos, icons, diagrams)
- ✅ Fonts (typography files)
- ✅ Seed data (sample datasets, fixtures)

**Output:** A list of specific files to create with correct categorization (script vs reference)
**Output:** A list of specific files to create with correct categorization.

---

## Step 3: Skill Scaffolding

**⚠️ MANDATORY STEP - DO NOT SKIP ⚠️**
Run the scaffolding script (use `--simple` for SKILL.md only):

You MUST execute the scaffolding script. Manual file creation is PROHIBITED.

### Command:
```bash
python3 scripts/scaffold_skill.py --name <skill-name>
```

### Options:
- **Default mode:** Creates SKILL.md + example files in scripts/, references/, assets/
- **Simple mode:** Use `--simple` flag for minimal structure (SKILL.md only)

**The script will:**
- ✅ Validate naming conventions (lowercase, hyphens, alphanumeric)
- ✅ Create skill directory in .github/skills/
- ✅ Generate SKILL.md with structuring guidance
- ✅ Create example files to demonstrate resource organization
Validates naming (`^[a-z0-9][a-z0-9-]*[a-z0-9]$`), creates the directory under `.github/skills/`, and generates SKILL.md with placeholders. Verify with `ls -la .github/skills/<skill-name>/` before proceeding.

@srbouffard srbouffard May 29, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you be able to add back a few explicit stop conditions here? Something like: "If SKILL.md does NOT exist after running the script → do NOT proceed." These act as LLM behavioral anchors, they're different from documentation in that they halt the agent on failure rather than just inform it.


**Note:** The script auto-detects `.github/skills` from git root. Naming must match regex: `^[a-z0-9][a-z0-9-]*[a-z0-9]$`

---

### ✅ **Verification Checkpoint**

After running the scaffolding script, confirm these files exist:
```bash
ls -la .github/skills/<skill-name>/
```

**Expected output:**
- `SKILL.md` (with "Structuring This Skill" guidance section)
- `scripts/example.py` (placeholder script)
- `references/example_reference.md` (placeholder reference)
- `assets/README.md` (if using default mode)

**🛑 STOP CONDITIONS:**
- If `SKILL.md` does NOT exist → Scaffolding failed, do NOT proceed
- If you created files manually → You have violated the workflow, DELETE and re-run script
- If the script reported errors → Fix errors before proceeding to Step 4
**Stop conditions:**
- If `SKILL.md` does NOT exist after running the script → do NOT proceed; the scaffolding failed.
- If you created files manually instead of running the script → delete them and re-run the script.
- If the script reported errors → fix them before continuing to Step 4.

---

## Step 4: Content Generation

Populate the skill with actual content.

### 4.1: Implement Reusable Resources First

Start with scripts/, references/, and assets/ identified in Step 2.

**For scripts:**
- Replace `scripts/example.py` with actual implementation
- Test by running: `python3 scripts/<script_name>.py`
- Ensure error messages are descriptive (print to stderr)

**For references:**
- Replace `references/example_reference.md` with actual docs
- Keep SKILL.md lean - move details here
- For large files (>100 lines), add Table of Contents
### 4.1: Implement Reusable Resources

**For assets:**
- Add actual template files, images, fonts
- Replace or delete `assets/README.md`
- Use descriptive filenames

**Important:** Delete any example files you don't need!
Replace placeholder files from scaffolding with actual implementations:
- **scripts/**: Replace `example.py` with real scripts. Test each: `python3 scripts/<name>.py`
- **references/**: Replace `example_reference.md` with actual docs. Keep SKILL.md lean — move details here.
- **assets/**: Add template files, images, seed data. Delete unused placeholders.

### 4.2: Write SKILL.md Content

Follow the structuring guidance embedded in the generated SKILL.md template.

**Choose your structure pattern:**
- **Workflow-Based:** Sequential processes (see `references/workflows.md`)
- **Task-Based:** Tool collections with different operations
- **Reference/Guidelines:** Standards, specifications, coding rules
- **Capabilities-Based:** Integrated systems with multiple features

**Key elements:**
**Frontmatter (YAML):**
- `name`: Must match directory name exactly.
- `description`: High-entropy, keyword-rich, 3rd person. Include a "Use when..." clause with trigger scenarios and concrete capabilities.
- Example: `"Processes PDF documents for form filling, text extraction, and merging. Use when working with PDF files or when user requests document manipulation tasks."`

1. **Frontmatter (YAML):**
- `name`: Must match directory name exactly
- `description`: High-entropy, keyword-rich, 3rd person
- Include WHEN to use this skill (triggers)
- Include WHAT the skill does (capabilities)
- Example: "Processes PDF documents for form filling, text extraction, and merging. Use when working with PDF files or when user requests document manipulation tasks."
**Body (Markdown):**
- Use imperative form ("Run the script", not "You should run").
- Reference scripts/references explicitly by path.
- Choose a structure pattern: workflow-based, task-based, reference/guidelines, or capabilities-based (see `references/workflows.md`).
- Consult `references/BEST_PRACTICES.md` for the Freedom Scale, `references/output-patterns.md` for output formatting.

2. **Body (Markdown):**
- Use imperative/infinitive form ("Run the script", not "You should run")
- Reference scripts/references explicitly by path
- Consult `references/BEST_PRACTICES.md` for the "Freedom Scale"
- Consult `references/output-patterns.md` for output formatting

**Delete the "Structuring This Skill" section** when done - it's guidance only!
Delete the "Structuring This Skill" guidance section when done.

### 4.3: Design Patterns

Expand All @@ -231,126 +114,39 @@ Follow the structuring guidance embedded in the generated SKILL.md template.

## Step 5: Validation

**⚠️ MANDATORY STEP - DO NOT SKIP ⚠️**

Run the validation script to ensure specification compliance.
Run the validation script to ensure specification compliance:

### Command:
```bash
python3 scripts/validate_skill.py --path <path-to-skill-root>
python3 scripts/validate_skill.py --path .github/skills/<skill-name>
```

**Example:**
```bash
python3 scripts/validate_skill.py --path .github/skills/diagnose-ci-failure
```
Checks: directory naming, SKILL.md exists, required frontmatter fields (`name`, `description`), name matches directory. Warnings about missing `references/` or `scripts/` are advisory.

### What it checks:
- ✅ Directory naming regex (`^[a-z0-9][a-z0-9-]*[a-z0-9]$`)
- ✅ SKILL.md exists
- ✅ YAML frontmatter has required fields (name, description)
- ✅ Name in YAML matches directory name
- ⚠️ Advisory: Presence of references/ and scripts/ (warnings only)
Fix critical errors before proceeding. Then confirm:
- [ ] Ran `scripts/scaffold_skill.py` (did not create files manually)
- [ ] Ran `scripts/validate_skill.py` with no critical errors
- [ ] Frontmatter `description` includes a "Use when..." clause
- [ ] No placeholder files (`example.py`, `example_reference.md`) remain

**If validation fails:**
- Read the error output carefully
- Fix critical violations immediately
- Warnings are informational (acceptable for simple skills)

**When valid:** Proceed to testing!

---

### ✅ **Post-Validation Checklist**

Before proceeding to Step 6, confirm:

**Workflow Compliance:**
- [ ] I RAN `scripts/scaffold_skill.py` (Step 3)
- [ ] I USED the generated templates from scaffolding
- [ ] I CONSULTED `references/TEMPLATES.md` and `references/BEST_PRACTICES.md` (Step 4)
- [ ] I RAN `scripts/validate_skill.py` (Step 5)
- [ ] Validation script reported SUCCESS (no critical errors)

**Content Quality:**
- [ ] YAML frontmatter includes `name` and `description`
- [ ] Description is high-entropy and keyword-rich
- [ ] No "Structuring This Skill" guidance section remains in SKILL.md
- [ ] Example files (`example.py`, `example_reference.md`) are deleted or replaced
- [ ] Scripts are in `scripts/`, references in `references/`, templates in `assets/`

**🛑 STOP CONDITION:**
If you did NOT run the scaffolding script or manually created files, STOP and re-do from Step 3.
If you did not run the scaffolding script or manually created files, STOP and re-do from Step 3.

---

## Step 6: Testing and Iteration

After creating the skill, test and refine based on real usage.

### Testing Workflow:

1. **Test with real examples** from Step 1
- Does the skill trigger on expected queries?
- Do scripts execute without errors?
- Is output quality acceptable?

2. **Identify friction points:**
- Are instructions clear enough?
- Are there missing scripts or references?
- Is context loaded efficiently?

3. **Iterate on improvements:**
- Update SKILL.md for clarity
- Add missing examples or edge cases
- Optimize script error handling
- Split large references if needed (progressive disclosure)

4. **Re-validate** after changes

### Common Iteration Patterns:

**Problem:** Skill isn't triggering when expected
**Solution:** Enhance description with more keywords and trigger scenarios

**Problem:** Agent struggles with workflow steps
**Solution:** Add decision tree or flowchart; consult `references/workflows.md`

**Problem:** Context feels bloated
**Solution:** Move content from SKILL.md to references/; add grep hints

**Problem:** Scripts fail in edge cases
**Solution:** Add error handling; print descriptive messages to stderr

**Problem:** Output quality inconsistent
**Solution:** Add templates or validation checklist; see `references/output-patterns.md`

### When to Stop Iterating:

✅ Skill triggers reliably on target queries
✅ Workflows execute without confusion
✅ Output quality meets requirements
✅ No critical errors in testing

---

## Knowledge Retrieval

If questions arise during skill creation:

**Specification questions** (naming, structure, required files):
→ Read `references/SPECIFICATION.md`

**Best practices** (context economy, freedom scale, anti-patterns):
→ Read `references/BEST_PRACTICES.md`
Test the skill with real examples from Step 1, then iterate:

**Templates and examples** (frontmatter, structure patterns):
→ Read `references/TEMPLATES.md`
1. **Trigger testing**: Does the skill activate on expected queries? If not, add keywords to the description.
2. **Execution testing**: Do scripts run without errors? Is output quality acceptable?
3. **Context efficiency**: If SKILL.md feels bloated, move content to `references/`.
4. **Re-validate** after each round of changes with `scripts/validate_skill.py`.

**Workflow design** (sequential, conditional, iterative):
→ Read `references/workflows.md`
## Reference Index

**Output formatting** (templates, examples, validation):
→ Read `references/output-patterns.md`
- **Specification** (naming, structure): `references/SPECIFICATION.md`
- **Best practices** (context economy, freedom scale): `references/BEST_PRACTICES.md`
- **Templates** (frontmatter, structure patterns): `references/TEMPLATES.md`
- **Workflows** (sequential, conditional, iterative): `references/workflows.md`
- **Output patterns** (templates, validation checklists): `references/output-patterns.md`

@srbouffard srbouffard May 29, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add **Do not hallucinate answers.** Always consult the authoritative sources. back after the table? It's a direct instruction to the model and helps meaningfully reduces fabricated references.


**Do not hallucinate answers.** Always consult the authoritative sources.