Skip to content

Commit fab07b5

Browse files
priyanshu92claude
priyanshu92
and
claude
authored
Add skill evaluation framework (#7)
- Add Python-based eval framework for testing Claude Code skills - Implement two-stage grading: deterministic checks + model-assisted rubrics - Create per-skill rubric.json files with weighted assertions - Add prompts.csv test cases for all 8 power-pages skills - Include full response logging in report.json - Add shared utilities (runner, grader, helpers) - Add root .gitignore for artifacts/temp/reports Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
1 parent 465fca9 commit fab07b5

24 files changed

Lines changed: 1916 additions & 0 deletions

File tree

‎.gitignore‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
# Eval artifacts (generated during test runs)
2+
evals/artifacts/
3+
evals/temp/
4+
5+
# Eval reports (aggregated results)
6+
evals/reports/
7+
8+
# Python
9+
__pycache__/
10+
*.py[cod]
11+
*$py.class
12+
*.so
13+
.Python
14+
build/
15+
develop-eggs/
16+
dist/
17+
downloads/
18+
eggs/
19+
.eggs/
20+
lib64/
21+
parts/
22+
sdist/
23+
var/
24+
wheels/
25+
*.egg-info/
26+
.installed.cfg
27+
*.egg
28+
.venv/
29+
venv/
30+
ENV/
31+
32+
# Logs
33+
*.log
34+
35+
# IDE
36+
.idea/
37+
.vscode/
38+
*.swp
39+
*.swo
40+
41+
# OS
42+
.DS_Store
43+
Thumbs.db

‎evals/README.md‎

Lines changed: 183 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,183 @@
1+
# Evals Framework for Claude Code Skills
2+
3+
A Python framework for systematically evaluating Claude Code skills using a two-stage grading approach.
4+
5+
## Quick Start
6+
7+
```bash
8+
# Run a single test
9+
python run_eval.py --plugin power-pages --skill create-site --test test-01
10+
11+
# Run all tests for a skill
12+
python run_eval.py --plugin power-pages --skill create-site --all
13+
14+
# Run with model-assisted grading
15+
python run_eval.py --plugin power-pages --skill create-site --all --grade
16+
17+
# Run all skills in a plugin
18+
python run_eval.py --plugin power-pages --all
19+
```
20+
21+
## Folder Structure
22+
23+
```
24+
evals/
25+
├── README.md # This file
26+
├── run_eval.py # Main eval runner CLI
27+
├── grade_eval.py # Standalone grading CLI
28+
├── __init__.py
29+
├── shared/ # Shared utilities
30+
│ ├── __init__.py
31+
│ ├── helpers.py # Data classes and helper functions
32+
│ ├── runner.py # EvalRunner class
33+
│ ├── grader.py # Grader class
34+
│ └── rubrics/ # Grading rubric schemas
35+
│ ├── skill-trigger.schema.json
36+
│ ├── code-quality.schema.json
37+
│ └── process.schema.json
38+
├── power-pages/ # Plugin-specific evals
39+
│ ├── __init__.py
40+
│ ├── create-site/
41+
│ │ └── prompts.csv
42+
│ ├── setup-webapi/
43+
│ │ └── prompts.csv
44+
│ ├── setup-dataverse/
45+
│ │ └── prompts.csv
46+
│ ├── setup-auth/
47+
│ │ └── prompts.csv
48+
│ ├── integrate-webapi/
49+
│ │ └── prompts.csv
50+
│ ├── add-seo/
51+
│ │ └── prompts.csv
52+
│ ├── add-tests/
53+
│ │ └── prompts.csv
54+
│ └── add-sample-data/
55+
│ └── prompts.csv
56+
├── artifacts/ # Test outputs (gitignored)
57+
└── reports/ # Aggregate reports (gitignored)
58+
```
59+
60+
## Prompt CSV Format
61+
62+
Each skill has a `prompts.csv` file defining test cases:
63+
64+
```csv
65+
id,should_trigger,prompt,expected_skill,notes
66+
test-01,true,"Create a demo app using /create-site",create-site,Explicit skill invocation
67+
test-02,true,"Build a React Power Pages site",create-site,Implicit invocation
68+
test-03,false,"Add Tailwind to my existing app",create-site,Negative control
69+
```
70+
71+
**Fields:**
72+
- `id`: Unique test identifier
73+
- `should_trigger`: Whether the skill should be invoked (true/false)
74+
- `prompt`: The user prompt to test
75+
- `expected_skill`: The skill expected to trigger (for positive cases)
76+
- `notes`: Description of what the test validates
77+
78+
## Two-Stage Grading
79+
80+
### Stage 1: Deterministic Checks (Fast)
81+
82+
Automated checks parsed from the execution trace:
83+
- Execution success (no errors)
84+
- Response generated
85+
- Reasonable token usage (< 50k)
86+
- Reasonable turn count (< 20)
87+
88+
### Stage 2: Model-Assisted Rubric (Qualitative)
89+
90+
Claude evaluates against rubric criteria:
91+
- **skill_invocation**: Correct skill invoked/not invoked
92+
- **response_quality**: Helpful, clear, appropriate response
93+
- **tool_usage**: Tools used efficiently
94+
- **plan_mode**: Plan mode used when appropriate
95+
96+
## CLI Reference
97+
98+
### run_eval.py
99+
100+
```
101+
python run_eval.py --plugin PLUGIN --skill SKILL --test TEST_ID
102+
python run_eval.py --plugin PLUGIN --skill SKILL --all [--grade]
103+
python run_eval.py --plugin PLUGIN --all [--grade]
104+
105+
Arguments:
106+
--plugin Plugin name (required, e.g., power-pages)
107+
--skill Skill name (optional, e.g., create-site)
108+
--test Specific test ID (e.g., test-01)
109+
--all Run all tests
110+
--grade Enable model-assisted grading
111+
--timeout Timeout per test in seconds (default: 300)
112+
```
113+
114+
### grade_eval.py
115+
116+
```
117+
python grade_eval.py --plugin PLUGIN --skill SKILL --test TEST_ID
118+
python grade_eval.py --artifacts-path PATH
119+
120+
Arguments:
121+
--plugin Plugin name
122+
--skill Skill name
123+
--test Test ID to grade
124+
--artifacts-path Direct path to artifacts (alternative)
125+
```
126+
127+
## How It Works
128+
129+
1. **Load prompts**: Read test cases from `prompts.csv`
130+
2. **Execute Claude**: Run `claude -p --output-format=json "<prompt>"`
131+
3. **Parse output**: Extract result, metrics, and tool usage
132+
4. **Check trigger**: Determine if skill was invoked
133+
5. **Grade (optional)**: Run two-stage grading
134+
6. **Save artifacts**: Store results in `artifacts/` directory
135+
136+
## Output Format
137+
138+
Claude outputs JSON in this format:
139+
140+
```json
141+
{
142+
"type": "result",
143+
"subtype": "success",
144+
"is_error": false,
145+
"result": "Response text...",
146+
"session_id": "...",
147+
"duration_ms": 4437,
148+
"num_turns": 1,
149+
"total_cost_usd": 0.19,
150+
"usage": {
151+
"input_tokens": 2,
152+
"output_tokens": 12,
153+
"cache_creation_input_tokens": 30452
154+
}
155+
}
156+
```
157+
158+
## Adding New Evals
159+
160+
1. Create a new skill folder under the plugin directory
161+
2. Add `prompts.csv` with test cases
162+
3. Run with `python run_eval.py --plugin <plugin> --skill <skill> --all`
163+
164+
## Interpreting Results
165+
166+
### Pass Criteria
167+
168+
- **Skill Trigger**: Correct skill invoked (or not invoked for negative cases)
169+
- **Process**: Execution completed successfully without errors
170+
- **Quality**: Response is helpful and appropriate (model-assisted)
171+
172+
### Score Ranges
173+
174+
- **90-100**: Excellent - All checks pass
175+
- **70-89**: Good - Minor issues
176+
- **50-69**: Fair - Some checks fail
177+
- **0-49**: Poor - Major issues
178+
179+
## Requirements
180+
181+
- Python 3.10+
182+
- Claude CLI installed and configured
183+
- No additional Python dependencies (uses stdlib only)

‎evals/grade_eval.py‎

Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
#!/usr/bin/env python3
2+
"""
3+
Standalone grading script for Claude Code skill evaluations.
4+
5+
Usage:
6+
python grade_eval.py --plugin power-pages --skill create-site --test test-01
7+
python grade_eval.py --artifacts-path artifacts/power-pages/create-site/test-01
8+
"""
9+
10+
import argparse
11+
import os
12+
import sys
13+
from pathlib import Path
14+
15+
# Add parent to path for imports
16+
sys.path.insert(0, str(Path(__file__).parent))
17+
18+
from shared import Grader, load_prompts, print_colored
19+
20+
21+
def main():
22+
parser = argparse.ArgumentParser(
23+
description="Grade Claude Code skill evaluation results",
24+
formatter_class=argparse.RawDescriptionHelpFormatter,
25+
epilog="""
26+
Examples:
27+
python grade_eval.py --plugin power-pages --skill create-site --test test-01
28+
python grade_eval.py --artifacts-path artifacts/power-pages/create-site/test-01
29+
""",
30+
)
31+
32+
parser.add_argument(
33+
"--plugin",
34+
help="Plugin name (e.g., power-pages)",
35+
)
36+
parser.add_argument(
37+
"--skill",
38+
help="Skill name (e.g., create-site)",
39+
)
40+
parser.add_argument(
41+
"--test",
42+
help="Test ID to grade (e.g., test-01)",
43+
)
44+
parser.add_argument(
45+
"--artifacts-path",
46+
help="Direct path to artifacts directory (alternative to plugin/skill/test)",
47+
)
48+
49+
args = parser.parse_args()
50+
51+
base_dir = Path(__file__).parent
52+
53+
# Determine artifacts path
54+
if args.artifacts_path:
55+
artifacts_dir = Path(args.artifacts_path)
56+
if not artifacts_dir.is_absolute():
57+
artifacts_dir = base_dir / artifacts_dir
58+
test_id = artifacts_dir.name
59+
test_metadata = None
60+
elif args.plugin and args.skill and args.test:
61+
artifacts_dir = base_dir / "artifacts" / args.plugin / args.skill
62+
test_id = args.test
63+
64+
# Load test metadata
65+
prompts_file = base_dir / args.plugin / args.skill / "prompts.csv"
66+
if prompts_file.exists():
67+
tests = load_prompts(str(prompts_file))
68+
test_metadata = next((t for t in tests if t.id == test_id), None)
69+
else:
70+
test_metadata = None
71+
else:
72+
parser.error("Either --artifacts-path or (--plugin, --skill, --test) must be specified")
73+
74+
if not artifacts_dir.exists():
75+
print_colored(f"Artifacts directory not found: {artifacts_dir}", "red")
76+
sys.exit(1)
77+
78+
test_artifacts = artifacts_dir / test_id if not args.artifacts_path else artifacts_dir
79+
if not test_artifacts.exists():
80+
print_colored(f"Test artifacts not found: {test_artifacts}", "red")
81+
sys.exit(1)
82+
83+
# Load skill-specific rubric if available
84+
rubric_path = None
85+
if args.plugin and args.skill:
86+
rubric_file = base_dir / args.plugin / args.skill / "rubric.json"
87+
if rubric_file.exists():
88+
rubric_path = str(rubric_file)
89+
90+
print_colored("=" * 50, "cyan")
91+
print_colored("CLAUDE CODE EVAL GRADING", "cyan")
92+
print_colored("=" * 50, "cyan")
93+
print_colored(f"Test: {test_id}", "white")
94+
print_colored(f"Artifacts: {test_artifacts}", "gray")
95+
if rubric_path:
96+
print_colored(f"Rubric: {rubric_path}", "gray")
97+
print_colored("=" * 50, "cyan")
98+
99+
grader = Grader(
100+
artifacts_dir=str(artifacts_dir if not args.artifacts_path else test_artifacts.parent),
101+
rubric_path=rubric_path,
102+
)
103+
grader.grade(test_id, test_metadata)
104+
105+
106+
if __name__ == "__main__":
107+
main()

‎evals/power-pages/add-sample-data/prompts.csv‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
id,should_trigger,prompt,expected_skill,notes
2+
test-01,true,"Add sample data using /add-sample-data",add-sample-data,Explicit skill invocation
3+
test-02,true,"Insert test data into my Dataverse tables",add-sample-data,Implicit - inserting test data
4+
test-03,true,"Populate the database with sample records",add-sample-data,Implicit - populating with samples
5+
test-04,true,"Create demo data for products and orders tables",add-sample-data,Implicit - demo data creation
6+
test-05,true,"I need fake data to test my portal",add-sample-data,Implicit - fake data for testing
7+
test-06,false,"How do I delete sample data?",add-sample-data,Negative control - deletion not creation
8+
test-07,false,"Export my Dataverse data to CSV",add-sample-data,Negative control - export not import
9+
test-08,false,"Why is my data not showing up?",add-sample-data,Negative control - troubleshooting
10+
test-09,true,"Generate realistic test records for my tables",add-sample-data,Implicit - generating test records
11+
test-10,false,"What format should my import data be in?",add-sample-data,Negative control - informational question

‎evals/power-pages/add-sample-data/rubric.json‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
{
2+
"name": "add-sample-data",
3+
"description": "Evaluates the add-sample-data skill execution",
4+
"checks": [
5+
{
6+
"id": "read_memory_bank",
7+
"description": "Read memory-bank.md to understand table structure",
8+
"weight": 25
9+
},
10+
{
11+
"id": "mentioned_relationships",
12+
"description": "Discussed handling foreign key relationships in sample data",
13+
"weight": 20
14+
},
15+
{
16+
"id": "mentioned_realistic_data",
17+
"description": "Discussed generating realistic/meaningful test data",
18+
"weight": 20
19+
},
20+
{
21+
"id": "mentioned_pac_cli",
22+
"description": "Mentioned using PAC CLI or Web API for data insertion",
23+
"weight": 15
24+
},
25+
{
26+
"id": "response_quality",
27+
"description": "Response is helpful, clear, and appropriate for the request",
28+
"weight": 20
29+
}
30+
]
31+
}

‎evals/power-pages/add-seo/prompts.csv‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
id,should_trigger,prompt,expected_skill,notes
2+
test-01,true,"Add SEO using /add-seo",add-seo,Explicit skill invocation
3+
test-02,true,"Add meta tags and sitemap to my Power Pages site",add-seo,Implicit - SEO assets mentioned
4+
test-03,true,"Optimize my site for search engines",add-seo,Implicit - SEO optimization
5+
test-04,true,"I need robots.txt and sitemap.xml for my portal",add-seo,Implicit - specific SEO files
6+
test-05,true,"Add favicon and Open Graph tags",add-seo,Implicit - SEO-related assets
7+
test-06,false,"How does Google index Power Pages sites?",add-seo,Negative control - informational question
8+
test-07,false,"Why isn't my site appearing in Google search?",add-seo,Negative control - troubleshooting
9+
test-08,false,"Change the page title in my React component",add-seo,Negative control - component change not SEO setup
10+
test-09,true,"Set up search engine optimization assets",add-seo,Implicit - SEO assets setup
11+
test-10,false,"What meta tags should I use for my site?",add-seo,Negative control - recommendation question

0 commit comments

Comments
 (0)