Skip to content

Commit 7157d03

Browse files
spboyerspboyer
committed
docs: Add Product Requirements Document (PRD)
- Defines waza platform vision and requirements - Documents 7 epics with 42 user stories - Establishes compliance scoring system - Outlines phased roadmap (Primary/Secondary) References: microsoft#66 (tracking issue) Source: squad-proposal.md
1 parent c4141dc commit 7157d03

1 file changed

Lines changed: 266 additions & 0 deletions

File tree

‎docs/PRD.md‎

Lines changed: 266 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,266 @@
1+
# Waza Skills Development Platform - Product Requirements Document
2+
3+
**Status:** Active
4+
**Version:** 1.0
5+
**Last Updated:** 2026-02-06
6+
**Owner:** @spboyer
7+
**Source:** [Squad Proposal](https://github.com/spboyer/azure-mcp-v-skills/blob/main/squad-proposal.md)
8+
9+
---
10+
11+
## Executive Summary
12+
13+
Waza (技 - Japanese for "skill/technique") is a unified CLI platform for creating, testing, and evaluating AI agent skills. It consolidates existing skill development tools into a single binary that provides the complete developer experience for contributing to [microsoft/skills](https://github.com/microsoft/skills).
14+
15+
### The Problem
16+
17+
The microsoft/skills repository hosts 132+ skills for AI coding agents, but the contribution process lacks automated tooling for:
18+
19+
- **Compliance validation** — No standardized scoring before PR submission
20+
- **Trigger testing** — Manual verification of skill activation patterns
21+
- **Cross-model evaluation** — No framework for testing skills across GPT-4o, Claude, etc.
22+
- **Token budget enforcement** — Guidelines exist but aren't automatically checked
23+
24+
### The Solution
25+
26+
A single `waza` CLI built in **Go** that automates the skill development workflow:
27+
28+
| Phase | Capability |
29+
|-------|------------|
30+
| **Scaffold** | Generate compliant skill structure matching microsoft/skills conventions |
31+
| **Develop** | Iterate with real-time compliance scoring (Sensei engine) |
32+
| **Test** | Run agentic test loops with real LLM execution via Copilot SDK |
33+
| **Evaluate** | Cross-model comparison with task completion, trigger accuracy, behavior quality metrics |
34+
35+
---
36+
37+
## User Personas
38+
39+
### Primary: Skill Author
40+
- **Role:** Developer contributing skills to microsoft/skills
41+
- **Goals:** Create high-quality skills that pass CI, work across models
42+
- **Pain Points:** Manual testing, unclear compliance requirements, no cross-model validation
43+
44+
### Secondary: Skill Reviewer
45+
- **Role:** Maintainer reviewing skill PRs
46+
- **Goals:** Quickly assess skill quality, ensure compliance
47+
- **Pain Points:** Inconsistent quality, manual verification
48+
49+
### Tertiary: Platform Engineer
50+
- **Role:** CI/CD pipeline maintainer
51+
- **Goals:** Automate skill validation in pipelines
52+
- **Pain Points:** Lack of CLI tools for automation
53+
54+
---
55+
56+
## Feature Requirements
57+
58+
### Epic 1: Go CLI Foundation (P0)
59+
60+
Port existing Python waza functionality to Go for single-binary distribution.
61+
62+
| ID | User Story | Acceptance Criteria |
63+
|----|------------|---------------------|
64+
| E1-01 | As a developer, I can run evaluations with `waza run` | Parses eval.yaml, executes tasks, outputs results |
65+
| E1-02 | As a developer, I can initialize new eval suites with `waza init` | Creates compliant directory structure |
66+
| E1-03 | As a developer, I can generate evals from SKILL.md with `waza generate` | Parses SKILL.md, creates tasks and fixtures |
67+
| E1-04 | As a developer, I can compare results across models with `waza compare` | Loads multiple result files, generates comparison report |
68+
| E1-05 | As a developer, I can use all 8 grader types | code, model, regex, file, keyword, json, script, composite |
69+
| E1-06 | As a developer, I can execute against Copilot SDK | Full integration with streaming responses |
70+
| E1-07 | As a developer, I can use verbose mode for debugging | Real-time conversation display |
71+
| E1-08 | As a developer, I can save transcripts for analysis | JSON log output with full conversation |
72+
73+
### Epic 2: Sensei Engine (P0)
74+
75+
Compliance scoring and iterative improvement loop.
76+
77+
| ID | User Story | Acceptance Criteria |
78+
|----|------------|---------------------|
79+
| E2-01 | As a developer, I can run `waza dev` to start the improvement loop | Iterative scoring with feedback |
80+
| E2-02 | As a developer, I can see my compliance score (Low/Medium/Medium-High/High) | Clear scoring rubric applied |
81+
| E2-03 | As a developer, I get specific improvement suggestions | Actionable feedback per issue |
82+
| E2-04 | As a developer, I can set a target score | Loop until target reached |
83+
| E2-05 | As a developer, I can run trigger accuracy tests | shouldTrigger/shouldNotTrigger prompts |
84+
| E2-06 | As a developer, I can skip integration tests with `--skip-integration` | Unit + trigger tests only |
85+
| E2-07 | As a developer, I can use fast mode with `--fast` | Skip tests for rapid iteration |
86+
87+
### Epic 3: Evaluation Framework (P0)
88+
89+
Cross-model testing and comprehensive metrics.
90+
91+
| ID | User Story | Acceptance Criteria |
92+
|----|------------|---------------------|
93+
| E3-01 | As a developer, I can run evals against multiple models | Model parameter support |
94+
| E3-02 | As a developer, I can see task completion metrics | Pass rate, composite score |
95+
| E3-03 | As a developer, I can see trigger accuracy metrics | Activation pattern validation |
96+
| E3-04 | As a developer, I can see behavior quality metrics | Response quality scoring |
97+
| E3-05 | As a developer, I can run trials for statistical confidence | Multiple runs per task |
98+
| E3-06 | As a developer, I can get LLM-powered improvement suggestions | --suggestions flag |
99+
| E3-07 | As a developer, I can run tasks in parallel | --parallel flag |
100+
| E3-08 | As a developer, I can filter to specific tasks | --task flag |
101+
102+
### Epic 4: Token Management (P1)
103+
104+
Budget tracking and optimization tools.
105+
106+
| ID | User Story | Acceptance Criteria |
107+
|----|------------|---------------------|
108+
| E4-01 | As a developer, I can count tokens with `waza tokens count` | Token count for all markdown files |
109+
| E4-02 | As a developer, I can check limits with `waza tokens check` | Validate against budget |
110+
| E4-03 | As a developer, I can use strict mode with `--strict` | Exit 1 if limits exceeded |
111+
| E4-04 | As a developer, I can get optimization suggestions with `waza tokens suggest` | LLM-powered reduction tips |
112+
| E4-05 | As a developer, I can compare with previous commits | `waza tokens compare HEAD~1` |
113+
114+
### Epic 5: Waza Skill (P1)
115+
116+
Conversational interface for guided skill development.
117+
118+
| ID | User Story | Acceptance Criteria |
119+
|----|------------|---------------------|
120+
| E5-01 | As a developer, I can use waza as a skill in Copilot | SKILL.md published to microsoft/skills |
121+
| E5-02 | As a developer, I get guided requirements gathering | Interactive prompts for skill creation |
122+
| E5-03 | As a developer, I can check readiness conversationally | "Is my skill ready?" triggers validation |
123+
| E5-04 | As a developer, I get interpreted results | Plain language explanation of scores |
124+
| E5-05 | As a developer, the skill invokes CLI commands | Skill wraps waza CLI |
125+
126+
### Epic 6: CI/CD Integration (P1)
127+
128+
GitHub Actions and microsoft/skills compatibility.
129+
130+
| ID | User Story | Acceptance Criteria |
131+
|----|------------|---------------------|
132+
| E6-01 | As a developer, I can run waza in GitHub Actions | Action workflow template |
133+
| E6-02 | As a developer, I can fail PRs on low compliance | Exit codes for CI |
134+
| E6-03 | As a developer, I can post results to PR comments | GitHub reporter output |
135+
| E6-04 | As a developer, waza works with microsoft/skills CI | Compatible with existing test harness |
136+
| E6-05 | As a developer, I can cache evaluation results | Incremental testing support |
137+
138+
### Epic 7: AZD Extension (P2)
139+
140+
Package waza as an Azure Developer CLI extension.
141+
142+
| ID | User Story | Acceptance Criteria |
143+
|----|------------|---------------------|
144+
| E7-01 | As a developer, I can install waza with `azd extension install waza` | Published to azd extension registry |
145+
| E7-02 | As a developer, I can run `azd waza <command>` | All commands available via azd |
146+
| E7-03 | As a developer, I get IntelliSense for waza commands | Metadata support |
147+
| E7-04 | As a developer, waza integrates with azure.yaml | Configuration schema support |
148+
149+
---
150+
151+
## Technical Architecture
152+
153+
### System Overview
154+
155+
```
156+
┌─────────────────────────────────────────────────────────────────────────┐
157+
│ DEVELOPER WORKFLOW │
158+
│ │
159+
│ ┌─────────────────────────────────────────────────────────────────┐ │
160+
│ │ WAZA CLI (Go) │ │
161+
│ │ │ │
162+
│ │ init → generate → dev → run → compare │ │
163+
│ └─────────────────────────────────────────────────────────────────┘ │
164+
│ │ │
165+
│ ┌─────────────────────┼─────────────────────┐ │
166+
│ │ │ │ │
167+
│ ▼ ▼ ▼ │
168+
│ ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ │
169+
│ │ Sensei Engine │ │ Eval Framework │ │ Waza Skill │ │
170+
│ │ (Compliance) │ │ (Testing/Metrics)│ │ (Guidance) │ │
171+
│ └──────────────────┘ └──────────────────┘ └──────────────────┘ │
172+
│ │
173+
└─────────────────────────────────────────────────────────────────────────┘
174+
```
175+
176+
### Component Breakdown
177+
178+
| Component | Language | Purpose |
179+
|-----------|----------|---------|
180+
| waza CLI | Go | Main binary, all commands |
181+
| Sensei Engine | Go | Compliance scoring, improvement loop |
182+
| Eval Framework | Go | Task execution, grading, metrics |
183+
| Copilot Executor | Go | Copilot SDK integration |
184+
| Waza Skill | Markdown | SKILL.md for conversational interface |
185+
186+
### Directory Structure
187+
188+
```
189+
waza-go/
190+
├── cmd/waza/ # CLI entrypoint
191+
├── internal/
192+
│ ├── config/ # Configuration loading
193+
│ ├── execution/ # Executors (mock, copilot)
194+
│ ├── models/ # Data models (spec, task, outcome)
195+
│ ├── orchestration/ # Runner, task coordination
196+
│ ├── scoring/ # Graders, validators
197+
│ └── sensei/ # Compliance engine (new)
198+
├── go.mod
199+
└── Makefile
200+
```
201+
202+
---
203+
204+
## Compliance Scoring System
205+
206+
Skills are scored on frontmatter compliance. Target: **Medium-High** or better for publishing.
207+
208+
| Score | Requirements | Description |
209+
|-------|--------------|-------------|
210+
| **Low** | Description < 150 chars OR no triggers | Basic, agent can't route reliably |
211+
| **Medium** | Description >= 150 chars AND has trigger keywords | Functional but may have false positives |
212+
| **Medium-High** | Has "USE FOR:" AND "DO NOT USE FOR:" | Clear boundaries, reliable routing |
213+
| **High** | Medium-High + INVOKES + FOR SINGLE OPERATIONS | Full routing clarity, MCP integration |
214+
215+
---
216+
217+
## Success Metrics
218+
219+
| Metric | Target | Measurement |
220+
|--------|--------|-------------|
221+
| Skill Compliance Rate | >80% Medium-High | Automated scoring via `waza dev` |
222+
| Trigger Accuracy | >90% | Evaluation framework pass rate |
223+
| Time to First Skill | <30 minutes | Developer surveys |
224+
| Cross-Model Consistency | >85% pass rate across 3+ models | Comparison reports |
225+
| Token Efficiency | <500 lines for SKILL.md | `waza tokens check` |
226+
227+
---
228+
229+
## Dependencies
230+
231+
| Dependency | Status | Risk Level |
232+
|------------|--------|------------|
233+
| Copilot SDK | Available | Medium - API stability |
234+
| Go Runtime | Stable | Low |
235+
| AZD Extension Framework | Available (alpha) | Medium - API changes |
236+
| microsoft/skills repo | Exists (132+ skills) | Low - Established |
237+
238+
---
239+
240+
## Phased Roadmap
241+
242+
### Primary Phase (Core Features)
243+
| Epic | Description | Priority |
244+
|------|-------------|----------|
245+
| E1: Go CLI Foundation | Port Python features to Go CLI | P0 |
246+
| E2: Sensei Engine | Compliance scoring & dev loop | P0 |
247+
| E3: Evaluation Framework | Cross-model testing & metrics | P0 |
248+
| E4: Token Management | Budget tracking & optimization | P1 |
249+
| E5: Waza Skill | Conversational skill for guidance | P1 |
250+
251+
### Secondary Phase (Integration & Extensions)
252+
| Epic | Description | Priority |
253+
|------|-------------|----------|
254+
| E6: CI/CD Integration | GitHub Actions & microsoft/skills | P1 |
255+
| E7: AZD Extension | Package as `azd extension` | P2 |
256+
257+
---
258+
259+
## Appendix: References
260+
261+
| Source | Description |
262+
|--------|-------------|
263+
| [microsoft/skills](https://github.com/microsoft/skills) | Target repository, contribution conventions |
264+
| [waza repo](https://github.com/spboyer/evals-for-skills) | Current implementation |
265+
| [Squad Proposal](https://github.com/spboyer/azure-mcp-v-skills/blob/main/squad-proposal.md) | Original proposal document |
266+
| [AZD Extension Framework](https://github.com/Azure/azure-dev/blob/main/cli/azd/docs/extensions/extension-framework.md) | Extension packaging guide |

0 commit comments

Comments
 (0)