Skip to content

Commit 4f6e887

Browse files
authored
Add grader showcase examples demonstrating all grader types (microsoft#134)
Adds grader showcase examples with all 5 grader types, fixtures, and comprehensive README. Closes microsoft#108.
1 parent daacb7d commit 4f6e887

10 files changed

Lines changed: 891 additions & 0 deletions

File tree

examples/README.md

Lines changed: 154 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,154 @@
1+
# Waza Examples
2+
3+
This directory contains example evaluation suites demonstrating various features and use cases of waza.
4+
5+
## Available Examples
6+
7+
### 1. [code-explainer](./code-explainer/)
8+
**Purpose**: Complete eval suite for a skill that explains code snippets
9+
10+
**Demonstrates**:
11+
- Multi-trial testing across different programming languages
12+
- Custom script graders for complex evaluation logic
13+
- Task organization with YAML files
14+
- Metrics definition and weighting
15+
16+
**Quick Start**:
17+
```bash
18+
waza run examples/code-explainer/eval.yaml --context-dir examples/code-explainer/fixtures -v
19+
```
20+
21+
---
22+
23+
### 2. [grader-showcase](./grader-showcase/)
24+
**Purpose**: Comprehensive demonstration of all grader types
25+
26+
**Demonstrates**:
27+
- `code` grader: Python assertion-based validation
28+
- `regex` grader: Pattern matching in output
29+
- `file` grader: File existence and content validation
30+
- `behavior` grader: Agent behavior constraints
31+
- `action_sequence` grader: Tool call sequence validation
32+
33+
**Quick Start**:
34+
```bash
35+
waza run examples/grader-showcase/eval.yaml --context-dir examples/grader-showcase/fixtures -v
36+
```
37+
38+
**What's Inside**:
39+
- 5 task examples, one for each grader type
40+
- Detailed README with configuration options
41+
- Runnable fixtures and test data
42+
- Examples of global vs. task-specific graders
43+
44+
---
45+
46+
### 3. [ci](./ci/)
47+
**Purpose**: GitHub Actions workflow examples for CI/CD integration
48+
49+
**Demonstrates**:
50+
- Basic waza evaluation in CI pipeline
51+
- Matrix testing across multiple models
52+
- Result comparison and reporting
53+
- Reusable workflow patterns
54+
55+
**Quick Start**:
56+
See [ci/README.md](./ci/README.md) for integration instructions.
57+
58+
---
59+
60+
## Usage Patterns
61+
62+
### Running Examples
63+
64+
Run an entire eval suite:
65+
```bash
66+
waza run examples/<example-name>/eval.yaml -v
67+
```
68+
69+
Run with context directory (for file-based tasks):
70+
```bash
71+
waza run examples/<example-name>/eval.yaml --context-dir examples/<example-name>/fixtures -v
72+
```
73+
74+
Run specific tasks only:
75+
```bash
76+
waza run examples/<example-name>/eval.yaml --filter="task-name-pattern" -v
77+
```
78+
79+
Save results to JSON:
80+
```bash
81+
waza run examples/<example-name>/eval.yaml -o results.json
82+
```
83+
84+
### Using Examples as Templates
85+
86+
1. **Copy an example**:
87+
```bash
88+
cp -r examples/grader-showcase my-eval
89+
cd my-eval
90+
```
91+
92+
2. **Modify for your use case**:
93+
- Update `eval.yaml` with your skill name and config
94+
- Edit or create task files in `tasks/`
95+
- Add fixture files to `fixtures/`
96+
- Adjust graders to match your validation needs
97+
98+
3. **Run your eval**:
99+
```bash
100+
waza run eval.yaml -v
101+
```
102+
103+
## Example Comparison
104+
105+
| Example | Best For | Complexity | Grader Types |
106+
|---------|----------|------------|--------------|
107+
| **code-explainer** | Complete real-world eval | Medium | code, regex, script |
108+
| **grader-showcase** | Learning grader types | Low | All types |
109+
| **ci** | GitHub Actions integration | Low | N/A (workflow examples) |
110+
111+
## Learning Path
112+
113+
1. **Start with grader-showcase** to understand grader types
114+
2. **Study code-explainer** for realistic eval structure
115+
3. **Reference ci** for production integration
116+
117+
## File Structure Convention
118+
119+
Most examples follow this structure:
120+
```
121+
example-name/
122+
├── eval.yaml # Main benchmark spec
123+
├── README.md # Documentation
124+
├── fixtures/ # Context files for tasks
125+
│ ├── file1.ext
126+
│ └── file2.ext
127+
├── tasks/ # Individual task definitions
128+
│ ├── task-1.yaml
129+
│ └── task-2.yaml
130+
└── graders/ # Optional: custom script graders
131+
└── custom_grader.py
132+
```
133+
134+
## Related Documentation
135+
136+
- **Grader Reference**: [docs/GRADERS.md](../docs/GRADERS.md)
137+
- **Main README**: [README.md](../README.md)
138+
- **Implementation Details**: [IMPLEMENTATION.md](../IMPLEMENTATION.md)
139+
140+
## Contributing Examples
141+
142+
To add a new example:
143+
144+
1. Create a directory: `examples/your-example/`
145+
2. Add required files: `eval.yaml`, `README.md`, `tasks/`, `fixtures/`
146+
3. Document in this README
147+
4. Test with `waza run examples/your-example/eval.yaml -v`
148+
5. Submit a PR
149+
150+
## Support
151+
152+
- **Issues**: Open an issue on GitHub
153+
- **Discussions**: Check existing examples for patterns
154+
- **Documentation**: See linked docs above

0 commit comments

Comments
 (0)