|
| 1 | +# Waza Examples |
| 2 | + |
| 3 | +This directory contains example evaluation suites demonstrating various features and use cases of waza. |
| 4 | + |
| 5 | +## Available Examples |
| 6 | + |
| 7 | +### 1. [code-explainer](./code-explainer/) |
| 8 | +**Purpose**: Complete eval suite for a skill that explains code snippets |
| 9 | + |
| 10 | +**Demonstrates**: |
| 11 | +- Multi-trial testing across different programming languages |
| 12 | +- Custom script graders for complex evaluation logic |
| 13 | +- Task organization with YAML files |
| 14 | +- Metrics definition and weighting |
| 15 | + |
| 16 | +**Quick Start**: |
| 17 | +```bash |
| 18 | +waza run examples/code-explainer/eval.yaml --context-dir examples/code-explainer/fixtures -v |
| 19 | +``` |
| 20 | + |
| 21 | +--- |
| 22 | + |
| 23 | +### 2. [grader-showcase](./grader-showcase/) |
| 24 | +**Purpose**: Comprehensive demonstration of all grader types |
| 25 | + |
| 26 | +**Demonstrates**: |
| 27 | +- `code` grader: Python assertion-based validation |
| 28 | +- `regex` grader: Pattern matching in output |
| 29 | +- `file` grader: File existence and content validation |
| 30 | +- `behavior` grader: Agent behavior constraints |
| 31 | +- `action_sequence` grader: Tool call sequence validation |
| 32 | + |
| 33 | +**Quick Start**: |
| 34 | +```bash |
| 35 | +waza run examples/grader-showcase/eval.yaml --context-dir examples/grader-showcase/fixtures -v |
| 36 | +``` |
| 37 | + |
| 38 | +**What's Inside**: |
| 39 | +- 5 task examples, one for each grader type |
| 40 | +- Detailed README with configuration options |
| 41 | +- Runnable fixtures and test data |
| 42 | +- Examples of global vs. task-specific graders |
| 43 | + |
| 44 | +--- |
| 45 | + |
| 46 | +### 3. [ci](./ci/) |
| 47 | +**Purpose**: GitHub Actions workflow examples for CI/CD integration |
| 48 | + |
| 49 | +**Demonstrates**: |
| 50 | +- Basic waza evaluation in CI pipeline |
| 51 | +- Matrix testing across multiple models |
| 52 | +- Result comparison and reporting |
| 53 | +- Reusable workflow patterns |
| 54 | + |
| 55 | +**Quick Start**: |
| 56 | +See [ci/README.md](./ci/README.md) for integration instructions. |
| 57 | + |
| 58 | +--- |
| 59 | + |
| 60 | +## Usage Patterns |
| 61 | + |
| 62 | +### Running Examples |
| 63 | + |
| 64 | +Run an entire eval suite: |
| 65 | +```bash |
| 66 | +waza run examples/<example-name>/eval.yaml -v |
| 67 | +``` |
| 68 | + |
| 69 | +Run with context directory (for file-based tasks): |
| 70 | +```bash |
| 71 | +waza run examples/<example-name>/eval.yaml --context-dir examples/<example-name>/fixtures -v |
| 72 | +``` |
| 73 | + |
| 74 | +Run specific tasks only: |
| 75 | +```bash |
| 76 | +waza run examples/<example-name>/eval.yaml --filter="task-name-pattern" -v |
| 77 | +``` |
| 78 | + |
| 79 | +Save results to JSON: |
| 80 | +```bash |
| 81 | +waza run examples/<example-name>/eval.yaml -o results.json |
| 82 | +``` |
| 83 | + |
| 84 | +### Using Examples as Templates |
| 85 | + |
| 86 | +1. **Copy an example**: |
| 87 | + ```bash |
| 88 | + cp -r examples/grader-showcase my-eval |
| 89 | + cd my-eval |
| 90 | + ``` |
| 91 | + |
| 92 | +2. **Modify for your use case**: |
| 93 | + - Update `eval.yaml` with your skill name and config |
| 94 | + - Edit or create task files in `tasks/` |
| 95 | + - Add fixture files to `fixtures/` |
| 96 | + - Adjust graders to match your validation needs |
| 97 | + |
| 98 | +3. **Run your eval**: |
| 99 | + ```bash |
| 100 | + waza run eval.yaml -v |
| 101 | + ``` |
| 102 | + |
| 103 | +## Example Comparison |
| 104 | + |
| 105 | +| Example | Best For | Complexity | Grader Types | |
| 106 | +|---------|----------|------------|--------------| |
| 107 | +| **code-explainer** | Complete real-world eval | Medium | code, regex, script | |
| 108 | +| **grader-showcase** | Learning grader types | Low | All types | |
| 109 | +| **ci** | GitHub Actions integration | Low | N/A (workflow examples) | |
| 110 | + |
| 111 | +## Learning Path |
| 112 | + |
| 113 | +1. **Start with grader-showcase** to understand grader types |
| 114 | +2. **Study code-explainer** for realistic eval structure |
| 115 | +3. **Reference ci** for production integration |
| 116 | + |
| 117 | +## File Structure Convention |
| 118 | + |
| 119 | +Most examples follow this structure: |
| 120 | +``` |
| 121 | +example-name/ |
| 122 | +├── eval.yaml # Main benchmark spec |
| 123 | +├── README.md # Documentation |
| 124 | +├── fixtures/ # Context files for tasks |
| 125 | +│ ├── file1.ext |
| 126 | +│ └── file2.ext |
| 127 | +├── tasks/ # Individual task definitions |
| 128 | +│ ├── task-1.yaml |
| 129 | +│ └── task-2.yaml |
| 130 | +└── graders/ # Optional: custom script graders |
| 131 | + └── custom_grader.py |
| 132 | +``` |
| 133 | + |
| 134 | +## Related Documentation |
| 135 | + |
| 136 | +- **Grader Reference**: [docs/GRADERS.md](../docs/GRADERS.md) |
| 137 | +- **Main README**: [README.md](../README.md) |
| 138 | +- **Implementation Details**: [IMPLEMENTATION.md](../IMPLEMENTATION.md) |
| 139 | + |
| 140 | +## Contributing Examples |
| 141 | + |
| 142 | +To add a new example: |
| 143 | + |
| 144 | +1. Create a directory: `examples/your-example/` |
| 145 | +2. Add required files: `eval.yaml`, `README.md`, `tasks/`, `fixtures/` |
| 146 | +3. Document in this README |
| 147 | +4. Test with `waza run examples/your-example/eval.yaml -v` |
| 148 | +5. Submit a PR |
| 149 | + |
| 150 | +## Support |
| 151 | + |
| 152 | +- **Issues**: Open an issue on GitHub |
| 153 | +- **Discussions**: Check existing examples for patterns |
| 154 | +- **Documentation**: See linked docs above |
0 commit comments