Skip to content

Commit 8a8b734

Browse files
danielholandadanielholanda
committed
Documentation
1 parent 4b46d79 commit 8a8b734

2 files changed

Lines changed: 140 additions & 135 deletions

File tree

‎playbooks/README.md‎

Lines changed: 0 additions & 135 deletions
Original file line numberDiff line numberDiff line change
@@ -116,141 +116,6 @@ Content displays directly since these are required steps, not optional reference
116116
- Include expected output so users know what success looks like
117117
- Keep code blocks copy-friendly (avoid `$` or `>` prompts)
118118

119-
### Testing Tags
120-
121-
Use `@test` tags to make existing code blocks testable. These tags **wrap your existing code** and are picked up by CI to run automated tests. The HTML comment tags themselves are invisible to website visitors, but the wrapped code remains visible by default. Add `hidden=true` to hide the code block from the website if needed (e.g., for test-only setup commands).
122-
123-
**Basic syntax — wrap existing code blocks:**
124-
125-
```markdown
126-
<!-- @test:id=install-deps timeout=300 -->
127-
```bash
128-
pip install transformers accelerate
129-
```
130-
<!-- @test:end -->
131-
```
132-
133-
The test tags wrap the code block that users see. No duplication needed — the same code that appears in the playbook is what gets tested.
134-
135-
**Test attributes:**
136-
137-
| Attribute | Required | Default | Description |
138-
|-----------|----------|---------|-------------|
139-
| `id` | Yes | — | Unique identifier for the test (use kebab-case) |
140-
| `timeout` | No | `300` | Maximum execution time in seconds |
141-
| `continue_on_error` | No | `false` | If `true`, test failure won't fail the CI job |
142-
| `hidden` | No | `false` | If `true`, hides the code block from the website (useful for test-only setup) |
143-
144-
> **Note:** Platform is automatically inferred from the surrounding `@os:` tags. Tests inside `<!-- @os:windows -->` run only on Windows, tests inside `<!-- @os:linux -->` run only on Linux, and tests outside any `@os:` block run on all platforms.
145-
146-
**Supported languages:**
147-
148-
| Language tag | Execution |
149-
|--------------|-----------|
150-
| `bash`, `sh`, `shell` | PowerShell on Windows, Bash on Linux |
151-
| `cmd`, `batch` | Windows CMD |
152-
| `powershell`, `pwsh`, `ps1` | PowerShell |
153-
| `python` | Python interpreter |
154-
155-
**Example: Ordering tests by README position**
156-
157-
Tests run in the order they appear in the README. Place prerequisite steps before the tests that need them:
158-
159-
```markdown
160-
### Create Virtual Environment
161-
162-
<!-- @os:windows -->
163-
<!-- @test:id=create-venv timeout=60 -->
164-
```cmd
165-
python -m venv myenv
166-
myenv\Scripts\activate.bat
167-
```
168-
<!-- @test:end -->
169-
<!-- @os:end -->
170-
171-
### Install Dependencies
172-
173-
<!-- @test:id=install-deps timeout=300 -->
174-
```bash
175-
pip install transformers torch
176-
```
177-
<!-- @test:end -->
178-
179-
### Run the Script
180-
181-
<!-- @test:id=run-script timeout=60 -->
182-
```bash
183-
python run_model.py --help
184-
```
185-
<!-- @test:end -->
186-
```
187-
188-
In this example, `create-venv` runs first, then `install-deps`, then `run-script` — matching the natural reading order of the playbook.
189-
190-
**Example: Platform-specific tests**
191-
192-
Combine with `@os` tags for platform-specific instructions:
193-
194-
```markdown
195-
<!-- @os:windows -->
196-
<!-- @test:id=setup timeout=120 -->
197-
```cmd
198-
pip install torch
199-
python -c "import torch; print('OK')"
200-
```
201-
<!-- @test:end -->
202-
<!-- @os:end -->
203-
204-
<!-- @os:linux -->
205-
<!-- @test:id=setup timeout=120 -->
206-
```bash
207-
pip3 install torch
208-
python3 -c "import torch; print('OK')"
209-
```
210-
<!-- @test:end -->
211-
<!-- @os:end -->
212-
```
213-
214-
**Example: Verification tests (not shown to users)**
215-
216-
For tests that verify things but shouldn't appear in the playbook, place them after visible content:
217-
218-
```markdown
219-
| [run_llm.py](assets/run_llm.py) | Basic LLM script |
220-
221-
<!-- @test:id=verify-scripts timeout=30 -->
222-
```python
223-
import os, sys
224-
missing = [f for f in ['run_llm.py'] if not os.path.exists(f)]
225-
if missing:
226-
sys.exit(1)
227-
```
228-
<!-- @test:end -->
229-
230-
Both scripts support...
231-
```
232-
233-
**Best practices:**
234-
235-
1. **Wrap, don't duplicate**: Put test tags around existing code blocks instead of writing separate test code
236-
2. **Order matters**: Tests run in README order, so place prerequisites before the steps that need them
237-
3. **Keep tests fast**: Use appropriate timeouts; most tests should complete in under 60 seconds
238-
4. **Handle missing hardware**: CI machines may not have GPUs; tests should handle this gracefully
239-
5. **Test the happy path**: Focus on verifying instructions work, not edge cases
240-
241-
**Running tests locally:**
242-
243-
```bash
244-
python .github/scripts/run_playbook_tests.py --playbook your-playbook-id --platform windows
245-
```
246-
247-
**CI behavior:**
248-
249-
- Tests run automatically on PRs that modify playbook files
250-
- Tests run on self-hosted runners tagged with `Windows` and `halo`
251-
- Tests run in the order they appear in the README
252-
- Test results are uploaded as artifacts for debugging
253-
254119
---
255120

256121
## The `platform.md` File

‎playbooks/TESTING.md‎

Lines changed: 140 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,140 @@
1+
# Playbook Testing Guide
2+
3+
> **Note:** The testing infrastructure is still being developed. Playbook creators are not expected to add tests to their playbooks yet.
4+
5+
---
6+
7+
## Testing Tags
8+
9+
Use `@test` tags to make existing code blocks testable. These tags **wrap your existing code** and are picked up by CI to run automated tests. The HTML comment tags themselves are invisible to website visitors, but the wrapped code remains visible by default. Add `hidden=true` to hide the code block from the website if needed (e.g., for test-only setup commands).
10+
11+
**Basic syntax — wrap existing code blocks:**
12+
13+
```markdown
14+
<!-- @test:id=install-deps timeout=300 -->
15+
```bash
16+
pip install transformers accelerate
17+
```
18+
<!-- @test:end -->
19+
```
20+
21+
The test tags wrap the code block that users see. No duplication needed — the same code that appears in the playbook is what gets tested.
22+
23+
**Test attributes:**
24+
25+
| Attribute | Required | Default | Description |
26+
|-----------|----------|---------|-------------|
27+
| `id` | Yes | — | Unique identifier for the test (use kebab-case) |
28+
| `timeout` | No | `300` | Maximum execution time in seconds |
29+
| `continue_on_error` | No | `false` | If `true`, test failure won't fail the CI job |
30+
| `hidden` | No | `false` | If `true`, hides the code block from the website (useful for test-only setup) |
31+
32+
> **Note:** Platform is automatically inferred from the surrounding `@os:` tags. Tests inside `<!-- @os:windows -->` run only on Windows, tests inside `<!-- @os:linux -->` run only on Linux, and tests outside any `@os:` block run on all platforms.
33+
34+
**Supported languages:**
35+
36+
| Language tag | Execution |
37+
|--------------|-----------|
38+
| `bash`, `sh`, `shell` | PowerShell on Windows, Bash on Linux |
39+
| `cmd`, `batch` | Windows CMD |
40+
| `powershell`, `pwsh`, `ps1` | PowerShell |
41+
| `python` | Python interpreter |
42+
43+
**Example: Ordering tests by README position**
44+
45+
Tests run in the order they appear in the README. Place prerequisite steps before the tests that need them:
46+
47+
```markdown
48+
### Create Virtual Environment
49+
50+
<!-- @os:windows -->
51+
<!-- @test:id=create-venv timeout=60 -->
52+
```cmd
53+
python -m venv myenv
54+
myenv\Scripts\activate.bat
55+
```
56+
<!-- @test:end -->
57+
<!-- @os:end -->
58+
59+
### Install Dependencies
60+
61+
<!-- @test:id=install-deps timeout=300 -->
62+
```bash
63+
pip install transformers torch
64+
```
65+
<!-- @test:end -->
66+
67+
### Run the Script
68+
69+
<!-- @test:id=run-script timeout=60 -->
70+
```bash
71+
python run_model.py --help
72+
```
73+
<!-- @test:end -->
74+
```
75+
76+
In this example, `create-venv` runs first, then `install-deps`, then `run-script` — matching the natural reading order of the playbook.
77+
78+
**Example: Platform-specific tests**
79+
80+
Combine with `@os` tags for platform-specific instructions:
81+
82+
```markdown
83+
<!-- @os:windows -->
84+
<!-- @test:id=setup timeout=120 -->
85+
```cmd
86+
pip install torch
87+
python -c "import torch; print('OK')"
88+
```
89+
<!-- @test:end -->
90+
<!-- @os:end -->
91+
92+
<!-- @os:linux -->
93+
<!-- @test:id=setup timeout=120 -->
94+
```bash
95+
pip3 install torch
96+
python3 -c "import torch; print('OK')"
97+
```
98+
<!-- @test:end -->
99+
<!-- @os:end -->
100+
```
101+
102+
**Example: Verification tests (not shown to users)**
103+
104+
For tests that verify things but shouldn't appear in the playbook, place them after visible content:
105+
106+
```markdown
107+
| [run_llm.py](assets/run_llm.py) | Basic LLM script |
108+
109+
<!-- @test:id=verify-scripts timeout=30 -->
110+
```python
111+
import os, sys
112+
missing = [f for f in ['run_llm.py'] if not os.path.exists(f)]
113+
if missing:
114+
sys.exit(1)
115+
```
116+
<!-- @test:end -->
117+
118+
Both scripts support...
119+
```
120+
121+
**Best practices:**
122+
123+
1. **Wrap, don't duplicate**: Put test tags around existing code blocks instead of writing separate test code
124+
2. **Order matters**: Tests run in README order, so place prerequisites before the steps that need them
125+
3. **Keep tests fast**: Use appropriate timeouts; most tests should complete in under 60 seconds
126+
4. **Handle missing hardware**: CI machines may not have GPUs; tests should handle this gracefully
127+
5. **Test the happy path**: Focus on verifying instructions work, not edge cases
128+
129+
**Running tests locally:**
130+
131+
```bash
132+
python .github/scripts/run_playbook_tests.py --playbook your-playbook-id --platform windows
133+
```
134+
135+
**CI behavior:**
136+
137+
- Tests run automatically on PRs that modify playbook files
138+
- Tests run on self-hosted runners tagged with `Windows` and `halo`
139+
- Tests run in the order they appear in the README
140+
- Test results are uploaded as artifacts for debugging

0 commit comments

Comments
 (0)