Skip to content

Commit 66f2032

Browse files
nhortonclaude
nhorton
and
claude
committed
Create manual test files for rule styles (#61)
* Add manual test files for testing hook/rule functionality Creates manual_tests/claude/ directory with test files that exercise different rule styles: - Trigger/Safety mode (basic conditional) - Set mode (bidirectional correspondence) - Pair mode (directional correspondence) - Command action (automatic command execution) - Multi-safety (multiple safety patterns) Each test file includes documentation explaining what it tests, how to trigger it, and expected behavior. Corresponding rule definitions added to .deepwork/rules/. * Move manual test files from manual_tests/claude/ to manual_tests/ Flatten directory structure as requested. Updated all rule definitions to reference the new paths. * Reorganize manual tests into subfolders per test type Group related files together: - test_trigger_safety_mode/ - test_set_mode/ - test_pair_mode/ - test_command_action/ - test_multi_safety/ Updated rule definitions and README to match new structure. * Add compare_to: prompt to manual test rules This ensures rules evaluate against changes since the last prompt rather than against the merge-base, allowing them to fire during the current conversation when files are edited. * Add sub-agent testing instructions to manual tests README Explains that the best way to run these tests is as sub-agents using a fast model (haiku), with example prompts and verification commands. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Update manual test files with both-case test instructions - Updated README with test matrix showing expected results - Added TEST CASE sections to each test file documenting both "should fire" and "should NOT fire" scenarios - Added test results tracking table to README Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent cf756dd commit 66f2032

17 files changed

Lines changed: 487 additions & 0 deletions

‎.deepwork/rules/manual-test-command-action.md‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
---
2+
name: "Manual Test: Command Action"
3+
trigger: manual_tests/test_command_action/test_command_action.txt
4+
action:
5+
command: echo "$(date '+%Y-%m-%d %H:%M:%S') - Command triggered by edit to {file}" >> manual_tests/test_command_action/test_command_action_log.txt
6+
run_for: each_match
7+
compare_to: prompt
8+
---
9+
10+
# Manual Test: Command Action
11+
12+
This rule automatically appends a timestamped log entry when the
13+
test file is edited. No agent prompt is shown - the command runs
14+
automatically.
15+
16+
## This tests:
17+
18+
The command action feature where rules can execute shell commands
19+
instead of prompting the agent. The command should be idempotent.

‎.deepwork/rules/manual-test-multi-safety.md‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
---
2+
name: "Manual Test: Multi Safety"
3+
trigger: manual_tests/test_multi_safety/test_multi_safety.py
4+
safety:
5+
- manual_tests/test_multi_safety/test_multi_safety_changelog.md
6+
- manual_tests/test_multi_safety/test_multi_safety_version.txt
7+
compare_to: prompt
8+
---
9+
10+
# Manual Test: Multiple Safety Patterns
11+
12+
You changed the source file without updating version info!
13+
14+
**Changed:** `{trigger_files}`
15+
16+
## What to do:
17+
18+
1. Update the changelog: `manual_tests/test_multi_safety/test_multi_safety_changelog.md`
19+
2. And/or update the version: `manual_tests/test_multi_safety/test_multi_safety_version.txt`
20+
3. Or acknowledge with `<promise>Manual Test: Multi Safety</promise>`
21+
22+
## This tests:
23+
24+
Trigger/safety mode with MULTIPLE safety patterns. The rule is
25+
suppressed if ANY of the safety files are also edited.

‎.deepwork/rules/manual-test-pair-mode.md‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
---
2+
name: "Manual Test: Pair Mode"
3+
pair:
4+
trigger: manual_tests/test_pair_mode/test_pair_mode_trigger.py
5+
expects: manual_tests/test_pair_mode/test_pair_mode_expected.md
6+
compare_to: prompt
7+
---
8+
9+
# Manual Test: Pair Mode (Directional Correspondence)
10+
11+
API code changed without documentation update!
12+
13+
**Changed:** `{trigger_files}`
14+
**Expected:** `{expected_files}`
15+
16+
## What to do:
17+
18+
1. Update the API documentation in `test_pair_mode_expected.md`
19+
2. Or acknowledge with `<promise>Manual Test: Pair Mode</promise>`
20+
21+
## This tests:
22+
23+
The "pair" detection mode where there's a ONE-WAY relationship.
24+
When the trigger file changes, the expected file must also change.
25+
BUT the expected file can change independently (docs can be updated
26+
without requiring code changes).

‎.deepwork/rules/manual-test-set-mode.md‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
---
2+
name: "Manual Test: Set Mode"
3+
set:
4+
- manual_tests/test_set_mode/test_set_mode_source.py
5+
- manual_tests/test_set_mode/test_set_mode_test.py
6+
compare_to: prompt
7+
---
8+
9+
# Manual Test: Set Mode (Bidirectional Correspondence)
10+
11+
Source and test files must change together!
12+
13+
**Changed:** `{trigger_files}`
14+
**Missing:** `{expected_files}`
15+
16+
## What to do:
17+
18+
1. If you changed the source file, update the corresponding test file
19+
2. If you changed the test file, ensure the source file reflects those changes
20+
3. Or acknowledge with `<promise>Manual Test: Set Mode</promise>`
21+
22+
## This tests:
23+
24+
The "set" detection mode where files in a set must ALL change together.
25+
This is bidirectional - the rule fires regardless of which file in the set
26+
was edited first.

‎.deepwork/rules/manual-test-trigger-safety.md‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
---
2+
name: "Manual Test: Trigger Safety"
3+
trigger: manual_tests/test_trigger_safety_mode/test_trigger_safety_mode.py
4+
safety: manual_tests/test_trigger_safety_mode/test_trigger_safety_mode_doc.md
5+
compare_to: prompt
6+
---
7+
8+
# Manual Test: Trigger/Safety Mode
9+
10+
You edited `{trigger_files}` without updating the documentation.
11+
12+
## What to do:
13+
14+
1. Review the changes in the source file
15+
2. Update `manual_tests/test_trigger_safety_mode/test_trigger_safety_mode_doc.md` to reflect changes
16+
3. Or acknowledge this is intentional with `<promise>Manual Test: Trigger Safety</promise>`
17+
18+
## This tests:
19+
20+
The basic trigger/safety detection mode where editing the trigger file
21+
causes the rule to fire UNLESS the safety file is also edited.

‎manual_tests/README.md‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
# Manual Hook/Rule Tests for Claude
2+
3+
This directory contains files designed to manually test different types of deepwork rules/hooks.
4+
Each test must verify BOTH that the rule fires when it should AND does not fire when it shouldn't.
5+
6+
## How to Run These Tests
7+
8+
**The best way to run these tests is as sub-agents using a fast model (e.g., haiku).**
9+
10+
This approach works because:
11+
1. Sub-agents run in isolated contexts where changes can be detected
12+
2. The Stop hook evaluates rules when the sub-agent completes
13+
3. Using a fast model keeps test iterations quick and cheap
14+
15+
After each sub-agent returns, run the hook to verify:
16+
```bash
17+
echo '{}' | python -m deepwork.hooks.rules_check
18+
```
19+
20+
Then revert changes before the next test:
21+
```bash
22+
git checkout -- manual_tests/
23+
```
24+
25+
## Test Matrix
26+
27+
Each test has two cases: one where the rule SHOULD fire, and one where it should NOT.
28+
29+
| Test | Should Fire | Should NOT Fire | Rule Name |
30+
|------|-------------|-----------------|-----------|
31+
| **Trigger/Safety** | Edit `.py` only | Edit `.py` AND `_doc.md` | Manual Test: Trigger Safety |
32+
| **Set Mode** | Edit `_source.py` only | Edit `_source.py` AND `_test.py` | Manual Test: Set Mode |
33+
| **Pair Mode** | Edit `_trigger.py` only | Edit `_trigger.py` AND `_expected.md` | Manual Test: Pair Mode |
34+
| **Pair Mode (reverse)** | — | Edit `_expected.md` only (should NOT fire) | Manual Test: Pair Mode |
35+
| **Command Action** | Edit `.txt` → log appended | — (always runs) | Manual Test: Command Action |
36+
| **Multi Safety** | Edit `.py` only | Edit `.py` AND any safety file | Manual Test: Multi Safety |
37+
38+
## Test Results Tracking
39+
40+
| Test Case | Fires When Should | Does NOT Fire When Shouldn't |
41+
|-----------|:-----------------:|:----------------------------:|
42+
| Trigger/Safety | ☐ | ☐ |
43+
| Set Mode | ☐ | ☐ |
44+
| Pair Mode (forward) | ☐ | ☐ |
45+
| Pair Mode (reverse - expected only) | — | ☐ |
46+
| Command Action | ☐ | — |
47+
| Multi Safety | ☐ | ☐ |
48+
49+
## Test Folders
50+
51+
| Folder | Rule Type | Description |
52+
|--------|-----------|-------------|
53+
| `test_trigger_safety_mode/` | Trigger/Safety | Basic conditional: fires unless safety file also edited |
54+
| `test_set_mode/` | Set (Bidirectional) | Files must change together (either direction) |
55+
| `test_pair_mode/` | Pair (Directional) | One-way: trigger requires expected, but not vice versa |
56+
| `test_command_action/` | Command Action | Automatically runs command on file change |
57+
| `test_multi_safety/` | Multiple Safety | Fires unless ANY of the safety files also edited |
58+
59+
## Corresponding Rules
60+
61+
Rules are defined in `.deepwork/rules/`:
62+
- `manual-test-trigger-safety.md`
63+
- `manual-test-set-mode.md`
64+
- `manual-test-pair-mode.md`
65+
- `manual-test-command-action.md`
66+
- `manual-test-multi-safety.md`

‎manual_tests/test_command_action/test_command_action.txt‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
MANUAL TEST: Command Action Rule
2+
3+
=== WHAT THIS TESTS ===
4+
Tests the "command action" feature where a rule automatically
5+
runs a shell command instead of prompting the agent.
6+
7+
=== HOW TO TRIGGER ===
8+
Edit this file (add text, modify content, etc.)
9+
10+
=== EXPECTED BEHAVIOR ===
11+
When this file is edited, the rule automatically runs a command
12+
that appends a timestamped line to test_command_action_log.txt
13+
14+
The command is idempotent: running it multiple times produces
15+
consistent results (a log entry is appended).
16+
17+
=== RULE LOCATION ===
18+
.deepwork/rules/manual-test-command-action.md
19+
20+
=== LOG FILE ===
21+
Check test_command_action_log.txt for command execution results.
22+
23+
---
24+
Edit below this line to trigger the command:
25+
---

‎manual_tests/test_command_action/test_command_action_log.txt‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# Command Action Log
2+
# Lines below are added automatically when test_command_action.txt is edited
3+
# ---

‎manual_tests/test_multi_safety/test_multi_safety.py‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
"""
2+
MANUAL TEST: Multiple Safety Patterns
3+
4+
=== WHAT THIS TESTS ===
5+
Tests trigger/safety mode with MULTIPLE safety patterns:
6+
- Rule fires when this file is edited alone
7+
- Rule is suppressed if ANY of the safety files are also edited:
8+
- test_multi_safety_changelog.md
9+
- test_multi_safety_version.txt
10+
11+
=== TEST CASE 1: Rule SHOULD fire ===
12+
1. Edit this file (add a comment below the marker)
13+
2. Do NOT edit any safety files
14+
3. Run: echo '{}' | python -m deepwork.hooks.rules_check
15+
4. Expected: "Manual Test: Multi Safety" appears in output
16+
17+
=== TEST CASE 2: Rule should NOT fire (changelog edited) ===
18+
1. Edit this file (add a comment below the marker)
19+
2. ALSO edit test_multi_safety_changelog.md
20+
3. Run: echo '{}' | python -m deepwork.hooks.rules_check
21+
4. Expected: "Manual Test: Multi Safety" does NOT appear
22+
23+
=== TEST CASE 3: Rule should NOT fire (version edited) ===
24+
1. Edit this file (add a comment below the marker)
25+
2. ALSO edit test_multi_safety_version.txt
26+
3. Run: echo '{}' | python -m deepwork.hooks.rules_check
27+
4. Expected: "Manual Test: Multi Safety" does NOT appear
28+
29+
=== RULE LOCATION ===
30+
.deepwork/rules/manual-test-multi-safety.md
31+
"""
32+
33+
34+
VERSION = "1.0.0"
35+
36+
37+
def get_version():
38+
"""Return the current version."""
39+
return VERSION
40+
41+
42+
# Edit below this line to trigger the rule
43+
# -------------------------------------------

‎manual_tests/test_multi_safety/test_multi_safety_changelog.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# Changelog (Multi-Safety Test)
2+
3+
## What This File Does
4+
5+
This is one of the "safety" files for the multi-safety test.
6+
Editing this file suppresses the rule when the source is edited.
7+
8+
## Changelog
9+
10+
### v1.0.0
11+
- Initial release
12+
13+
---
14+
15+
Edit below this line to suppress the multi-safety rule:
16+
<!-- Changes here -->

0 commit comments

Comments
 (0)