Skip to content

Commit 2d3d42b

Browse files
committed
Automate interactive tests (#2)
1 parent 0591254 commit 2d3d42b

60 files changed

Lines changed: 3623 additions & 263 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.claude/settings.local.json

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,9 @@
77
"Bash(Select-Object -First 50)",
88
"Bash(cat:*)",
99
"Bash(git log:*)",
10-
"Bash(find:*)"
10+
"Bash(find:*)",
11+
"Skill(implement-test-handler)",
12+
"Bash(grep:*)"
1113
],
1214
"deny": [],
1315
"ask": []
Lines changed: 167 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,167 @@
1+
---
2+
name: implement-test-handler
3+
description: Implements Z-Wave CTT (Certification Test Tool) test handlers for automating certification tests. Use when asked to create handlers for a given test, which includes automating CTT log parsing and prompt responses.
4+
---
5+
6+
# CTT Test Handler Implementation
7+
8+
## Overview
9+
10+
CTT handlers automate Z-Wave certification tests by:
11+
1. Parsing CTT log messages and executing corresponding Z-Wave commands
12+
2. Responding to CTT prompts with Yes/No/Ok based on device state
13+
14+
## File Structure
15+
16+
```
17+
dut/zwave-js/handlers/
18+
├── tests/ # Test-specific handlers (CCR_*, CDR_*, RT_*, ...)
19+
├── behaviors/ # Reusable handlers (capabilities, addMode, etc.)
20+
├── utils.ts # Utility functions
21+
└── index.ts # Auto-imports all handlers
22+
```
23+
24+
## Handler Template
25+
26+
```typescript
27+
import { <CCValues>, <CCEnums> } from "zwave-js";
28+
import { registerHandler } from "../../prompt-handlers.ts";
29+
30+
// Map CTT names to zwave-js enum values
31+
const nameToEnum: Record<string, EnumType> = {
32+
CTTName: EnumType["ZWave JS Name"],
33+
};
34+
35+
registerHandler("CCR_<TestName>_Rev##", {
36+
async onLog(ctx) {
37+
const node = ctx.includedNodes.at(-1);
38+
if (!node) return;
39+
40+
// Parse log and execute command
41+
const match = /<PATTERN>/i.exec(ctx.logText);
42+
if (match?.groups) {
43+
// Execute Z-Wave command
44+
return true; // Mark as handled
45+
}
46+
},
47+
48+
async onPrompt(ctx) {
49+
const node = ctx.includedNodes.at(-1);
50+
if (!node) return;
51+
52+
// Check condition and respond (prefer regex over .includes())
53+
if (/some pattern/i.test(ctx.promptText)) {
54+
const actual = node.getValue(CCValues.property.id);
55+
return actual === expected ? "Yes" : "No";
56+
}
57+
},
58+
});
59+
```
60+
61+
## Key Patterns
62+
63+
### Executing Commands
64+
65+
Prefer `setValue` for simple single-value commands. Use `commandClasses` methods for complex operations with multiple parameters.
66+
67+
**Simple value set (preferred):**
68+
```typescript
69+
node.setValue(DoorLockCCValues.targetMode.id, DoorLockMode.Secured);
70+
```
71+
72+
**Command class method (for complex operations):**
73+
```typescript
74+
await node.commandClasses["Thermostat Setpoint"].set(type, value, scale);
75+
await node.commandClasses["Door Lock"].setConfiguration(config);
76+
```
77+
78+
### Reading Values
79+
80+
```typescript
81+
const current = node.getValue(ThermostatModeCCValues.thermostatMode.id);
82+
```
83+
84+
### State Management
85+
86+
Store values for later verification:
87+
```typescript
88+
// In onLog:
89+
ctx.state.set("lastValue", value);
90+
91+
// In onPrompt:
92+
const stored = ctx.state.get("lastValue") as number;
93+
```
94+
95+
### Enum Mapping Conventions
96+
97+
CTT typically uses Z-Wave JS enum names without spaces:
98+
- `FullPower``ThermostatSetpointType["Full Power"]`
99+
- `AutoChangeover``ThermostatSetpointType["Auto Changeover"]`
100+
101+
For UPPER_CASE log formats (like `THERMOSTAT_MODE_SET`):
102+
- `HEAT``ThermostatMode["Heat"]`
103+
- `MANUFACTURER_SPECIFIC``ThermostatMode["Manufacturer specific"]`
104+
105+
### Pattern Matching
106+
107+
Prefer regex over `.includes()` due to case variations and potential typos in CTT output:
108+
109+
```typescript
110+
// Good: case-insensitive, handles typos
111+
if (/setpoint.+set succ?essfully/i.test(ctx.promptText)) { ... }
112+
113+
// Avoid: brittle, case-sensitive
114+
if (ctx.promptText.includes("Setpoint been set successfully")) { ... }
115+
```
116+
117+
## Common Log Patterns
118+
119+
```typescript
120+
// Value with mode/type
121+
/COMMAND_SET to mode = '(?<mode>\w+)'/i
122+
123+
// Value with numeric parameter
124+
/COMMAND_SET.+value=(?<value>[\d.]+)/i
125+
126+
// Hex value in parentheses
127+
/'(?<name>[^']+)' \((?<hex>0x[0-9a-fA-F]+)\)/i
128+
129+
// Multi-line configuration
130+
if (/Set Configuration:/i.test(ctx.logText)) {
131+
const field1 = /Field 1:\s+'(\w+)'/i.exec(ctx.logText)?.[1];
132+
}
133+
```
134+
135+
## Common Prompt Patterns
136+
137+
```typescript
138+
// Confirmation with expected value
139+
/current mode is set to '(?<mode>\w+)'/i
140+
/last known .+ is '(?<value>[^']+)' \((?<hex>0x[0-9a-fA-F]+)\)/i
141+
142+
// Simple yes/no capability questions → add to behaviors/capabilities.ts
143+
{ pattern: /capable to do something/i, answer: "Yes" }
144+
```
145+
146+
## Reference Files
147+
148+
- [prompt-handlers.ts](../../../dut/zwave-js/prompt-handlers.ts) - Handler registration
149+
- [utils.ts](../../../dut/zwave-js/handlers/utils.ts) - Utilities like `parseDurationFromLog`
150+
- [capabilities.ts](../../../dut/zwave-js/handlers/behaviors/capabilities.ts) - Capability prompts
151+
- [zwave-js CC Documentation](https://zwave-js.github.io/zwave-js/#/api/CCs/index) - Official CC API docs
152+
153+
## Existing Handler Examples
154+
155+
- [CCR_DoorLockCC_Rev02.ts](../../../dut/zwave-js/handlers/tests/CCR_DoorLockCC_Rev02.ts) - Complex config
156+
- [CCR_ThermostatModeCC_Rev02.ts](../../../dut/zwave-js/handlers/tests/CCR_ThermostatModeCC_Rev02.ts) - Mode with manufacturer data
157+
- [CCR_ThermostatSetpointCC_Rev03.ts](../../../dut/zwave-js/handlers/tests/CCR_ThermostatSetpointCC_Rev03.ts) - State verification
158+
- [CCR_WindowCoveringCC_Rev02.ts](../../../dut/zwave-js/handlers/tests/CCR_WindowCoveringCC_Rev02.ts) - Level change with setTimeout
159+
160+
## Workflow
161+
162+
1. **Identify the test name** (e.g., `CCR_DoorLockCC_Rev02`)
163+
2. **Get log/prompt patterns** from user
164+
3. **Find the CC API** at https://zwave-js.github.io/zwave-js/#/api/CCs/index - only if this leaves questions open, look in `node_modules/@zwave-js/cc/build/esm/cc/<CC>CC.d.ts`
165+
4. **Find enum values** in `node_modules/@zwave-js/cc/build/esm/lib/_Types.d.ts`
166+
5. **Create handler** in `dut/zwave-js/handlers/tests/CCR_<Name>.ts`
167+
6. **Add capability questions** to `behaviors/capabilities.ts` if needed

.github/workflows/run-zwave-wsl.yml

Lines changed: 26 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,12 @@ jobs:
4242
working-directory: dut/zwave-js
4343

4444
## Set up CTT
45+
- name: Download CTT setup archive
46+
shell: powershell
47+
env:
48+
GH_TOKEN: ${{ secrets.CTT_ARCHIVE_TOKEN }}
49+
run: .\setup\download-ctt-archive.ps1
50+
4551
- name: Extract CTT setup files
4652
shell: powershell
4753
run: .\setup\unpack-ctt-archive.ps1
@@ -62,34 +68,31 @@ jobs:
6268
shell: powershell
6369
run: .\setup\unpack-network-state-archive.ps1
6470

65-
- name: Copy Z-Wave storage to WSL native filesystem
71+
- name: Copy Z-Wave stack to WSL native filesystem
6672
shell: wsl-bash {0}
6773
run: |
68-
# Create storage in WSL native filesystem
69-
rm -rf ~/zwave_storage
70-
mkdir -p ~/zwave_storage
71-
72-
# Copy extracted storage files to WSL native filesystem
73-
if [ -d "./zwave_stack/storage" ]; then
74-
cp -r ./zwave_stack/storage/* ~/zwave_storage/ 2>/dev/null || true
75-
fi
76-
77-
# Get list of storage subdirectories, then replace with symlinks
78-
if [ -d "./zwave_stack/storage" ]; then
79-
STORAGE_DIRS=$(find ./zwave_stack/storage -mindepth 1 -maxdepth 1 -type d -exec basename {} \;)
80-
rm -rf ./zwave_stack/storage
81-
mkdir -p ./zwave_stack/storage
82-
for dir in $STORAGE_DIRS; do
83-
ln -sf ~/zwave_storage/$dir ./zwave_stack/storage/$dir
84-
done
85-
fi
86-
87-
echo "Storage symlinks created:"
88-
ls -la ./zwave_stack/storage/
74+
# Copy entire zwave_stack to native WSL filesystem for better performance
75+
rm -rf ~/zwave_stack
76+
cp -r ./zwave_stack ~/zwave_stack
77+
chmod +x ~/zwave_stack/bin/*.elf
78+
79+
echo "Z-Wave stack copied to native WSL filesystem:"
80+
ls -la ~/zwave_stack/
81+
ls -la ~/zwave_stack/bin/
82+
ls -la ~/zwave_stack/storage/
8983
9084
## Do the testing stuff
9185
- name: Run CTT tests
92-
run: npm start -- --group=Automatic
86+
shell: powershell
87+
run: |
88+
npm start -- `
89+
--group=Automatic `
90+
--group=Interactive `
91+
--exclude=CCT_CRC16EncapsulationCmdClassV1_Rev02 `
92+
--exclude=PI_CertificationData_Rev01 `
93+
--exclude=DD_ManufacturerSpecificCCData_Rev01 `
94+
--exclude=DD_VersionCCData_Rev01 `
95+
--verbose
9396
env:
9497
CI: true
9598

.gitignore

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,4 +11,7 @@ dut/zwave-js/log/
1111
dut/zwave-js/node_modules/
1212

1313
# Z-Wave stack binaries (downloaded via setup script)
14-
zwave_stack/bin/*.elf
14+
zwave_stack/bin/*.elf
15+
16+
# CTT setup archive (downloaded via setup script)
17+
setup/ctt-setup.zip

README.md

Lines changed: 21 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -176,7 +176,7 @@ See [dut/zwave-js/run.ts](dut/zwave-js/run.ts) for a reference implementation an
176176
"dut": {
177177
"name": "Your DUT Name",
178178
"runnerPath": "your-dut/run.ts",
179-
"homeId": "e6d68af7",
179+
"homeId": "d34db33f",
180180
"storageDir": "your-dut/storage",
181181
"storageFileFilter": ["%HOME_ID_LOWER%.jsonl"]
182182
}
@@ -278,6 +278,26 @@ The project comes with a ready-to-use GitHub Actions workflow for running CTT te
278278

279279
To use the workflow, configure a repository secret named `ZW_STACK_TOKEN` with a GitHub PAT that has **Contents: read** permission for the [Z-Wave-Alliance/z-wave-stack-binaries](https://github.com/Z-Wave-Alliance/z-wave-stack-binaries) repository.
280280

281+
## Troubleshooting
282+
283+
### CTT fails with `Error - RequestNodeInfo failed!`**
284+
285+
Make sure that the DUT has all command classes set (factory reset may be required) before joining the CTT network
286+
287+
### CTT fails to communicate securely with the DUT
288+
289+
Make sure that:
290+
- The DUT storage contains the correct files
291+
- The home ID is configured correctly everywhere:
292+
- `config.json`
293+
- `ctt/project/Config/TestCaseStaticController.xml` (HomeId attribute)
294+
- `ctt/project/Config/Saved Items/002_<HOMEID>.xml` (filename)
295+
296+
### Z-Wave stack binaries frequently crash or trigger the watchdog
297+
298+
Something in the storage is likely corrupted. Delete all stack storage files and re-create the CTT network from scratch.
299+
Make sure to update the home ID accordingly, see above.
300+
281301
## Documentation
282302

283303
- **[docs/ipc-protocol.md](docs/ipc-protocol.md)** - DUT Runner IPC protocol specification

config.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@
77
"runnerPath": "dut/zwave-js/run.ts",
88

99
// Z-Wave network home ID (used for storage file filtering)
10-
"homeId": "e6d68af7",
10+
"homeId": "d2658d0f",
1111

1212
// Directory containing DUT storage/cache files
1313
"storageDir": "dut/zwave-js/storage",

ctt/project/Config/Saved Items/002_E6D68AF7.xml renamed to ctt/project/Config/Saved Items/002_D2658D0F.xml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,8 @@
55
<NetworkKey Class="0" Value="CE07372267DCB354DB216761B6E9C378" />
66
<NetworkKey Class="1" Value="30B5CCF3F482A92E2F63A5C5E218149A" />
77
<NetworkKey Class="2" Value="21A29A69145E38C1601DFF55E2658521" />
8-
<NetworkKey Class="3" Value="3D54A7E5DE9D444B28B479CF13B4CA70" />
9-
<NetworkKey Class="4" Value="C2B0A8CE070B71F409A889BCCC54A49B" />
8+
<NetworkKey Class="3" Value="0F4F7E178A4207A0BBEFBF991C66F814" />
9+
<NetworkKey Class="4" Value="72D42391F7ECE63BE1B38B25D085ECD4" />
1010
</SecuritySettings>
1111
<Node Id="1" NodeInfo="D39603020100" RoleType="0" NodeType="255" CommandClasses="5E989F556C568F74">
1212
<SecurityExtension Keys="0x87" CommandClasses="85595A7A87728E7386" />

ctt/project/Config/TestCaseStaticController.xml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
<?xml version="1.0" encoding="utf-8"?>
2-
<TestCaseStaticController SocketConnectionString="127.0.0.1" HomeId="E6D68AF7" NodeId="2">
2+
<TestCaseStaticController SocketConnectionString="127.0.0.1" HomeId="D2658D0F" NodeId="2">
33
<IncludedNodes>
44
<NodeItem NodeId="1" NodeType="Z-Wave Plus v2 - Generic Controller DT">
55
<Capability>211</Capability>

0 commit comments

Comments
 (0)