Skip to content

Commit 7df9ae9

Browse files
ncrmroclaude
andcommitted
docs: Add validation section to job porting design
Adds seven validation scenarios covering audit accuracy, round-trip porting, precedence override, concretization, blocked ports, doc spec collisions, and sync isolation. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
1 parent b0bafcc commit 7df9ae9

1 file changed

Lines changed: 91 additions & 0 deletions

File tree

doc/job-porting.md

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -141,3 +141,94 @@ The port skill depends on changes to the DeepWork CLI and sync pipeline:
141141
2. **`deepwork sync --global`** — sync must be scope-aware, generating skills to `~/.claude/skills/` from `~/.deepwork/jobs/`.
142142
3. **Precedence in sync** — when both global and local jobs exist with the same name, local wins. The merged set gets synced.
143143
4. **`deepwork install` awareness** — optionally inject global user jobs into projects, or let sync handle the merge at skill generation time.
144+
145+
## Validation
146+
147+
Validation should confirm that ported jobs work correctly in both directions and that the portability audit catches real problems. The approach uses a known test job with intentional portability issues alongside a clean portable job.
148+
149+
### Test Fixture: Two Reference Jobs
150+
151+
Create two jobs in a test project for use across all validation scenarios:
152+
153+
1. **`portable_example`** — a minimal job with `scope: portable`, parameterized `{output_dir}`, prompt-based hooks, no project-specific paths in instructions, and no `AGENTS.md`. This should port cleanly in both directions with zero warnings.
154+
155+
2. **`local_example`** — a job with `scope: local`, hardcoded output paths (`src/reports/analysis.md`), a script hook (`hooks/validate.sh` that calls `npm test`), an `AGENTS.md` with bespoke learnings, and a step instruction referencing `app/models/user.rb`. This should trigger every portability warning.
156+
157+
### 1. Portability Audit Accuracy
158+
159+
Verify the audit catches all known issues and produces no false positives.
160+
161+
**Using `local_example`, confirm the audit flags:**
162+
163+
- Hardcoded output path `src/reports/analysis.md` (no `{output_dir}` parameterization).
164+
- Script hook `hooks/validate.sh` with project-specific `npm test` call.
165+
- Step instruction containing `app/models/user.rb` reference.
166+
- `AGENTS.md` with bespoke content.
167+
168+
**Using `portable_example`, confirm the audit produces:**
169+
170+
- Zero warnings.
171+
- Zero blocking issues.
172+
173+
### 2. Project-to-Global Round Trip
174+
175+
Port `portable_example` from project to global, then back to a second empty project. Compare the result against the original.
176+
177+
1. Run `/deepwork_jobs.port` to push `portable_example` to `~/.deepwork/jobs/`.
178+
2. Verify files land in:
179+
- `~/.deepwork/jobs/portable_example/job.yml`
180+
- `~/.deepwork/jobs/portable_example/steps/*.md`
181+
- `~/.deepwork/doc_specs/` (if the job references any doc specs)
182+
3. Verify `doc_spec:` paths in the global `job.yml` reference `~/.deepwork/doc_specs/`, not `.deepwork/doc_specs/`.
183+
4. Verify no `AGENTS.md` was copied (or it is empty/generalized).
184+
5. Run `deepwork sync --global` and verify skills appear in `~/.claude/skills/portable_example/`.
185+
6. In a fresh test project, run `/deepwork_jobs.port` to pull the global job.
186+
7. Verify `doc_spec:` paths are rewritten back to `.deepwork/doc_specs/`.
187+
8. Run `deepwork sync` and verify skills generate in `.claude/skills/portable_example/`.
188+
9. Diff the original and round-tripped `job.yml` and step instructions — they should be semantically identical (path prefixes differ, content matches).
189+
190+
### 3. Precedence Override
191+
192+
Confirm local jobs take priority over global jobs with the same name.
193+
194+
1. Port `portable_example` to global.
195+
2. In a project that also has a local `portable_example` with a modified description, run `deepwork sync`.
196+
3. Verify the generated skill in `.claude/skills/portable_example/SKILL.md` contains the local description, not the global one.
197+
4. Delete the local job directory and re-run `deepwork sync`.
198+
5. Verify the generated skill now contains the global description.
199+
200+
### 4. Global-to-Project Concretization
201+
202+
Confirm that parameterized output paths can be replaced with concrete project paths.
203+
204+
1. Port `portable_example` to global (outputs use `{output_dir}/report.md`).
205+
2. In a new project, run `/deepwork_jobs.port` to pull the job, choosing to concretize `{output_dir}` as `reports/q1`.
206+
3. Verify the local `job.yml` outputs read `reports/q1/report.md`.
207+
4. Run the job and verify output lands at `reports/q1/report.md`.
208+
209+
### 5. Blocked Port on Unresolved Issues
210+
211+
Confirm that the port operation does not silently produce a broken global job.
212+
213+
1. Run `/deepwork_jobs.port` on `local_example` toward global.
214+
2. Verify the audit presents all flagged issues to the user.
215+
3. If the user does not resolve or acknowledge every issue, confirm the port aborts without writing to `~/.deepwork/`.
216+
4. If the user acknowledges and transforms all issues, confirm the port completes and the resulting global job has no project-specific references remaining.
217+
218+
### 6. Doc Spec Collision Handling
219+
220+
Confirm that name collisions in `~/.deepwork/doc_specs/` are handled safely.
221+
222+
1. Place a doc spec `report.md` in `~/.deepwork/doc_specs/` with content A.
223+
2. Port a project job that references its own `.deepwork/doc_specs/report.md` with content B.
224+
3. Verify the port warns about the collision and asks the user whether to overwrite, rename, or skip.
225+
4. Confirm no silent overwrite occurs.
226+
227+
### 7. Sync --global Isolation
228+
229+
Confirm that global sync does not interfere with project-local skills.
230+
231+
1. In a project with local jobs, run `deepwork sync --global`.
232+
2. Verify no files in `.claude/skills/` (project) were modified.
233+
3. Run `deepwork sync` (local).
234+
4. Verify no files in `~/.claude/skills/` (global) were modified.

0 commit comments

Comments
 (0)