Skip to content

Commit 25d5dca

Browse files
nhortonclaude
andcommitted
feat: Add review_pr standard job with expert-driven PR review
- Add new review_pr job with 3 steps: check_relevance, deep_review, improve_and_rereview - Use inline bash completion $(gh pr diff) for efficient token usage in expert prompts - Experts focus only on their domain expertise for specialized feedback - Iterative improvement cycles until all experts approve or 3 max iterations - Add learnings: domain focus in expert prompts, efficiency improvements - Code cleanup: remove redundant try/except and unused context variables - Include PR review output files from workflow execution Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
1 parent 988db47 commit 25d5dca

21 files changed

Lines changed: 1490 additions & 48 deletions
Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
---
2+
name: Expert prompts must emphasize domain focus
3+
last_updated: 2026-02-01
4+
summarized_result: |
5+
When invoking experts for tasks like PR review, prompts must explicitly tell
6+
experts to ONLY comment on their domain and not provide generic feedback.
7+
Without this, experts provide overlapping generic reviews instead of unique
8+
specialized perspectives.
9+
---
10+
11+
## Context
12+
13+
During a PR review using the `/review_pr` job, both the `deepwork-jobs` and `experts` experts were invoked to review changes to the experts system implementation. The expectation was that each expert would bring their unique domain knowledge to identify issues that generalist reviewers might miss.
14+
15+
## Problem
16+
17+
Both experts provided similar generic code review feedback. They identified the same issues (redundant exception handling, unused variables) rather than focusing on aspects specific to their domains:
18+
19+
- The `deepwork-jobs` expert should have focused on how the experts system integrates with jobs, skill generation patterns, and hook systems
20+
- The `experts` expert should have focused on expert.yml schema design, topic/learning structure, and evolution strategies
21+
22+
Instead, both provided overlapping feedback on general code quality issues.
23+
24+
## Resolution
25+
26+
Updated the expert prompts in the `review_pr` job steps to explicitly emphasize domain focus:
27+
28+
```
29+
IMPORTANT: Only comment on aspects that fall within your area of expertise.
30+
Do not provide general code review feedback on things outside your domain -
31+
other experts will cover those areas. Use your specialized knowledge to
32+
identify issues that a generalist reviewer might miss.
33+
```
34+
35+
Also added:
36+
- "Your domain: [brief description from discovery_description]" to remind experts of their focus
37+
- Request for feedback "from your expert perspective" throughout the prompt
38+
- Quality criteria including "Feedback is focused on each expert's specific domain of expertise"
39+
40+
## Key Takeaway
41+
42+
When designing job steps that invoke experts, the prompt must explicitly constrain experts to their domain. Without this, experts default to providing generic assistance rather than specialized insight. The value of the experts system comes from each expert contributing unique perspective - overlapping generic reviews wastes the multi-expert approach.
Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
---
2+
name: Review PR job efficiency improvements
3+
last_updated: 2026-02-01
4+
summarized_result: |
5+
The review_pr job used excessive tokens by sending full file contents to all
6+
experts instead of using the per-expert file segmentation from the relevance
7+
step. Also, quality validation sub-agents add overhead that may not be needed
8+
for simple checklist verification.
9+
---
10+
11+
## Context
12+
13+
During execution of the `/review_pr` job on PR #192, the workflow completed successfully but used significantly more tokens and time than necessary.
14+
15+
## Problems Identified
16+
17+
### 1. Ignored per-expert file segmentation
18+
19+
The `check_relevance` step correctly identified which files each expert should review:
20+
- deepwork-jobs: schema, parser, generator, CLI, templates, tests
21+
- experts: same files (significant overlap in this case)
22+
23+
But the `deep_review` step ignored this segmentation and sent ALL files to BOTH experts. Each expert received ~1000+ lines of code when they should have received only their relevant subset.
24+
25+
### 2. Full file contents vs. targeted excerpts
26+
27+
Experts received full file contents when often they only needed:
28+
- The diff (what changed)
29+
- Specific sections relevant to their domain
30+
- Perhaps function signatures for context
31+
32+
Sending full test files (500+ lines) to review production code changes is wasteful.
33+
34+
### 3. Quality validation sub-agent overhead
35+
36+
Each step spawned a Haiku sub-agent purely to verify a checklist of 3-5 criteria. This adds:
37+
- Network round-trip latency
38+
- Token overhead for the sub-agent context
39+
- Complexity
40+
41+
For simple boolean criteria checks, the main agent could self-verify.
42+
43+
### 4. Redundant data fetching
44+
45+
The PR diff was fetched separately in both `check_relevance` and `deep_review` steps instead of being passed along or cached.
46+
47+
## Recommendations
48+
49+
### For step instructions:
50+
51+
1. **Use inline bash completion `$(command)`**: Instead of the orchestrating agent reading
52+
data and passing it to sub-agents, embed commands directly in prompts:
53+
```
54+
$(gh pr diff)
55+
$(gh pr diff --name-only)
56+
```
57+
This executes when the sub-agent spawns, avoiding token overhead in the main conversation.
58+
59+
2. **Use relevance segmentation**: The deep_review step should explicitly use the "Relevant files" list from each expert's relevance assessment to scope what content is sent to each expert.
60+
61+
3. **Prefer diffs over full files**: For review tasks, send the diff plus minimal context (function signatures, class definitions) rather than entire files.
62+
63+
4. **Skip validation sub-agents for simple criteria**: If quality criteria are simple boolean checks (e.g., "output file exists", "all experts invoked"), the main agent can verify these directly.
64+
65+
### For job design:
66+
67+
1. **Let sub-agents fetch their own data**: Using `$(command)` in prompts means sub-agents
68+
get fresh data directly without the orchestrator reading and forwarding it.
69+
70+
2. **Consider expert-specific prompts**: Generate different prompts for each expert containing only their relevant files, rather than one massive prompt for all.
71+
72+
## Key Takeaway
73+
74+
The experts system's value comes from specialized, focused review. Sending all content to all experts defeats this purpose and wastes tokens. Job steps that invoke multiple experts should segment the workload based on each expert's declared relevance.

CHANGELOG.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
88
## [Unreleased]
99

1010
### Added
11+
- New `review_pr` standard job for expert-driven PR review
12+
- Three-step workflow: check_relevance, deep_review, improve_and_rereview
13+
- Uses inline bash completion `$(gh pr diff)` for efficient token usage
14+
- Experts focus only on their domain of expertise for specialized feedback
15+
- Iterative improvement cycles until all experts approve or max 3 iterations
1116
- Concurrent steps support in workflow definitions
1217
- Workflows can now specify nested arrays of step IDs to indicate steps that can run in parallel
1318
- Example: `steps: [setup, [task_a, task_b, task_c], finalize]` runs task_a/b/c concurrently

pr_review/deepwork-jobs/review.md

Lines changed: 149 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,149 @@
1+
# deepwork-jobs Expert Review
2+
3+
**PR**: #192 - feat: Add experts system for auto-improving domain knowledge
4+
**Date**: 2026-02-01
5+
**Reviewer**: deepwork-jobs expert
6+
7+
## Summary
8+
9+
The experts system is a well-designed addition to DeepWork that follows established patterns from the jobs system. The architecture cleanly separates concerns across schema definition (`expert_schema.py`), parsing (`experts_parser.py`), generation (`experts_generator.py`), and CLI (`experts.py`). The use of dynamic command embedding (`$(deepwork topics ...)`) in the generated agent template is an elegant solution for keeping expert agents up-to-date without regenerating them.
10+
11+
From a DeepWork Jobs perspective, this integration is solid. The experts system complements jobs by providing domain knowledge that can be leveraged during job execution. The sync command properly handles both jobs and experts, and the naming conventions (`dwe_` prefix for expert agents) avoid conflicts with job-generated skills.
12+
13+
## Issues Found
14+
15+
### Issue 1
16+
- **File**: `src/deepwork/schemas/expert_schema.py`
17+
- **Line(s)**: 22, 50, 75
18+
- **Severity**: Minor
19+
- **Issue**: The `additionalProperties: False` constraint on all three schemas is strict and may cause friction when evolving the schema. For example, if users want to add custom metadata fields to topics or learnings (like `author`, `tags`, or `priority`), they would get validation errors.
20+
- **Suggestion**: Consider whether this strictness is intentional. If extensibility is desired, either remove `additionalProperties: False` or document clearly that custom fields are not supported. The jobs schema (`JOB_SCHEMA`) also uses `additionalProperties: False`, so this is consistent with existing patterns.
21+
22+
### Issue 2
23+
- **File**: `src/deepwork/core/experts_parser.py`
24+
- **Line(s)**: 377-379, 387-389
25+
- **Severity**: Minor
26+
- **Issue**: The exception handling for topic/learning parsing re-raises `ExpertParseError` with a bare `raise`. This works but loses context about which file caused the error. The error message is already set in `parse_topic_file`/`parse_learning_file`, but the pattern is unusual.
27+
- **Suggestion**: Consider either removing the try/except entirely (letting errors propagate naturally) or adding file context to the error message.
28+
29+
### Issue 3
30+
- **File**: `src/deepwork/core/experts_generator.py`
31+
- **Line(s)**: 77-78
32+
- **Severity**: Suggestion
33+
- **Issue**: The `_build_expert_context` method includes `topics_count` and `learnings_count` but these are not used in the template (`agent-expert.md.jinja`). This is dead code that could confuse future maintainers.
34+
- **Suggestion**: Either remove these unused context variables or add them to the template if there is a future use case (e.g., showing counts in agent description).
35+
36+
### Issue 4
37+
- **File**: `src/deepwork/templates/claude/agent-expert.md.jinja`
38+
- **Line(s)**: 14
39+
- **Severity**: Minor
40+
- **Issue**: The `truncate(200)` filter is applied after `replace('\n', ' ')`, which means the 200-character limit includes any spaces that replaced newlines. A multiline description might get truncated unexpectedly short if it has many newlines.
41+
- **Suggestion**: This is likely acceptable behavior, but consider if `truncate(200, killwords=False, end='...')` would be better for cleaner truncation at word boundaries.
42+
43+
### Issue 5
44+
- **File**: `src/deepwork/cli/sync.py`
45+
- **Line(s)**: (expert agent generation section)
46+
- **Severity**: Minor
47+
- **Issue**: Expert agent generation is hardcoded to only run for Claude (`adapter.name == "claude"`). The comment says "agents live in .claude/agents/" but this should be abstracted through the adapter if/when other platforms support agents.
48+
- **Suggestion**: Consider adding an `adapter.supports_agents` property or similar abstraction so this check is not a string comparison. However, this is acceptable for now given Claude is the only platform with agent support.
49+
50+
### Issue 6
51+
- **File**: `src/deepwork/core/experts_parser.py`
52+
- **Line(s)**: 61
53+
- **Severity**: Minor
54+
- **Issue**: The frontmatter regex pattern allows optional trailing content after the closing `---` on the same line, but then requires the closing `---` to be at the start of a line. This could cause parsing issues with edge cases.
55+
- **Suggestion**: The current pattern works correctly for well-formed files. Consider adding a test case for edge cases like trailing whitespace after the closing `---`.
56+
57+
### Issue 7
58+
- **File**: `src/deepwork/standard/experts/deepwork_jobs/expert.yml`
59+
- **Line(s)**: 151
60+
- **Severity**: Suggestion
61+
- **Issue**: The expert documentation mentions "Claude Code currently only supports script hooks. Prompt hooks are parsed but not executed (documented limitation)." This is valuable but might become stale if prompt hooks are implemented.
62+
- **Suggestion**: The learning file `prompt_hooks_not_executed.md` already exists in learnings/ which is the right approach. Consider referencing it or adding a note about checking current platform capabilities.
63+
64+
## Code Suggestions
65+
66+
### Suggestion 1: Remove unused context variables
67+
68+
**File**: `src/deepwork/core/experts_generator.py`
69+
70+
Before:
71+
```python
72+
def _build_expert_context(self, expert: ExpertDefinition) -> dict:
73+
return {
74+
"expert_name": expert.name,
75+
"discovery_description": expert.discovery_description,
76+
"full_expertise": expert.full_expertise,
77+
"topics_count": len(expert.topics),
78+
"learnings_count": len(expert.learnings),
79+
}
80+
```
81+
82+
After:
83+
```python
84+
def _build_expert_context(self, expert: ExpertDefinition) -> dict:
85+
return {
86+
"expert_name": expert.name,
87+
"discovery_description": expert.discovery_description,
88+
"full_expertise": expert.full_expertise,
89+
}
90+
```
91+
92+
### Suggestion 2: Simplify redundant exception handling
93+
94+
**File**: `src/deepwork/core/experts_parser.py`
95+
96+
Before:
97+
```python
98+
for topic_file in topics_dir.glob("*.md"):
99+
try:
100+
topic = parse_topic_file(topic_file)
101+
topics.append(topic)
102+
except ExpertParseError:
103+
raise
104+
```
105+
106+
After:
107+
```python
108+
for topic_file in topics_dir.glob("*.md"):
109+
topic = parse_topic_file(topic_file)
110+
topics.append(topic)
111+
```
112+
113+
The try/except here adds no value since it just re-raises. The `parse_topic_file` function already includes the filename in its error messages.
114+
115+
### Suggestion 3: Add platform abstraction for agent support
116+
117+
**File**: `src/deepwork/core/adapters.py` (add property to `AgentAdapter` base class):
118+
119+
```python
120+
@property
121+
def supports_agents(self) -> bool:
122+
"""Whether this platform supports expert agents."""
123+
return False
124+
125+
# In ClaudeAdapter:
126+
@property
127+
def supports_agents(self) -> bool:
128+
return True
129+
```
130+
131+
Then in `sync.py`:
132+
```python
133+
if experts and adapter.supports_agents:
134+
```
135+
136+
## Approval Status
137+
138+
**APPROVED**: No blocking issues
139+
140+
The experts system is well-implemented and follows DeepWork's established patterns. The issues identified are minor improvements that do not block merging. The system correctly:
141+
142+
1. Defines clear schemas for expert, topic, and learning validation
143+
2. Parses expert definitions including nested topics and learnings
144+
3. Generates agent files with dynamic command embedding for up-to-date content
145+
4. Integrates with the sync command to generate expert agents alongside job skills
146+
5. Provides CLI commands (`deepwork topics`, `deepwork learnings`) for dynamic content retrieval
147+
6. Has comprehensive test coverage across unit and integration tests
148+
149+
The architecture cleanly separates the experts system from jobs while allowing them to coexist and complement each other.

0 commit comments

Comments
 (0)