Skip to content

Commit 1a615a8

Browse files
nhortonclaude
andauthored
feat: add requirements traceability annotations and new LA-REQ tests (#346)
* feat: add requirements traceability annotations and 218 new LA-REQ tests - Add REQ ID annotations to 32 existing test files so the traceability system can match tests to their spec requirements (section-level IDs in module docstrings + sub-item IDs in test method comments) - Write 218 new tests across 5 files for learning-agents requirements: - LA-REQ-001: plugin structure (22 tests) - LA-REQ-003/005: agent file structure and issue lifecycle (49 tests) - LA-REQ-004: session tracking hooks (45 tests) - LA-REQ-006/010: learning cycle and issue reporting (65 tests) - LA-REQ-011: skill routing (37 tests) - Add PLUG-REQ-001.12 tests for session/agent identity injection hooks - Coverage improvement: 610 → 987 covered IDs (363 → 93 uncovered) - All non-LA deepwork core requirements now fully covered Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: complete LA-REQ coverage with DeepSchemas — 100% requirements coverage - Add 6 anonymous DeepSchemas for judgment-based LA-REQ requirements: - create-agent/.deepschema.SKILL.md.yml (LA-REQ-002) - learn/.deepschema.SKILL.md.yml (LA-REQ-006, partial 003) - identify/.deepschema.SKILL.md.yml (LA-REQ-007, partial 005) - investigate-issues/.deepschema.SKILL.md.yml (LA-REQ-008) - incorporate-learnings/.deepschema.SKILL.md.yml (LA-REQ-009) - prompt-review/.deepschema.SKILL.md.yml (LA-REQ-012) - Place REQ IDs in YAML comments (not requirement body text) so they serve as metadata for traceability without polluting the prompt content sent to reviewer agents - Add section-level REQ docstring annotations to LA-REQ test files - Note JOBS-REQ-009 as deprecated in test_quality_gate docstring - Add JOBS-REQ-012 parent ID to deepwork_jobs deepschema Coverage: 987 → 1080 covered (93 → 0 uncovered). All 420 spec IDs across DW-REQ, JOBS-REQ, REVIEW-REQ, PLUG-REQ, and LA-REQ are now traceable to either tests, deepschemas, or review rules. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: require REQ IDs in YAML comments not requirement body text Add the req-ids-in-comments requirement to the standard DeepSchema definition. Requirement IDs (e.g., LA-REQ-007.4) MUST be placed in YAML comments above the entry, not in the body text. The body text gets sent to the reviewer agent as the rule prompt — IDs there add noise without informational value. EXCEPTION: when a requirement is functionally about citing or producing a specific ID, the ID may appear in the body. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: address review findings on requirements traceability PR - Fix malformed docstring in test_hook_wrapper.py (banner block was inside the docstring, splitting the prose summary) - Remove duplicate `import json` in test_state.py - Remove redundant `project_root`/`state_manager` fixture redefinitions in 9 test classes (pytest inherits module-scoped fixtures into classes) - Fix tautological assertion in test_learning_agents_plugin.py (`has_structured or has_noop` after asserting has_noop is dead code) - Remove tautological tests from test_learning_agents_file_structure.py that only verified test fixture constants, not real behavior - Update bash wildcard patterns in .claude/settings.json to use `:*` word boundary instead of bare `*` - Add `dict[str, Any]` typing to test helper return types - Reframe LA-REQ-006.8/006.10 in learn skill DeepSchema to be filesystem- verifiable ("SKILL.md MUST instruct the agent to...") - Update CHANGELOG.md with PR #346 entry Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: address second-iteration review findings - Add `# YOU MUST NOT MODIFY THIS TEST UNLESS THE REQUIREMENT CHANGES` second line to all 435 traceability comments in the diff - Migrate fragile keyword-on-prose tests for LA-REQ-006.1/3/6/7/9/11/12 from test_learning_agents_cycle.py into the learn deepschema as judgment-based requirements - Update doc/learning_agents_architecture.md: add `model:` field to agent file example, name conversation_transcript.jsonl in Session Logs, expand create-agent/learn/identify/investigate-issues/ incorporate-learnings/prompt-review subsections, document dynamic inclusion of additional_learning_guidelines/*.md - Apply mypy fixes from python_lint review (drop now-unneeded type:ignore comments in test_tools_coverage.py) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: address test-quality findings from python_code_review - test_learning_agents_session_tracking.py: lift inline `import re` and `from glob import glob` to module top; extract `_locate_reminder_file()` helper to remove duplicated reminder-path fallback logic - test_learning_agents_file_structure.py: add `check=True` to the three `subprocess.run` calls that were silently masking script failures - test_learning_agents_cycle.py: harden the `content.find()` slicing tests for LA-REQ-006.5 by asserting both positions are >= 0 before the order comparison; cache `_read_skill()` via `@functools.cache` so each SKILL.md is read once across the test module instead of 35 times Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: move PR #346 changelog entries to Unreleased section The requirements traceability entries were incorrectly placed under [0.13.0] (an already-released section). They belong in [Unreleased] since this PR has not yet been released. Also adds an anonymous DeepSchema for CHANGELOG.md that enforces this: new entries added in a PR must go under [Unreleased], not under any already-released version section. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 3a6954d commit 1a615a8

51 files changed

Lines changed: 4139 additions & 176 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.claude/settings.json

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -100,8 +100,10 @@
100100
"Write(./.deepwork/**)",
101101
"Bash(deepwork:*)",
102102
"Bash(gh pr edit:*)",
103-
"Bash(learning_agents/scripts/*)",
104-
"Bash(bash learning_agents/scripts/*)",
103+
"Bash(gh api:*)",
104+
"Bash(learning_agents/scripts/*:*)",
105+
"Bash(bash learning_agents/scripts/*:*)",
106+
"Bash(bash .deepwork/requirements_traceability_info.sh:*)",
105107
"WebSearch",
106108
"Skill(deepwork)",
107109
"mcp__deepwork__get_workflows",

.deepschema.CHANGELOG.md.yml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
requirements:
2+
new-entries-in-unreleased: |
3+
New changelog entries added in a PR MUST be placed under the `## [Unreleased]`
4+
section, not under any already-released version section (e.g., `## [0.13.0]`).
5+
Released version sections are immutable historical records — only the release
6+
workflow may modify them when cutting a new release.

CHANGELOG.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
1010
### Added
1111

1212
- Integration tests for quality gate review caching (JOBS-REQ-004.5.7)
13+
- Requirements traceability coverage now at 100% (#346)
14+
- Added section-level REQ ID annotations to 32 existing test files for traceability
15+
- Wrote 218 new tests across 5 files for learning-agents requirements (LA-REQ-001, 003, 004, 005, 006, 010, 011)
16+
- Added 6 anonymous DeepSchemas for judgment-based learning-agents skill requirements (LA-REQ-002, 007, 008, 009, 012)
17+
- Added new PLUG-REQ-001.12 tests for session/agent identity injection hooks
18+
- New `req-ids-in-comments` requirement in the standard DeepSchema definition: requirement IDs must be placed in YAML comments, not requirement body text
1319

1420
### Changed
1521

doc/learning_agents_architecture.md

Lines changed: 18 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -91,6 +91,7 @@ Example structure:
9191
---
9292
name: <agent-name>
9393
description: "<discovery description>"
94+
model: sonnet
9495
---
9596

9697
# Core Knowledge
@@ -108,16 +109,24 @@ Located in `.deepwork/learning-agents/<agent-name>/topics/`
108109
Learnings are incident post-mortems from past agent sessions capturing mistakes, root causes, and generalizable insights. Review them before starting work to avoid repeating past mistakes. Located in `.deepwork/learning-agents/<agent-name>/learnings/`.
109110
```
110111

112+
The frontmatter `model:` field is required (LA-REQ-002.9) and defaults to `sonnet` to keep agent invocations cost-effective.
113+
111114
### Session Logs (transient, gitignored)
112115

113116
```
114117
.deepwork/tmp/agent_sessions/<session_id>/<agent_id>/
115118
├── needs_learning_as_of_timestamp # Flag file (body = ISO 8601 timestamp)
116119
├── learning_last_performed_timestamp # When learning was last run on this conversation
117120
├── agent_used # Body = LearningAgent folder name
121+
├── conversation_transcript.jsonl # Snapshot of the agent's conversation transcript
118122
└── <brief-name>.issue.yml # Issues found during learning
119123
```
120124

125+
The `conversation_transcript.jsonl` file is the captured agent transcript used by the
126+
`investigate-issues` skill to research root causes (LA-REQ-008.6). If it cannot be
127+
found for a session, the learning cycle skips that session and reports the issue in
128+
the final summary.
129+
121130
See `learning_log_folder_structure.md` for full details.
122131

123132
## Hooks
@@ -185,7 +194,7 @@ Entry point skill. Dynamically lists existing LearningAgents at load time, then
185194
- `/learning-agents report_issue <agent_id> <details>` → invokes `report-issue`
186195

187196
#### create-agent
188-
Creates a new LearningAgent. Accepts an optional `template-path` argument — when provided, the new agent is seeded with the template agent's `core-knowledge.md`, `topics/`, and `learnings/` as a starting point. The skill first invokes a setup script, then guides the user through filling in the content.
197+
Creates a new LearningAgent. Accepts an optional `template-path` argument — when provided, the new agent is seeded with the template agent's `core-knowledge.md`, `topics/`, and `learnings/` as a starting point. The skill first invokes a setup script, then guides the user through filling in the content. The skill enforces dash-separated lowercase agent names, refuses to overwrite existing agents, and ensures the generated `.claude/agents/<agent-name>.md` frontmatter includes a `model:` field (defaulting to `sonnet`).
189198

190199
**Step 1 — Setup script (`scripts/create_agent.sh`)**
191200

@@ -241,21 +250,23 @@ After the script runs, the skill prompts the user to describe what the agent is
241250
Fills in key files in the LearningAgent directory — initial topics and/or learnings if the user provides seed knowledge about the domain.
242251

243252
#### learn
244-
Runs the full learning cycle on all sessions needing it. Workflow:
253+
Runs the full learning cycle on all sessions needing it. The skill takes no arguments — any text after `learn` is ignored. Sessions are processed sequentially (not in parallel) to avoid conflicts when the same agent appears in multiple sessions. Workflow:
245254
1. Uses `!`find ...`` to inject a list of all paths containing a `needs_learning_as_of_timestamp` file into the prompt
246255
2. For each such folder, spawns a Task with the `LearningAgentExpert` agent using **Sonnet model** to run the `identify` skill
247256
3. After identification completes, spawns a Task with the `LearningAgentExpert` to run `investigate-issues` then `incorporate-learnings` in sequence
248257

258+
If no pending sessions are found (or the `.deepwork/tmp/agent_sessions/` directory is missing), the skill informs the user and stops. If a sub-skill Task fails for a session, the skill logs the failure, skips that session, continues processing remaining sessions, and does NOT mark `needs_learning_as_of_timestamp` as resolved. On completion, the skill outputs a summary containing total sessions processed, total issues identified, list of agents updated, key learnings per agent, and any skipped sessions with reasons. The `learn` skill itself MUST NOT modify agent files directly — all knowledge base updates are delegated to the sub-skills.
259+
249260
#### setup
250261
Configures project permissions for the LearningAgents plugin. Adds required Bash and file access rules to `.claude/settings.json` so hooks and scripts can run without manual approval.
251262

252263
#### prompt-review
253-
Standalone skill (not routed through the `/learning-agents` dispatcher). Reviews prompt/instruction files against Anthropic prompt engineering best practices. Can be invoked directly as `/learning-agents:prompt-review`.
264+
Standalone skill (not routed through the `/learning-agents` dispatcher). Reviews prompt/instruction files against Anthropic prompt engineering best practices. Can be invoked directly as `/learning-agents:prompt-review`. Accepts a target file path argument and produces actionable feedback grouped by best-practice category — it does not modify the target file.
254265

255266
### Hidden Skills (used by LearningAgentExpert during learning)
256267

257268
#### identify
258-
Reviews a session transcript to find issues. Takes the session/agent_id folder path as an argument.
269+
Reviews a session transcript to find issues. Takes the session/agent_id folder path as an argument. The skill reads the transcript and surfaces concrete mistakes, underperformance, or knowledge gaps — emitting one issue per finding via `report-issue`. Per-agent identification guidance is loaded dynamically from the agent's `additional_learning_guidelines/issue_identification.md` file (empty file = no extra guidance).
259270
- Uses `!`cat ...`` to inject `learning_last_performed_timestamp` value into the prompt
260271
- Reads the transcript and identifies mistakes, underperformance, or knowledge gaps
261272
- Calls `report-issue` for each issue found
@@ -265,14 +276,14 @@ Reviews a session transcript to find issues. Takes the session/agent_id folder p
265276
Creates an `<brief-name>.issue.yml` file in the session's agent log folder. Sets initial status to `identified` with the issue description and observed timestamps. See `issue_yml_format.md` for the schema.
266277

267278
#### investigate-issues
268-
Processes all `identified` issues in a session folder:
279+
Processes all `identified` issues in a session folder, researching root causes from transcript evidence and the agent's existing knowledge. Per-agent investigation guidance is loaded dynamically from `additional_learning_guidelines/issue_investigation.md`.
269280
1. Finds issues with status `identified` (includes example bash command in skill prompt)
270281
2. Uses `!`cat $0/agent_used`` to inject the agent name, then `!`cat`` to load the agent's instructions — avoiding extra round trips
271-
3. Reads the agent's full expertise and knowledge
282+
3. Reads the agent's full expertise and knowledge, plus the captured `conversation_transcript.jsonl`
272283
4. For each issue: reads relevant transcript sections, determines root cause, updates status to `investigated` with `investigation_report`
273284

274285
#### incorporate-learnings
275-
Integrates investigated issues into the LearningAgent:
286+
Integrates investigated issues into the LearningAgent. Topics document conceptual reference material (how things work — patterns, APIs, conventions); learnings document incident post-mortems of specific mistakes (what went wrong and why). Per-agent incorporation guidance is loaded dynamically from `additional_learning_guidelines/learning_from_issues.md`.
276287
1. Finds issues with status `investigated` (includes example bash command)
277288
2. For each issue, takes a learning action — one of:
278289
- **Update core knowledge**: Modify `core-knowledge.md` to address the knowledge gap
Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
# DeepSchema for create-agent skill — validates LA-REQ-002 judgment-based requirements
2+
requirements:
3+
# LA-REQ-002.1
4+
skill-invocation-and-name-prompt: >
5+
The skill MUST be invocable via /learning-agents create <name>, and if
6+
no name is provided, MUST prompt the user for a name.
7+
8+
# LA-REQ-002.2
9+
name-normalization: >
10+
The skill MUST normalize agent names by converting spaces and uppercase
11+
to lowercase dashes (e.g. "Rails ActiveJob" -> "rails-activejob").
12+
13+
# LA-REQ-002.3
14+
pre-scaffold-validation: >
15+
The skill MUST check that .deepwork/learning-agents/ exists (or create it)
16+
and verify no agent with the same name already exists before scaffolding.
17+
18+
# LA-REQ-002.4
19+
agent-directory-path: >
20+
The skill MUST place the new agent under .deepwork/learning-agents/<name>/.
21+
22+
# LA-REQ-002.5
23+
scaffold-invocation: >
24+
The skill MUST invoke the create_agent.sh script to create the directory
25+
structure rather than creating files inline.
26+
27+
# LA-REQ-002.7
28+
file-paths-for-scaffold: >
29+
The skill MUST pass the correct agent name and plugin root path to the
30+
scaffold script so it creates files in the right locations.
31+
32+
# LA-REQ-002.9
33+
claude-agent-file-config: >
34+
The skill MUST configure the Claude Code agent file with appropriate
35+
name, description, and model settings after scaffolding.
36+
37+
# LA-REQ-002.12
38+
interactive-configuration: >
39+
After scaffolding, the skill MUST ask the user about the agent's domain
40+
of expertise and update core-knowledge.md and the agent file accordingly.
41+
42+
# LA-REQ-002.13
43+
optional-knowledge-seeding: >
44+
The skill MUST offer the user the option to seed initial topics or
45+
learnings, creating files following the formats from LA-REQ-003.
46+
47+
# LA-REQ-002.14
48+
creation-summary: >
49+
Upon completion, the skill MUST output a summary listing created files,
50+
usage instructions for invoking the agent, and a note about the learning
51+
cycle.
52+
53+
# LA-REQ-002.15
54+
no-overwrites: >
55+
The skill MUST NOT overwrite any existing file without explicit user
56+
confirmation.
57+
58+
# LA-REQ-002.23
59+
template-configuration-guidance: >
60+
When a template was used, the skill MUST present the copied core-knowledge.md
61+
and offer to keep, modify, or replace it, and list copied topics/learnings
62+
for review.
63+
64+
# LA-REQ-002.24
65+
template-summary: >
66+
When a template was used, the completion summary MUST note which template
67+
was used and how many files were seeded from it.
Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
requirements:
2+
# LA-REQ-007.1
3+
not-user-invocable: >
4+
The skill frontmatter MUST set `user-invocable: false` so it cannot be
5+
directly invoked by the user.
6+
7+
# LA-REQ-007.2
8+
input-argument-path: >
9+
The skill MUST accept a single argument: the path to the session/agent_id
10+
folder (e.g., `.deepwork/tmp/agent_sessions/<session_id>/<agent_id>/`).
11+
12+
# LA-REQ-007.3
13+
agent-context-loading: >
14+
The skill MUST dynamically load the agent name from `$ARGUMENTS/agent_used`,
15+
the last learning timestamp from `$ARGUMENTS/learning_last_performed_timestamp`,
16+
and additional identification guidelines from
17+
`.deepwork/learning-agents/<agent-name>/additional_learning_guidelines/issue_identification.md`.
18+
19+
# LA-REQ-007.4
20+
transcript-location: >
21+
The skill MUST instruct the agent to read the transcript from the session
22+
folder. The transcript file MUST be located relative to `$ARGUMENTS`.
23+
24+
# LA-REQ-007.5
25+
missing-transcript-handling: >
26+
The skill MUST instruct the agent to report an error and stop if no
27+
transcript file is found, without creating any issue files.
28+
29+
# LA-REQ-007.6
30+
transcript-format-jsonl: >
31+
The skill MUST instruct the agent to parse the transcript as JSONL and focus
32+
on `type: "assistant"` messages and `type: "tool_result"` entries to evaluate
33+
agent behavior.
34+
35+
# LA-REQ-007.7
36+
incremental-processing: >
37+
The skill MUST instruct the agent to skip transcript lines before
38+
`learning_last_performed_timestamp` when that value exists, analyzing only
39+
interactions since the last learning cycle.
40+
41+
# LA-REQ-007.9
42+
issue-categories: >
43+
The skill MUST enumerate all five issue categories: incorrect outputs,
44+
knowledge gaps, missed context, poor judgment, and pattern failures.
45+
46+
# LA-REQ-007.10
47+
trivial-issue-filtering: >
48+
The skill MUST instruct the agent to skip trivial issues including minor
49+
formatting differences, environmental failures, and issues already covered
50+
by existing learnings or issue files.
51+
52+
# LA-REQ-007.11
53+
report-via-sub-skill: >
54+
The skill MUST instruct the agent to invoke the `report-issue` skill once
55+
per issue found, passing the session folder path and a description of the
56+
problem.
57+
58+
# LA-REQ-007.12
59+
no-duplicate-issues: >
60+
The skill MUST instruct the agent to avoid creating duplicate issue files
61+
for the same problem, including by listing existing issue files for
62+
reference.
63+
64+
# LA-REQ-007.13
65+
no-root-cause-investigation: >
66+
The skill MUST explicitly prohibit root cause investigation, deferring that
67+
to the `investigate-issues` skill.
68+
69+
# LA-REQ-007.14
70+
no-knowledge-base-modification: >
71+
The skill MUST explicitly prohibit modification of the agent's knowledge
72+
base (core-knowledge.md, topics, or learnings).
73+
74+
# LA-REQ-007.15
75+
summary-output: >
76+
The skill MUST instruct the agent to output a summary containing session ID,
77+
agent name, count of issues found, and a table listing each issue with its
78+
category and description (or a message indicating no issues were found).
79+
80+
# LA-REQ-007.8
81+
agent-focus: >
82+
The skill MUST focus on interactions involving the agent identified in
83+
the agent_used file and SHOULD NOT analyze interactions with other
84+
agents.
85+
86+
# LA-REQ-007, LA-REQ-007.16
87+
allowed-tools-only: >
88+
The skill MUST restrict tool usage to only: Read, Grep, Glob, Skill.
89+
90+
# LA-REQ-005.8
91+
timestamp-source-accuracy: >
92+
The skill MUST instruct that seen_at_timestamps come from either exact
93+
transcript line timestamps or the current UTC time when reviewing live
94+
sessions.
95+
96+
# LA-REQ-005.11
97+
investigation-report-content: >
98+
The skill MUST instruct that investigation_report fields contain specific
99+
line numbers from the transcript as evidence.
100+
101+
# LA-REQ-005.13
102+
no-duplicate-issues-lifecycle: >
103+
The skill MUST instruct the agent to check for existing issues before
104+
creating new ones, preventing duplicate issue files for the same problem
105+
within a single session.

0 commit comments

Comments
 (0)