Skip to content

Commit ba595ea

Browse files
DavidJBiancoclaude
andcommitted
Restructure Phase 3 for skill-based architecture
Major documentation rewrite reflecting the shift from a built-in conversational CLI to Claude Code Skills for scenario creation. PRD.md: Complete rewrite - Architecture: "Skills for creation, CLI for generation" - Replace `forge new` with `/forge scenario` skill - Add `forge install-skills` command spec - Remove built-in LLM client from MVP scope - Add Skills Architecture section (4.4) - Update project structure to match actual codebase - Address all review findings (schema mismatches, missing specs) AGENTS.md: Targeted updates - Project overview reflects skill-based architecture - LLM Integration marked as deferred - Project structure matches actual codebase - New Skills section with adding-a-skill guide TODO.md: Phase 3 restructured - 3.1: Skills + install-skills command - 3.2: Persona library (15 personas) - 3.3: Evaluation framework (forge evaluate) - 3.4: Documentation updates - 3.5: MVP release preparation - Deferred to post-MVP: LLM client, checkpointing, semantic validation Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 0a1a437 commit ba595ea

3 files changed

Lines changed: 626 additions & 821 deletions

File tree

AGENTS.md

Lines changed: 72 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -6,13 +6,13 @@ This document provides AI coding agents with everything needed to write consiste
66

77
EvidenceForge generates realistic synthetic security logs for cybersecurity threat hunting training and research. The system uses a two-phase hybrid architecture:
88

9-
**Phase 1 - Scenario Creation (LLM-intensive):** Conversational CLI interface accepts natural language descriptions of computing environments and activities. LLM researches TTPs via MITRE ATT&CK, expands high-level descriptions into detailed execution plans, and outputs structured YAML scenario files with companion research markdown.
9+
**Phase 1 - Scenario Creation (Skill-assisted):** Claude Code Skills guide users through scenario creation via structured interviews. Skills research TTPs via MITRE ATT&CK, expand high-level descriptions into detailed execution plans, and output structured YAML scenario files with companion research markdown.
1010

1111
**Phase 2 - Log Generation (Deterministic):** Generation engine executes the detailed scenario plan WITHOUT any LLM calls, producing large-scale, temporally consistent datasets across multiple log formats (Windows Event Logs, Zeek, Syslog, Snort/Suricata, web logs) with coordinated cross-references (matching LogonIDs, PIDs, session data).
1212

1313
This architecture combines LLM flexibility/realism with deterministic speed, cost-efficiency, and reproducibility.
1414

15-
**Key Principle:** Phase 1 does ALL the creative work. Phase 2 is a deterministic renderer that executes the plan. Never call LLMs during generation.
15+
**Key Principle:** The `forge` CLI is a deterministic tool. Creative/interactive work happens through Claude Code Skills, not built-in LLM calls. Phase 2 is a deterministic renderer that executes the plan. Never call LLMs during generation.
1616

1717
## 🔴 MANDATORY: Implementation State Tracking
1818

@@ -81,11 +81,13 @@ Use markdown checkboxes organized by phase/feature:
8181
- uv for package management, virtual environments, and script running
8282
- Pydantic v2 for all data validation and schema management
8383

84-
**LLM Integration:**
85-
- boto3 for AWS Bedrock access (MVP only supports Bedrock, other backends are future enhancements)
86-
- Primary model: `anthropic.claude-sonnet-4-6-v1:0` (conversation & validation)
87-
- Research model: `anthropic.claude-sonnet-4-6-v1:0` (TTP research)
88-
- Generation model: `anthropic.claude-haiku-4-5-v1:0` (cost optimization for bulk tasks)
84+
**LLM Integration (deferred):**
85+
- Built-in LLM client via boto3/Bedrock is deferred to future phases
86+
- Scenario creation is handled by Claude Code Skills (external to the codebase)
87+
- The `llm/` directory is a placeholder; model IDs kept as reference for future use:
88+
- Primary model: `anthropic.claude-sonnet-4-6-v1:0` (conversation & validation)
89+
- Research model: `anthropic.claude-sonnet-4-6-v1:0` (TTP research)
90+
- Generation model: `anthropic.claude-haiku-4-5-v1:0` (cost optimization for bulk tasks)
8991

9092
**CLI & Output:**
9193
- Typer for CLI framework (excellent Pydantic integration)
@@ -115,11 +117,13 @@ log-generator/
115117
├── config.example.yaml # Example configuration
116118
├── .env.example # Example environment variables
117119
118-
├── personas/ # Pre-built persona library (reduce LLM usage)
119-
│ ├── developer.yaml
120-
│ ├── accountant.yaml
121-
│ ├── executive.yaml
122-
│ └── ... # 10-15 common personas
120+
├── skills/
121+
│ └── forge/ # Claude Code Skills for scenario creation
122+
│ ├── scenario.md # /forge scenario - guided scenario creation
123+
│ └── generate.md # /forge generate - generation workflow
124+
125+
├── personas/ # Pre-built persona library
126+
│ └── ... # Persona YAML files (developer, accountant, etc.)
123127
124128
├── src/
125129
│ └── log_generator/
@@ -128,8 +132,7 @@ log-generator/
128132
│ │
129133
│ ├── cli/
130134
│ │ ├── __init__.py
131-
│ │ ├── commands.py # Typer CLI command implementations
132-
│ │ └── conversation.py # Interactive conversation interface
135+
│ │ └── commands.py # Typer CLI command implementations
133136
│ │
134137
│ ├── models/
135138
│ │ ├── __init__.py
@@ -140,25 +143,24 @@ log-generator/
140143
│ │
141144
│ ├── validation/
142145
│ │ ├── __init__.py
143-
│ │ ├── schema.py # Pydantic-based schema validation
144-
│ │ ├── semantic.py # LLM-based semantic validation
145-
│ │ └── repair.py # Interactive repair logic
146+
│ │ └── schema.py # Pydantic-based schema validation
146147
│ │
147148
│ ├── generation/
148149
│ │ ├── __init__.py
149-
│ │ ├── engine.py # Main generation orchestrator
150+
│ │ ├── engine.py # Main generation orchestrator (includes persona logic)
150151
│ │ ├── state_manager.py # State tracking (sessions, processes, connections)
151-
│ │ ├── persona.py # Persona-based activity generation
152-
│ │ ├── activity.py # Activity script execution
153-
│ │ ├── checkpoint.py # Checkpoint/resume logic
152+
│ │ ├── activity.py # Activity script execution (includes persona behavior)
153+
│ │ ├── network_visibility.py # Network visibility/perspective logic
154154
│ │ └── emitters/
155155
│ │ ├── __init__.py
156-
│ │ ├── base.py # Base emitter ABC
157-
│ │ ├── windows.py # Windows Event Log emitter
158-
│ │ ├── zeek.py # Zeek log emitter
159-
│ │ ├── syslog.py # Syslog emitter
160-
│ │ ├── snort.py # Snort/Suricata emitter
161-
│ │ └── web.py # Web/proxy log emitter
156+
│ │ ├── base.py # Base emitter ABC
157+
│ │ ├── bash_history.py # Bash history emitter
158+
│ │ ├── ecar.py # ECAR emitter
159+
│ │ ├── snort.py # Snort/Suricata emitter
160+
│ │ ├── syslog.py # Syslog emitter
161+
│ │ ├── web.py # Web/proxy log emitter
162+
│ │ ├── windows.py # Windows Event Log emitter
163+
│ │ └── zeek.py # Zeek log emitter
162164
│ │
163165
│ ├── formats/
164166
│ │ ├── __init__.py
@@ -171,12 +173,8 @@ log-generator/
171173
│ │ ├── snort.yaml
172174
│ │ └── web.yaml
173175
│ │
174-
│ ├── llm/
175-
│ │ ├── __init__.py
176-
│ │ ├── client.py # Bedrock client wrapper
177-
│ │ ├── prompts.py # System prompts for various tasks
178-
│ │ ├── research.py # TTP research logic (30s timeout per query)
179-
│ │ └── retry.py # Retry logic with exponential backoff
176+
│ ├── llm/ # Placeholder for future built-in LLM integration
177+
│ │ └── __init__.py
180178
│ │
181179
│ ├── evaluation/
182180
│ │ ├── __init__.py
@@ -561,9 +559,11 @@ def redact_secrets(obj: dict[str, Any]) -> dict[str, Any]:
561559

562560
## Key Architecture Patterns
563561

564-
### LLM Client Abstraction
562+
### LLM Client Abstraction (Future)
565563

566-
The LLM client is abstracted behind a Protocol to support future backends (OpenAI, Ollama, etc.):
564+
The LLM client abstraction is planned for future built-in LLM integration. Currently, scenario creation is handled by Claude Code Skills (external to the codebase). The patterns below are kept as reference for when the `llm/` module is implemented.
565+
566+
The LLM client will be abstracted behind a Protocol to support future backends (OpenAI, Ollama, etc.):
567567

568568
```python
569569
from typing import Protocol
@@ -599,9 +599,9 @@ class BedrockClient:
599599
...
600600
```
601601

602-
**Usage pattern:**
602+
**Usage pattern (future):**
603603
```python
604-
# In cli/conversation.py
604+
# Example for future built-in LLM integration
605605
llm = BedrockClient(
606606
model_id=config.bedrock.model_primary,
607607
region=config.aws.region,
@@ -614,7 +614,7 @@ response = llm.chat(messages=[
614614
])
615615
```
616616

617-
**Critical:** Never call LLM during generation phase. Only in conversation (`new` command) and validation (`validate` command with `--interactive`).
617+
**Critical:** Never call LLM during generation phase. Scenario creation is currently handled by Claude Code Skills, not built-in LLM calls.
618618

619619
### Retry Logic with Backoff
620620

@@ -1347,11 +1347,44 @@ def test_state_manager_creates_unique_pids(user_count: int):
13471347
pids.add(pid)
13481348
```
13491349

1350+
## Skills
1351+
1352+
Claude Code Skills handle the interactive, creative aspects of scenario creation -- work that was originally planned as a built-in conversational CLI.
1353+
1354+
**Location:** `skills/forge/` directory
1355+
1356+
**Installation:**
1357+
```bash
1358+
# Install skills for the current project
1359+
forge install-skills --project
1360+
1361+
# Install skills globally
1362+
forge install-skills --global
1363+
```
1364+
1365+
**MVP Skills:**
1366+
- `/forge scenario` -- Guided scenario creation through a structured interview, producing a validated YAML scenario file
1367+
- `/forge generate` -- Generation workflow that validates a scenario and runs the deterministic engine
1368+
1369+
**Key design points:**
1370+
- Skills are markdown prompt files (`.md`), not Python code
1371+
- They run inside Claude Code, not inside the `forge` CLI process
1372+
- Skills follow a hybrid interview pattern: structured questions first (environment, users, systems), then free-form refinement
1373+
- Skills reference the scenario schema from `docs/scenario-reference.md`
1374+
1375+
### Adding a New Skill
1376+
1377+
1. Create `skills/forge/{name}.md` with the skill prompt
1378+
2. Follow the hybrid interview pattern: structured questions first, then free-form elaboration
1379+
3. Reference the scenario schema from `docs/scenario-reference.md` to ensure output validity
1380+
4. Test interactively by running the skill in Claude Code
1381+
5. Update the `install-skills` command if needed to include the new skill
1382+
13501383
## Common Pitfalls
13511384

13521385
### DO NOT
13531386

1354-
1. **Call LLMs during generation** - All LLM work happens in `new` and `validate --interactive` commands only
1387+
1. **Call LLMs during generation** - All creative/LLM work happens via Claude Code Skills before generation
13551388
```python
13561389
# WRONG
13571390
def generate_event(event_type: str) -> Event:

0 commit comments

Comments
 (0)