|
1 | 1 | # EvidenceForge Implementation Plan |
2 | 2 |
|
3 | | -**Status:** Phase 2 - Scalability ✅ COMPLETE. Ready for Phase 3. |
| 3 | +**Status:** Phase 3 - MVP Release ✅ COMPLETE. Ready for Phase 4. |
4 | 4 | **Started:** 2026-03-11 |
5 | | -**Last Updated:** 2026-03-12 (Phase 2 complete) |
| 5 | +**Last Updated:** 2026-03-13 (Phase 3 complete, MVP ready) |
6 | 6 | **Target MVP Completion:** 7-10 weeks from start |
7 | 7 |
|
8 | 8 | **Recent Completions:** |
9 | | -- ✅ Phase 2.1: Parallel Generation with Threaded Emitters |
10 | | -- ✅ Phase 2.2: 5 New Log Formats (eCAR, syslog, bash_history, snort_alert, web_access) |
11 | | -- ✅ Phase 2.3: Progress Reporting |
12 | | -- ✅ Phase 2.4: Enhanced Scenario Schema (work hours parsing, model expansion, timezone tests, validation, docs) |
13 | | -- ✅ Phase 2.5: Network Visibility Architecture (sensor placement, TAP vs SPAN, direction filtering) |
14 | | -- ✅ Phase 2.6: Persona-Based Activity Generation (work hours, peak hours, risk profiles, activity intensity) |
15 | | -- ✅ Phase 2.8+2.9: Medium Dataset Support (100-user 8h in ~14s, memory <500MB, 526 tests) |
16 | | -- ✅ Phase 2.10: OS-Aware Activity Generation (Windows + Linux support) |
| 9 | +- ✅ Phase 3.1: Claude Code Skills + Install Command (scenario, generate, validate skills; eforge install-skills) |
| 10 | +- ✅ Phase 3.2: Pre-Built Persona Library (15 personas) |
| 11 | +- ✅ Phase 3.3: Documentation (scenario reference, README, skill usage) |
| 12 | +- ✅ Phase 3.4: MVP Release Preparation |
17 | 13 |
|
18 | 14 | --- |
19 | 15 |
|
|
113 | 109 |
|
114 | 110 | - [x] `cli/commands.py` - Typer app setup with command structure |
115 | 111 | - [x] `__main__.py` - CLI entry point |
116 | | -- [x] Command: `forge init` - Write config.example.yaml to config.yaml |
117 | | -- [x] Command: `forge generate` - Generate logs from simplified scenario file |
| 112 | +- [x] Command: `eforge init` - Write config.example.yaml to config.yaml |
| 113 | +- [x] Command: `eforge generate` - Generate logs from simplified scenario file |
118 | 114 | - [x] Accept scenario file path |
119 | 115 | - [x] Accept --config, --output flags |
120 | 116 | - [x] Schema validation only (no LLM) |
|
352 | 348 |
|
353 | 349 | --- |
354 | 350 |
|
355 | | -## Phase 3: MVP Release (Skill-Based Architecture) |
| 351 | +## Phase 3: MVP Release (Skill-Based Architecture) ✅ COMPLETE |
356 | 352 |
|
357 | | -**Goal:** Ship skills for scenario creation, evaluation framework, persona library, and updated documentation. Core generation engine is already complete. |
| 353 | +**Goal:** Ship skills for scenario creation, persona library, install command, and documentation. Core generation engine already complete from Phase 2. |
358 | 354 |
|
359 | | -**Architecture shift:** Interactive/creative work (scenario creation) happens through Claude Code Skills, not a built-in LLM conversation engine. The `forge` CLI stays focused on deterministic operations (generate, validate, evaluate). |
| 355 | +**Architecture shift:** Interactive/creative work (scenario creation) happens through Claude Code Skills, not a built-in LLM conversation engine. The `eforge` CLI stays focused on deterministic operations (generate, validate, evaluate). |
360 | 356 |
|
361 | 357 | ### 3.1 Claude Code Skills + Install Command |
362 | 358 |
|
363 | | -- [ ] Create `skills/forge/scenario.md` — `/forge scenario` skill |
364 | | - - [ ] Hybrid interview flow: structured questions first, then free-form gap-filling |
365 | | - - [ ] Environment, network, personas, attacks, time window, output formats |
366 | | - - [ ] References persona library and scenario schema |
367 | | - - [ ] Generates valid scenario YAML, validates before saving |
368 | | - - [ ] Use `/skill-creator` to develop skill prompt content |
369 | | -- [ ] Create `skills/forge/generate.md` — `/forge generate` skill |
370 | | - - [ ] Runs `forge generate` on scenario file |
371 | | - - [ ] Monitors output, diagnoses errors |
372 | | - - [ ] Suggests fixes for common issues |
373 | | -- [ ] Add `forge install-skills` CLI command to `cli/commands.py` |
374 | | - - [ ] `--project` flag: copies to `.claude/skills/` (default) |
375 | | - - [ ] `--global` flag: copies to `~/.claude/skills/` |
376 | | - - [ ] Skills bundled as package data via `importlib.resources` |
377 | | -- [ ] Test: install-skills copies files correctly |
378 | | - |
379 | | -### 3.2 Pre-Built Persona Library |
380 | | - |
381 | | -- [ ] Create `personas/` directory with 15 YAML persona files |
382 | | - - [ ] Uses same schema as Persona model in scenario files |
383 | | - - [ ] developer, executive, analyst, sysadmin, help_desk, security_analyst |
384 | | - - [ ] accountant, sales, hr, marketing, data_analyst |
385 | | - - [ ] receptionist, intern, project_manager, legal_counsel |
386 | | - - [ ] Each with realistic work_hours, typical_activities, risk_profile |
387 | | -- [ ] Skills reference persona library when creating scenarios |
388 | | - |
389 | | -### 3.3 Evaluation Framework |
| 359 | +- [x] Create `skills/eforge/scenario.md` — `/eforge scenario` skill |
| 360 | + - [x] Hybrid interview flow: structured questions first, then free-form gap-filling |
| 361 | + - [x] Environment, network, personas, attacks, time window, output formats |
| 362 | + - [x] References persona library and scenario schema |
| 363 | + - [x] Generates valid scenario YAML, validates before saving |
| 364 | + - [x] Generates ENVIRONMENT.md student context document alongside scenario |
| 365 | + - [x] 10-tactic MITRE ATT&CK kill chain template |
| 366 | + - [x] Base64/encoded content must be generated via Bash, never fabricated |
| 367 | + - [x] Use `/skill-creator` to develop skill prompt content (2 iterations, 30/30 assertions) |
| 368 | +- [x] Create `skills/eforge/generate.md` — `/eforge generate` skill |
| 369 | + - [x] Runs `eforge generate` on scenario file |
| 370 | + - [x] Runs `eforge validate` as pre-flight check |
| 371 | + - [x] Monitors output, diagnoses errors |
| 372 | + - [x] Suggests fixes for common issues, escalates structural problems to `/eforge scenario` |
| 373 | + - [x] Copies ENVIRONMENT.md to output directory alongside GROUND_TRUTH.md |
| 374 | +- [x] Create `skills/eforge/validate.md` — `/eforge validate` skill |
| 375 | + - [x] Runs `eforge validate` and interprets output |
| 376 | + - [x] Fixes simple issues directly, escalates structural problems to `/eforge scenario` |
| 377 | +- [x] Add `eforge install-skills` CLI command to `cli/commands.py` |
| 378 | + - [x] `--project` flag: copies to `.claude/skills/` (default) |
| 379 | + - [x] `--global` flag: copies to `~/.claude/skills/` |
| 380 | + - [x] Skills bundled as package data via `importlib.resources` + hatch force-include |
| 381 | + - [x] Updates existing installations: overwrites changed files, removes stale files |
| 382 | + - [x] Bundles skills, personas, and scenario-reference.md |
| 383 | +- [x] Test: install-skills copies files correctly (12 tests) |
| 384 | + |
| 385 | +### 3.2 Pre-Built Persona Library ✅ COMPLETE |
| 386 | + |
| 387 | +- [x] Create `personas/` directory with 15 YAML persona files |
| 388 | + - [x] Uses same schema as Persona model in scenario files |
| 389 | + - [x] developer, executive, analyst, sysadmin, help_desk, security_analyst |
| 390 | + - [x] accountant, sales, hr, marketing, data_analyst |
| 391 | + - [x] receptionist, intern, project_manager, legal_counsel |
| 392 | + - [x] Each with realistic work_hours, typical_activities, risk_profile |
| 393 | +- [x] Skills reference persona library when creating scenarios |
| 394 | +- [x] Personas bundled with `eforge install-skills` command |
| 395 | + |
| 396 | +### 3.3 Documentation |
| 397 | + |
| 398 | +- [ ] Update `docs/scenario-reference.md` (already exists, may need refresh) |
| 399 | +- [ ] Create skill usage guide |
| 400 | +- [ ] Update README with skill-based workflow |
| 401 | +- [ ] Update TODO with Phase 3 completion status |
| 402 | + |
| 403 | +### 3.4 MVP Release Preparation |
| 404 | + |
| 405 | +- [ ] Run all tests |
| 406 | +- [ ] Manual testing: skills + generate workflow |
| 407 | +- [ ] Verify success metrics (see PRD Section 9) |
| 408 | +- [ ] Tag release: v1.0.0 |
| 409 | + |
| 410 | +**Phase 3 Milestone:** ✅ Skills-based scenario creation (3 skills), persona library (15 personas), install command, and documentation. Core generation engine already complete from Phase 2. 542+ tests passing. |
| 411 | + |
| 412 | +--- |
| 413 | + |
| 414 | +## Phase 4: Evaluation Framework (Post-MVP) |
| 415 | + |
| 416 | +**Goal:** Add a `eforge evaluate` command that assesses generated log quality with concrete, extensible metrics. |
| 417 | + |
| 418 | +### 4.1 Metrics Framework |
390 | 419 |
|
391 | 420 | - [ ] `evaluation/metrics.py` — Extensible metrics framework |
392 | 421 | - [ ] Format compliance (parse rate per format) |
393 | 422 | - [ ] Cross-reference consistency |
394 | 423 | - [ ] Ground truth IOC validation |
395 | 424 | - [ ] Specific checks defined iteratively during implementation |
| 425 | + |
| 426 | +### 4.2 Evaluator and Reporting |
| 427 | + |
396 | 428 | - [ ] `evaluation/evaluator.py` — Main evaluation logic |
397 | 429 | - [ ] `evaluation/report.py` — JSON report generation |
398 | | -- [ ] Add `forge evaluate` CLI command |
399 | | - - [ ] `--report` flag for output path |
400 | | - - [ ] `--verbose` flag for detailed findings |
401 | | -- [ ] Test: Evaluation metrics calculation |
402 | | -- [ ] Test: Report generation |
403 | 430 |
|
404 | | -### 3.4 Documentation |
| 431 | +### 4.3 CLI Command |
405 | 432 |
|
406 | | -- [ ] Update `docs/scenario-reference.md` (already exists, may need refresh) |
407 | | -- [ ] Create skill usage guide |
408 | | -- [ ] Update README with skill-based workflow |
409 | | -- [ ] Update TODO with Phase 3 completion status |
| 433 | +- [ ] Add `eforge evaluate` CLI command |
| 434 | + - [ ] `--report` flag for output path |
| 435 | + - [ ] `--verbose` flag for detailed findings |
| 436 | +- [ ] Report is informational only (no pass/fail thresholds) |
410 | 437 |
|
411 | | -### 3.5 MVP Release Preparation |
| 438 | +### 4.4 Tests |
412 | 439 |
|
413 | | -- [ ] Run all tests |
414 | | -- [ ] Manual testing: skills + generate + evaluate workflow |
415 | | -- [ ] Verify success metrics (see PRD Section 9) |
416 | | -- [ ] Tag release: v1.0.0 |
| 440 | +- [ ] Test: Evaluation metrics calculation |
| 441 | +- [ ] Test: Report generation |
| 442 | +- [ ] Test: CLI integration |
417 | 443 |
|
418 | | -**Phase 3 Milestone:** Skills-based scenario creation, evaluation framework, persona library, and documentation. Core generation engine already complete from Phase 2. |
| 444 | +**Phase 4 Milestone:** `eforge evaluate` command produces JSON quality reports for generated datasets. |
419 | 445 |
|
420 | 446 | --- |
421 | 447 |
|
|
424 | 450 | **Not part of MVP, but tracked here for future reference.** |
425 | 451 |
|
426 | 452 | ### Short-term (Post-MVP) |
427 | | -- [ ] Bedrock LLM client for semantic validation (`forge validate --semantic`) |
| 453 | +- [ ] Bedrock LLM client for semantic validation (`eforge validate --semantic`) |
428 | 454 | - [ ] Checkpointing and resume for long-running generation |
429 | 455 | - [ ] Additional skills: create-persona, create-log-format, create-network, analyze-output |
430 | 456 | - [ ] Example scenario collection (ransomware, credential stuffing, insider threat) |
|
433 | 459 | - [ ] PyPI package distribution |
434 | 460 | - [ ] Additional log formats (CloudTrail, Azure Activity, GCP Audit, database logs) |
435 | 461 | - [ ] Network diagram ingestion: auto-infer sensor placement (span vs tap) from diagram topology |
| 462 | +- [ ] Per-user work hours jitter: randomize start/end/lunch times ±30min per user for realistic staggered arrivals |
436 | 463 | - [ ] Performance optimizations (Rust extensions, better parallelization) |
| 464 | +- [ ] Full user directory export as separate CSV file for large scenarios (ENVIRONMENT.md enhancement) |
| 465 | +- [ ] Authentication and naming convention documentation in ENVIRONMENT.md |
| 466 | +- [ ] Separate student/instructor output packages (GROUND_TRUTH.md in instructor-only directory) |
437 | 467 |
|
438 | 468 | ### Medium-term |
439 | 469 | - [ ] Alternative LLM backends (OpenAI, Ollama, Anthropic native, Gemini) |
|
0 commit comments