Skip to content

Commit f5312db

Browse files
DavidJBiancoclaude
andcommitted
Add /eforge evaluate skill and clean up obsolete LLM references
- Create evaluate.md skill: runs eforge eval, interprets scores, reviews sample records for realism, suggests improvements - Clean up TODO.md: mark Bedrock client, semantic validation, and alternative LLM backends as obsolete (replaced by skills architecture) - Update install_skills.py to bundle the new evaluate skill (4 skills total) - Phase 4.7 now complete via skill-based approach Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 1b3772a commit f5312db

3 files changed

Lines changed: 137 additions & 10 deletions

File tree

TODO.md

Lines changed: 11 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -293,9 +293,9 @@
293293
- [x] Backward compatible: no persona = uniform activity at all hours
294294
- [x] 13 tests in `tests/unit/test_persona_activity.py`
295295

296-
### 2.7 LLM Integration (Bedrock Client) — DEFERRED TO PHASE 3
296+
### 2.7 LLM Integration (Bedrock Client) — ✅ OBSOLETE
297297

298-
**Moved to Phase 3.1** - LLM client and retry logic are only needed for the conversational interface. Will be implemented alongside Phase 3.1 (Conversational Interface).
298+
**Replaced by Claude Code Skills.** Phase 3 adopted a skills-based architecture where all LLM-driven work (scenario creation, validation, evaluation review) happens through Claude Code skills rather than a built-in Bedrock client. No code-level LLM calls are needed.
299299

300300
### 2.8+2.9 Medium Dataset Support & Phase 2 Completion ✅ COMPLETE
301301

@@ -340,7 +340,7 @@
340340

341341
**Phase 2 Status: ✅ COMPLETE**
342342
- ✅ All phases complete: 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.8+2.9, 2.10
343-
- 🔀 Deferred: 2.7 (LLM Integration) → folded into Phase 3.1
343+
- ✅ Obsolete: 2.7 (LLM Integration) → replaced by Claude Code Skills architecture
344344

345345
**Phase 2 Milestone:** Can generate datasets across 7 formats in parallel with threaded emitters. 100-user 8-hour scenarios complete in ~14 seconds. Persona-based temporal distributions, network visibility with TAP/SPAN sensors, OS-aware log routing, and cross-log consistency. 526 tests passing.
346346

@@ -468,10 +468,11 @@
468468
- [x] Organic Anomaly Rate (1-5% target)
469469
- [x] Tests: 13 tests
470470

471-
### 4.7 LLM Spot-Check Layer (optional) — DEFERRED
471+
### 4.7 Evaluate Skill (`/eforge evaluate`) ✅ COMPLETE
472472

473-
- [ ] Deferred until LLM client is available (Bedrock integration)
474-
- [ ] `--llm-review` flag placeholder in CLI (accepted but not yet functional)
473+
- [x] Create `commands/eforge/evaluate.md` — runs `eforge eval`, interprets results, provides qualitative LLM review
474+
- [x] Replaces the originally planned Bedrock-based LLM spot-check layer
475+
- [x] Qualitative review (record realism, narrative coherence, hunting feasibility) handled conversationally by the skill
475476

476477
### 4.8 Integration & Acceptance Criteria ✅ COMPLETE
477478

@@ -485,7 +486,7 @@
485486
- [ ] Update `commands/eforge/scenario.md` to check sensor coverage during authoring
486487
- [ ] Flag when storyline events may not be discoverable given declared topology
487488

488-
**Phase 4 Status:** ✅ All 5 scoring dimensions complete (23 sub-scores). `eforge eval` produces comprehensive quality reports. 653+ tests. LLM spot-check deferred.
489+
**Phase 4 Status:**COMPLETE. All 5 scoring dimensions (23 sub-scores), acceptance criteria, `/eforge evaluate` skill. 653+ tests.
489490

490491
**Baseline scores on retail-store-ftp-attack (24K records, 4 sources):**
491492
- Overall: 78/100
@@ -502,11 +503,11 @@
502503
**Not part of MVP, but tracked here for future reference.**
503504

504505
### Short-term (Post-MVP)
505-
- [ ] Bedrock LLM client for semantic validation (`eforge validate --semantic`)
506+
- [ ] ~~Bedrock LLM client for semantic validation~~ → Handled by `/eforge validate` skill
506507
- [ ] Checkpointing and resume for long-running generation
507508
- [ ] Additional skills: create-persona, create-log-format, create-network, analyze-output
508509
- [ ] Example scenario collection (ransomware, credential stuffing, insider threat)
509-
- [ ] ~~Subjective realism evaluation (LLM-based)~~Moved to Phase 4.7 (LLM Spot-Check Layer)
510+
- [ ] ~~Subjective realism evaluation (LLM-based)~~Handled by `/eforge evaluate` skill
510511
- [ ] Config file inheritance/templating
511512
- [ ] PyPI package distribution
512513
- [ ] Additional log formats (CloudTrail, Azure Activity, GCP Audit, database logs)
@@ -518,7 +519,7 @@
518519
- [ ] Separate student/instructor output packages (GROUND_TRUTH.md in instructor-only directory)
519520

520521
### Medium-term
521-
- [ ] Alternative LLM backends (OpenAI, Ollama, Anthropic native, Gemini)
522+
- [ ] ~~Alternative LLM backends (OpenAI, Ollama, Anthropic native, Gemini)~~ → No code-level LLM calls; skills use whatever model the user's Claude Code runs
522523
- [ ] Web UI for scenario creation
523524
- [ ] Streaming output to SIEM/data lakes
524525
- [ ] Log format auto-detection from samples

commands/eforge/evaluate.md

Lines changed: 125 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,125 @@
1+
---
2+
name: eforge-evaluate
3+
description: >
4+
Run EvidenceForge data quality evaluation on generated log output, interpret results, review records
5+
for realism, and suggest improvements. Use this skill whenever the user wants to evaluate generated
6+
data quality, check their logs for issues, review eval scores, assess hunting feasibility, or improve
7+
a scenario's output. Also trigger when the user says "evaluate", "check quality", "how did the data
8+
turn out", "review the output", or "eforge eval".
9+
---
10+
11+
# EvidenceForge Data Quality Evaluator
12+
13+
You are helping the user evaluate the quality of generated synthetic security log datasets using EvidenceForge's evaluation framework. The eval command scores datasets across 5 dimensions with 23 sub-scores, all deterministic and statistical. Your job is to run the eval, interpret the results, review sample records for realism, and provide actionable improvement suggestions.
14+
15+
## Quick Start
16+
17+
If the user has a generated output directory and scenario file:
18+
19+
```bash
20+
cd /Users/dabianco/projects/SURGe/data-gen-test
21+
uv run eforge eval <output_dir> --scenario <scenario.yaml> --verbose
22+
```
23+
24+
If they don't have generated output yet, suggest using `/eforge generate` first.
25+
26+
## Workflow
27+
28+
### Step 1: Locate the Output
29+
30+
The user needs to provide:
31+
1. **Output directory** — the directory containing generated log files (e.g., `output/retail-store-ftp-attack-20260316-140908/`)
32+
2. **Scenario file** — the YAML scenario used for generation
33+
34+
If the user doesn't specify, look for the most recent output directory under `output/` or wherever they typically generate. Ask if you can't find it.
35+
36+
### Step 2: Run the Evaluation
37+
38+
Run both text and JSON output:
39+
40+
```bash
41+
uv run eforge eval <output_dir> --scenario <scenario.yaml> --verbose
42+
```
43+
44+
Also capture the JSON for programmatic analysis:
45+
46+
```bash
47+
uv run eforge eval <output_dir> --scenario <scenario.yaml> --format json 2>/dev/null
48+
```
49+
50+
### Step 3: Interpret Results
51+
52+
Present a clear summary of the evaluation results. For each dimension, explain what the score means in practical terms:
53+
54+
**Dimension 1: Record-Level Fidelity (weight 0.15)**
55+
- Tier A (Parsability): Can every record be parsed? Missing fields? Type errors?
56+
- Tier B (Co-occurrence): Do field combinations make sense? (e.g., network logons have IP addresses)
57+
- Tier C (Distributions): Are event type distributions realistic?
58+
59+
**Dimension 2: Cross-Source Coherence (weight 0.25)**
60+
- Source Correctness: Are records in the right log sources for the system's OS?
61+
- Trace Coverage: Do storyline events leave traces in all expected formats?
62+
- Field Agreement: Do timestamps and identifiers match across sources?
63+
64+
**Dimension 3: Background Noise Realism (weight 0.25)**
65+
- Volume Adequacy: Is there enough background noise relative to the attack signal?
66+
- User Diversity: Do different users behave differently, or are they cookie-cutter?
67+
- Activity Plausibility: Are activities appropriate for the system/OS/persona?
68+
- Anomaly Rate: Is there a realistic 1-5% rate of anomalous-but-benign events?
69+
70+
**Dimension 4: Temporal Realism (weight 0.15)**
71+
- Work Hours: Do user events cluster in persona-defined work hours?
72+
- Burstiness: Are inter-event times bursty (realistic) or metronomic (robotic)?
73+
- Causal Ordering: Are logon→process→logoff sequences correctly ordered?
74+
- Timing Plausibility: No impossible timing (50 commands in 3 seconds)?
75+
76+
**Dimension 5: Signal Integrity (weight 0.20)**
77+
- Event Presence: Are all storyline events visible in the logs?
78+
- Indicator Accuracy: Do traces carry the correct IPs, usernames, hostnames?
79+
- Pivot Linkability: Can a hunter pivot between consecutive attack steps?
80+
- Temporal Integrity: Are attack events in the right order at the right times?
81+
82+
### Step 4: Qualitative Record Review
83+
84+
Sample ~10 records from the output directory across different formats. Read them and assess:
85+
86+
1. **Record Realism** — Do individual records look like they came from a real system? Flag anything that looks synthetic, implausible, or templated.
87+
2. **Narrative Coherence** — Read 15-20 events around a storyline step. Does the sequence tell a coherent story? Any gaps or contradictions?
88+
3. **Hunting Feasibility** — Given the scenario description and data, could a hunter realistically discover this attack? What approach would work? What obstacles exist?
89+
90+
Present these as qualitative observations, clearly separated from the numeric scores.
91+
92+
### Step 5: Suggest Improvements
93+
94+
For any sub-score below 70, provide specific, actionable suggestions:
95+
96+
| Common Issue | Suggestion |
97+
|-------------|-----------|
98+
| Low parsability | Check for empty required fields in the generator (e.g., empty SIDs) |
99+
| Low volume adequacy | Increase `baseline_activity.intensity` or add more users/systems |
100+
| Low user diversity | Add more persona types with different work patterns and activities |
101+
| Low burstiness | Known generator limitation — events are near-uniformly distributed |
102+
| Low work hour distribution | Check persona work_hours definitions; may need off-hours event generation |
103+
| Low anomaly rate | Generator may need more variation in baseline (failed logons, errors) |
104+
105+
If multiple issues trace back to the same root cause (e.g., generator limitations), group them and explain the root cause once.
106+
107+
### Step 6: Acceptance Criteria
108+
109+
Report whether hard acceptance criteria pass or fail:
110+
- Parsability ≥ 98%
111+
- Source Correctness ≥ 95%
112+
- Causal Ordering ≥ 99%
113+
- Event Presence ≥ 90%
114+
115+
If any hard criterion fails, explain what would need to change to pass.
116+
117+
## Command Reference
118+
119+
```
120+
eforge eval <output_dir> --scenario <scenario.yaml> [--format json|text] [--verbose]
121+
```
122+
123+
- `--format text` (default): Rich terminal output with colored scores
124+
- `--format json`: Machine-readable JSON (status messages go to stderr)
125+
- `--verbose`: Show sample failures and detailed sub-score information

src/evidenceforge/cli/install_skills.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@
1313
"scenario.md",
1414
"generate.md",
1515
"validate.md",
16+
"evaluate.md",
1617
]
1718

1819
REFERENCE_FILES = [

0 commit comments

Comments
 (0)