|
| 1 | +--- |
| 2 | +name: eforge-evaluate |
| 3 | +description: > |
| 4 | + Run EvidenceForge data quality evaluation on generated log output, interpret results, review records |
| 5 | + for realism, and suggest improvements. Use this skill whenever the user wants to evaluate generated |
| 6 | + data quality, check their logs for issues, review eval scores, assess hunting feasibility, or improve |
| 7 | + a scenario's output. Also trigger when the user says "evaluate", "check quality", "how did the data |
| 8 | + turn out", "review the output", or "eforge eval". |
| 9 | +--- |
| 10 | + |
| 11 | +# EvidenceForge Data Quality Evaluator |
| 12 | + |
| 13 | +You are helping the user evaluate the quality of generated synthetic security log datasets using EvidenceForge's evaluation framework. The eval command scores datasets across 5 dimensions with 23 sub-scores, all deterministic and statistical. Your job is to run the eval, interpret the results, review sample records for realism, and provide actionable improvement suggestions. |
| 14 | + |
| 15 | +## Quick Start |
| 16 | + |
| 17 | +If the user has a generated output directory and scenario file: |
| 18 | + |
| 19 | +```bash |
| 20 | +cd /Users/dabianco/projects/SURGe/data-gen-test |
| 21 | +uv run eforge eval <output_dir> --scenario <scenario.yaml> --verbose |
| 22 | +``` |
| 23 | + |
| 24 | +If they don't have generated output yet, suggest using `/eforge generate` first. |
| 25 | + |
| 26 | +## Workflow |
| 27 | + |
| 28 | +### Step 1: Locate the Output |
| 29 | + |
| 30 | +The user needs to provide: |
| 31 | +1. **Output directory** — the directory containing generated log files (e.g., `output/retail-store-ftp-attack-20260316-140908/`) |
| 32 | +2. **Scenario file** — the YAML scenario used for generation |
| 33 | + |
| 34 | +If the user doesn't specify, look for the most recent output directory under `output/` or wherever they typically generate. Ask if you can't find it. |
| 35 | + |
| 36 | +### Step 2: Run the Evaluation |
| 37 | + |
| 38 | +Run both text and JSON output: |
| 39 | + |
| 40 | +```bash |
| 41 | +uv run eforge eval <output_dir> --scenario <scenario.yaml> --verbose |
| 42 | +``` |
| 43 | + |
| 44 | +Also capture the JSON for programmatic analysis: |
| 45 | + |
| 46 | +```bash |
| 47 | +uv run eforge eval <output_dir> --scenario <scenario.yaml> --format json 2>/dev/null |
| 48 | +``` |
| 49 | + |
| 50 | +### Step 3: Interpret Results |
| 51 | + |
| 52 | +Present a clear summary of the evaluation results. For each dimension, explain what the score means in practical terms: |
| 53 | + |
| 54 | +**Dimension 1: Record-Level Fidelity (weight 0.15)** |
| 55 | +- Tier A (Parsability): Can every record be parsed? Missing fields? Type errors? |
| 56 | +- Tier B (Co-occurrence): Do field combinations make sense? (e.g., network logons have IP addresses) |
| 57 | +- Tier C (Distributions): Are event type distributions realistic? |
| 58 | + |
| 59 | +**Dimension 2: Cross-Source Coherence (weight 0.25)** |
| 60 | +- Source Correctness: Are records in the right log sources for the system's OS? |
| 61 | +- Trace Coverage: Do storyline events leave traces in all expected formats? |
| 62 | +- Field Agreement: Do timestamps and identifiers match across sources? |
| 63 | + |
| 64 | +**Dimension 3: Background Noise Realism (weight 0.25)** |
| 65 | +- Volume Adequacy: Is there enough background noise relative to the attack signal? |
| 66 | +- User Diversity: Do different users behave differently, or are they cookie-cutter? |
| 67 | +- Activity Plausibility: Are activities appropriate for the system/OS/persona? |
| 68 | +- Anomaly Rate: Is there a realistic 1-5% rate of anomalous-but-benign events? |
| 69 | + |
| 70 | +**Dimension 4: Temporal Realism (weight 0.15)** |
| 71 | +- Work Hours: Do user events cluster in persona-defined work hours? |
| 72 | +- Burstiness: Are inter-event times bursty (realistic) or metronomic (robotic)? |
| 73 | +- Causal Ordering: Are logon→process→logoff sequences correctly ordered? |
| 74 | +- Timing Plausibility: No impossible timing (50 commands in 3 seconds)? |
| 75 | + |
| 76 | +**Dimension 5: Signal Integrity (weight 0.20)** |
| 77 | +- Event Presence: Are all storyline events visible in the logs? |
| 78 | +- Indicator Accuracy: Do traces carry the correct IPs, usernames, hostnames? |
| 79 | +- Pivot Linkability: Can a hunter pivot between consecutive attack steps? |
| 80 | +- Temporal Integrity: Are attack events in the right order at the right times? |
| 81 | + |
| 82 | +### Step 4: Qualitative Record Review |
| 83 | + |
| 84 | +Sample ~10 records from the output directory across different formats. Read them and assess: |
| 85 | + |
| 86 | +1. **Record Realism** — Do individual records look like they came from a real system? Flag anything that looks synthetic, implausible, or templated. |
| 87 | +2. **Narrative Coherence** — Read 15-20 events around a storyline step. Does the sequence tell a coherent story? Any gaps or contradictions? |
| 88 | +3. **Hunting Feasibility** — Given the scenario description and data, could a hunter realistically discover this attack? What approach would work? What obstacles exist? |
| 89 | + |
| 90 | +Present these as qualitative observations, clearly separated from the numeric scores. |
| 91 | + |
| 92 | +### Step 5: Suggest Improvements |
| 93 | + |
| 94 | +For any sub-score below 70, provide specific, actionable suggestions: |
| 95 | + |
| 96 | +| Common Issue | Suggestion | |
| 97 | +|-------------|-----------| |
| 98 | +| Low parsability | Check for empty required fields in the generator (e.g., empty SIDs) | |
| 99 | +| Low volume adequacy | Increase `baseline_activity.intensity` or add more users/systems | |
| 100 | +| Low user diversity | Add more persona types with different work patterns and activities | |
| 101 | +| Low burstiness | Known generator limitation — events are near-uniformly distributed | |
| 102 | +| Low work hour distribution | Check persona work_hours definitions; may need off-hours event generation | |
| 103 | +| Low anomaly rate | Generator may need more variation in baseline (failed logons, errors) | |
| 104 | + |
| 105 | +If multiple issues trace back to the same root cause (e.g., generator limitations), group them and explain the root cause once. |
| 106 | + |
| 107 | +### Step 6: Acceptance Criteria |
| 108 | + |
| 109 | +Report whether hard acceptance criteria pass or fail: |
| 110 | +- Parsability ≥ 98% |
| 111 | +- Source Correctness ≥ 95% |
| 112 | +- Causal Ordering ≥ 99% |
| 113 | +- Event Presence ≥ 90% |
| 114 | + |
| 115 | +If any hard criterion fails, explain what would need to change to pass. |
| 116 | + |
| 117 | +## Command Reference |
| 118 | + |
| 119 | +``` |
| 120 | +eforge eval <output_dir> --scenario <scenario.yaml> [--format json|text] [--verbose] |
| 121 | +``` |
| 122 | + |
| 123 | +- `--format text` (default): Rich terminal output with colored scores |
| 124 | +- `--format json`: Machine-readable JSON (status messages go to stderr) |
| 125 | +- `--verbose`: Show sample failures and detailed sub-score information |
0 commit comments