Add temporal, cross-source, and noise realism dimensions to eval

Phase 4.3-4.5: Complete all 5 scoring dimensions (23 sub-scores total):
- Dimension 4: Temporal Realism (work hours, burstiness, causal ordering, timing)
- Dimension 2: Cross-Source Coherence (source correctness, trace coverage, field agreement)
- Dimension 3: Background Noise Realism (volume, diversity, plausibility, anomaly rate)
- Visibility model for OS-to-format mapping + network sensor reuse
- Anomaly detector for organic anomaly rate scoring
- Causal pair rule definitions (logon→process, logon→logoff, SSH→bash)
- 41 new tests (653 total passing)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Author: DavidJBianco

TODO.md

@@ -1,8 +1,8 @@
 # EvidenceForge Implementation Plan
 
-**Status:** Phase 3 - MVP Release ✅ COMPLETE. Phase 4 planned.
+**Status:** Phase 4 - Data Quality Evaluation ✅ SCORING COMPLETE (5/5 dimensions). LLM spot-check deferred.
 **Started:** 2026-03-11
-**Last Updated:** 2026-03-16 (Phase 4 Data Quality Evaluation planned, PRD at docs/data-quality-prd.md)
+**Last Updated:** 2026-03-16 (Phase 4 eval framework complete: 5 dimensions, 23 sub-scores, 653+ tests)
 **Target MVP Completion:** 7-10 weeks from start
 
 **Recent Completions:**
@@ -419,98 +419,81 @@
 
 **Scoring model:** 5 dimensions roll up to an overall 0-100 score. Each dimension has weighted sub-scores. Acceptance criteria are a separate pass/fail layer on top of scores.
 
-### 4.1 Report Framework & CLI Command
-
-- [ ] Create `src/evidenceforge/evaluation/` package structure
-- [ ] `evaluation/engine.py` — Orchestrator: runs all dimensions, collects scores, applies acceptance criteria
-- [ ] `evaluation/report.py` — Report formatting (Rich text + JSON output)
-- [ ] Add `eforge eval` CLI command to `cli/commands.py`
-  - [ ] Args: `<output_directory> --scenario <scenario.yaml>`
-  - [ ] Flags: `--format json|text`, `--verbose`, `--llm-review`
-- [ ] Data model for scores: DimensionScore, SubScore, AcceptanceCriterion, QualityReport
-- [ ] Test: CLI argument parsing, report formatting
-
-### 4.2 Dimension 1 — Record-Level Fidelity (weight: 0.15)
-
-- [ ] `evaluation/dimensions/record_fidelity.py`
-- [ ] **Tier A: Parsability & Structure** (0.40) — parse every record, check required fields, validate types
-  - [ ] Reuse/extend `src/evidenceforge/formats/validator.py` for per-record validation
-  - [ ] Format-specific parsers: XML (Windows), JSON (Zeek, eCAR), regex (syslog, web, snort, bash_history)
-- [ ] **Tier B: Co-occurrence Rules** (0.35) — field combination checks
-  - [ ] `evaluation/rules/co_occurrence.py` — rule engine + initial YAML rule sets per format
-  - [ ] Start with 5-10 rules per format (e.g., Win 4624 LogonType=3 requires IpAddress; Zeek dns requires port 53)
-  - [ ] Rules stored as YAML data files alongside format definitions
-- [ ] **Tier C: Population Statistics** (0.25) — aggregate distribution checks
-  - [ ] `evaluation/rules/distributions.py` — reference profiles + divergence scoring
-  - [ ] Event type distribution, user agent diversity, process name frequency
-- [ ] Test: known-good fixtures score high, known-bad fixtures score low
-
-### 4.3 Dimension 5 — Signal Integrity (weight: 0.20)
-
-- [ ] `evaluation/dimensions/signal_integrity.py`
-- [ ] **Event Presence** (0.25) — storyline events visible within sensor coverage produced traces
-- [ ] **Indicator Accuracy** (0.25) — present events carry correct IPs, usernames, hostnames, processes
-- [ ] **Pivot Linkability** (0.25) — consecutive storyline steps share pivotable indicators
-- [ ] **Storyline Temporal Integrity** (0.25) — events in correct order at correct times
-- [ ] Parse scenario storyline + build expected trace mapping
-- [ ] Test: scenarios with missing/wrong indicators score appropriately
-
-### 4.4 Dimension 4 — Temporal Realism (weight: 0.15)
-
-- [ ] `evaluation/dimensions/temporal.py`
-- [ ] **Work Hour Distribution** (0.20) — user events cluster in persona work hours (80-95%)
-- [ ] **Human Burstiness** (0.20) — CV of inter-event times per user (target: 1-3)
-- [ ] **System Process Regularity** (0.20) — autocorrelation of system event timestamps
-- [ ] **Causal Ordering** (0.20) — known causal pairs correctly sequenced
-  - [ ] `evaluation/rules/causal_pairs.py` — causal pair definitions per format
-- [ ] **Timing Plausibility** (0.20) — no physically impossible timing
-- [ ] Test: uniform-distribution data scores low on burstiness; correctly-ordered data scores high on causality
-
-### 4.5 Dimension 2 — Cross-Source Coherence (weight: 0.25)
-
-- [ ] `evaluation/dimensions/cross_source.py`
-- [ ] `evaluation/visibility.py` — build visibility model from scenario topology
-  - [ ] Which hosts have which logging (OS → format mapping)
-  - [ ] Which network sensors see which traffic (reuse NetworkVisibilityEngine patterns)
-- [ ] **Source Correctness** (0.20) — no records in wrong sources (Windows cmds in bash_history, syslog from Windows hosts)
-- [ ] **Storyline Trace Coverage** (0.20) — expected traces found in visible sources
-- [ ] **Cross-Source Field Agreement** (0.20) — timestamps (UTC-normalized, ±30s tolerance), IPs, usernames agree
-- [ ] **Baseline Coherence — Sampled** (0.20) — random 5-10% sample of baseline events checked
-- [ ] **Baseline Coherence — Aggregate** (0.20) — per-user/system event counts proportional across sources
-- [ ] Test: cross-OS mismatch fixtures, missing trace fixtures
-
-### 4.6 Dimension 3 — Background Noise Realism (weight: 0.25)
-
-- [ ] `evaluation/dimensions/noise_realism.py`
-- [ ] `evaluation/anomaly.py` — lightweight statistical anomaly detector
-- [ ] **Volume Adequacy** (0.25) — noise-to-signal ratio meets intensity thresholds (low ~500:1, med ~5K:1, high ~10K:1+)
-- [ ] **User Behavioral Diversity** (0.25) — entropy/pairwise similarity of per-user event distributions
-- [ ] **Activity Plausibility** (0.25) — activities match persona/system/OS assignments
-- [ ] **Organic Anomaly Rate** (0.25) — 1-5% of background flagged as anomalous (too clean = bad, too chaotic = bad)
-- [ ] Test: cookie-cutter user activity scores low on diversity; clean-only background scores low on anomaly rate
-
-### 4.7 LLM Spot-Check Layer (optional)
-
-- [ ] Implement `--llm-review` flag in eval command
-- [ ] Sample 20-50 records for qualitative LLM assessment
-- [ ] Three check types: Record Realism, Narrative Coherence, Hunting Feasibility
-- [ ] Append commentary to report (does not affect numeric scores)
-
-### 4.8 Integration & Acceptance Criteria
-
-- [ ] Acceptance criteria engine: hard requirements (reject) + quality targets (flag)
-  - [ ] Hard: Dim1 Tier A >= 98%, Dim2 Source Correctness >= 95%, Dim4 Causal Ordering >= 99%, Dim5 Event Presence >= 90%
-  - [ ] Targets: Overall >= 70, each dimension >= 60, Dim3 anomaly rate 1-5%
-- [ ] Supplementary metrics: Difficulty Estimate (indicator distinctiveness, signal ratio)
-- [ ] Integration test: `eforge generate` + `eforge eval` pipeline on test scenarios
-- [ ] Run eval on existing fixtures (minimal, attack, retail-store) to establish baselines
+### 4.1 Report Framework & CLI Command ✅ COMPLETE
+
+- [x] Create `src/evidenceforge/evaluation/` package structure
+- [x] `evaluation/engine.py` — Orchestrator with progress callbacks, acceptance criteria
+- [x] `evaluation/report.py` — Rich text + JSON report formatting
+- [x] `evaluation/models.py` — QualityReport, DimensionScore, SubScore, AcceptanceCriterion
+- [x] Add `eforge eval` CLI command with Rich progress bars
+- [x] 7 log parsers: XML (Windows), NDJSON (Zeek, eCAR), regex (syslog, snort, web, bash_history)
+- [x] Tests: 50 tests for parsers, models, CLI, report
+
+### 4.2 Dimension 1 — Record-Level Fidelity (weight: 0.15) ✅ COMPLETE
+
+- [x] `evaluation/dimensions/record_fidelity.py`
+- [x] Tier A: Parsability (reuses formats/validator.py)
+- [x] Tier B: Co-occurrence rules (YAML rule sets, 5-10 per format)
+- [x] Tier C: Population statistics (Jensen-Shannon divergence, hand-rolled)
+- [x] Tests: known-good/bad fixtures
+
+### 4.3 Dimension 5 — Signal Integrity (weight: 0.20) ✅ COMPLETE
+
+- [x] `evaluation/dimensions/signal_integrity.py`
+- [x] Event Presence, Indicator Accuracy, Pivot Linkability, Storyline Temporal Integrity
+- [x] Storyline resolution (ISO + relative offsets), keyword-based activity matching
+- [x] Tests: 17 tests covering all sub-scores
+
+### 4.4 Dimension 4 — Temporal Realism (weight: 0.15) ✅ COMPLETE
+
+- [x] `evaluation/dimensions/temporal.py`
+- [x] Work Hour Distribution, Human Burstiness (CV), System Process Regularity (autocorrelation)
+- [x] Causal Ordering with `evaluation/rules/causal_pairs.yaml` (4 pair types)
+- [x] Timing Plausibility (command rate + transfer speed checks)
+- [x] Tests: 17 tests
+
+### 4.5 Dimension 2 — Cross-Source Coherence (weight: 0.25) ✅ COMPLETE
+
+- [x] `evaluation/dimensions/cross_source.py`
+- [x] `evaluation/visibility.py` — OS→format mapping + NetworkVisibilityEngine reuse
+- [x] Source Correctness, Storyline Trace Coverage, Cross-Source Field Agreement
+- [x] Baseline Coherence (Sampled + Aggregate)
+- [x] Tests: 11 tests
+
+### 4.6 Dimension 3 — Background Noise Realism (weight: 0.25) ✅ COMPLETE
+
+- [x] `evaluation/dimensions/noise_realism.py`
+- [x] `evaluation/anomaly.py` — statistical anomaly detector (work hours, failed ops, rare processes, unexpected ports)
+- [x] Volume Adequacy, User Behavioral Diversity (cosine similarity), Activity Plausibility
+- [x] Organic Anomaly Rate (1-5% target)
+- [x] Tests: 13 tests
+
+### 4.7 LLM Spot-Check Layer (optional) — DEFERRED
+
+- [ ] Deferred until LLM client is available (Bedrock integration)
+- [ ] `--llm-review` flag placeholder in CLI (accepted but not yet functional)
+
+### 4.8 Integration & Acceptance Criteria ✅ COMPLETE
+
+- [x] Acceptance criteria engine: hard requirements (reject) + quality targets (flag)
+  - [x] Hard: Dim1 Tier A >= 98%, Dim2 Source Correctness >= 95%, Dim4 Causal Ordering >= 99%, Dim5 Event Presence >= 90%
+- [x] Integration test: `eforge generate` + `eforge eval` pipeline verified on retail-store-ftp-attack
+- [x] 653+ tests passing, all 5 dimensions scoring
 
 ### 4.9 Scenario Skill Update
 
 - [ ] Update `commands/eforge/scenario.md` to check sensor coverage during authoring
 - [ ] Flag when storyline events may not be discoverable given declared topology
 
-**Phase 4 Milestone:** `eforge eval` produces comprehensive quality reports with 5-dimension scoring, acceptance criteria, and optional LLM spot-checks. Reports feed back into generation improvements.
+**Phase 4 Status:** ✅ All 5 scoring dimensions complete (23 sub-scores). `eforge eval` produces comprehensive quality reports. 653+ tests. LLM spot-check deferred.
+
+**Baseline scores on retail-store-ftp-attack (24K records, 4 sources):**
+- Overall: 78/100
+- Dim 1 (Record Fidelity): 68 — empty SIDs, distribution skew
+- Dim 2 (Cross-Source): 100 — correct OS mapping, field agreement
+- Dim 3 (Noise Realism): 50 — volume too low, users too similar
+- Dim 4 (Temporal): 67 — work hours too concentrated, burstiness low
+- Dim 5 (Signal Integrity): 100 — all storyline events found with correct indicators
 
 ---
 

src/evidenceforge/evaluation/anomaly.py

@@ -0,0 +1,178 @@
+"""Lightweight statistical anomaly detection for background events.
+
+Flags events that are anomalous-but-benign (realistic noise for hunters).
+Used by Dimension 3 (Background Noise Realism) to score Organic Anomaly Rate.
+"""
+
+from collections import Counter
+from typing import Any
+
+from evidenceforge.evaluation.dimensions.temporal import _extract_username
+from evidenceforge.evaluation.parsers import ParsedRecord
+from evidenceforge.models.scenario import Scenario
+from evidenceforge.validation.schema import BUILTIN_ACCOUNTS
+
+# Failed operation indicators
+_FAILED_EVENT_IDS = {4625}  # Windows failed logon
+_FAILED_HTTP_CODES = set(range(400, 600))
+_FAILED_SYSLOG_KEYWORDS = ["failed", "denied", "error", "invalid", "unauthorized"]
+
+
+def detect_anomalies(
+    records: dict[str, list[ParsedRecord]],
+    scenario: Scenario,
+) -> tuple[int, int]:
+    """Detect anomalous events in background noise.
+
+    Returns:
+        (anomalous_count, total_checked) — both ints.
+    """
+    # Build context
+    persona_hours = _build_persona_hours(scenario)
+    service_ports = _build_service_ports(scenario)
+    process_freq = _build_process_frequency(records)
+    system_accounts = {a.lower() for a in BUILTIN_ACCOUNTS}
+
+    total = 0
+    anomalous = 0
+
+    for format_name, record_list in records.items():
+        for record in record_list:
+            if record.parse_errors:
+                continue
+            total += 1
+
+            is_anomalous = (
+                _is_off_hours(record, persona_hours)
+                or _is_failed_operation(record, format_name)
+                or _is_rare_process(record, format_name, process_freq)
+                or _is_unexpected_port(record, format_name, service_ports)
+            )
+            if is_anomalous:
+                anomalous += 1
+
+    return anomalous, total
+
+
+def _build_persona_hours(scenario: Scenario) -> dict[str, list[int]]:
+    """Map username → list of work hours from persona."""
+    result: dict[str, list[int]] = {}
+    persona_map = {}
+    if scenario.personas:
+        persona_map = {p.name: p for p in scenario.personas}
+
+    for user in scenario.environment.users:
+        if user.persona and user.persona in persona_map:
+            persona = persona_map[user.persona]
+            if persona.work_hours_parsed:
+                result[user.username.lower()] = persona.work_hours_parsed.get("hours", [])
+
+    return result
+
+
+def _build_service_ports(scenario: Scenario) -> set[int]:
+    """Collect all declared service ports from scenario systems."""
+    # Common service-to-port mappings
+    service_ports: set[int] = set()
+    port_map = {
+        "ssh": 22, "http": 80, "https": 443, "ftp": 21, "smtp": 25,
+        "dns": 53, "rdp": 3389, "smb": 445, "mysql": 3306, "postgres": 5432,
+        "iis": 80, "nginx": 80, "apache": 80, "sql server": 1433,
+    }
+    for system in scenario.environment.systems:
+        for svc in system.services:
+            port = port_map.get(svc.lower())
+            if port:
+                service_ports.add(port)
+    return service_ports
+
+
+def _build_process_frequency(records: dict[str, list[ParsedRecord]]) -> Counter:
+    """Count process/command frequencies across all records."""
+    freq: Counter = Counter()
+    for fmt, record_list in records.items():
+        for rec in record_list:
+            proc = _extract_process_key(rec, fmt)
+            if proc:
+                freq[proc] += 1
+    return freq
+
+
+def _extract_process_key(record: ParsedRecord, fmt: str) -> str | None:
+    """Extract a process/command identifier from a record."""
+    f = record.fields
+    if fmt == "windows_event_security" and f.get("EventID") == 4688:
+        return f.get("NewProcessName", "")
+    if fmt == "bash_history":
+        cmd = f.get("command", "")
+        return cmd.split()[0] if cmd else None
+    if fmt == "ecar" and f.get("object") == "PROCESS":
+        return f.get("image_path", "")
+    return None
+
+
+def _is_off_hours(record: ParsedRecord, persona_hours: dict[str, list[int]]) -> bool:
+    """Check if event is outside user's persona work hours."""
+    if not record.timestamp:
+        return False
+    user = _extract_username(record)
+    if not user or user not in persona_hours:
+        return False
+    hours = persona_hours[user]
+    if not hours:
+        return False
+    return record.timestamp.hour not in hours
+
+
+def _is_failed_operation(record: ParsedRecord, fmt: str) -> bool:
+    """Check if event represents a failed operation."""
+    f = record.fields
+    if fmt == "windows_event_security":
+        return f.get("EventID") in _FAILED_EVENT_IDS
+    if fmt == "web_access":
+        code = f.get("status_code")
+        return isinstance(code, int) and code in _FAILED_HTTP_CODES
+    if fmt == "syslog":
+        msg = f.get("message", "").lower()
+        return any(kw in msg for kw in _FAILED_SYSLOG_KEYWORDS)
+    return False
+
+
+def _is_rare_process(
+    record: ParsedRecord, fmt: str, process_freq: Counter,
+) -> bool:
+    """Check if event involves a rarely-seen process/command."""
+    proc = _extract_process_key(record, fmt)
+    if not proc or not process_freq:
+        return False
+
+    # Bottom 5% by frequency = rare
+    total_procs = sum(process_freq.values())
+    if total_procs == 0:
+        return False
+
+    threshold = max(1, total_procs * 0.05 / len(process_freq))
+    return process_freq[proc] <= threshold
+
+
+def _is_unexpected_port(
+    record: ParsedRecord, fmt: str, service_ports: set[int],
+) -> bool:
+    """Check if connection goes to a port not associated with declared services."""
+    if fmt != "zeek_conn" or not service_ports:
+        return False
+
+    resp_port = record.fields.get("id.resp_p")
+    if not isinstance(resp_port, int):
+        return False
+
+    # Common always-expected ports
+    always_ok = {80, 443, 53, 22}
+    if resp_port in always_ok or resp_port in service_ports:
+        return False
+
+    # High ports (ephemeral) are not unexpected
+    if resp_port > 1024:
+        return False
+
+    return True