Complete Phase 2.4: Enhanced Scenario Schema with timezone tests and test fixes

Phase 2.4 additions:
- parse_work_hours() utility for temporal activity distribution
- Persona model: optional expanded_activities, work_hours_parsed, activity_intensity
- StorylineEvent model: optional event_sequence, duration, retry_on_failure, success_probability
- Validation for expanded_activities and event_sequence structures
- Timezone unit tests (14 tests) and integration tests (4 tests)
- Scenario schema reference documentation (docs/scenario-reference.md)

Bug fix:
- Fix emitter _flush_unlocked() truncating output on every flush when no
  header template exists (mode was always "w" instead of "a")

Test fixes (21 previously failing tests):
- test_engine.py: mock all 7 emitter classes (was only mocking 2 from Phase 1)
- test_format_definitions.py: expect 7 formats instead of 2
- test_emitter_threading.py: fix header test using format without headers
- test_parallel_generation.py: fix timezone-naive datetime comparison

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Author: DavidJBianco

TODO.md

@@ -1,15 +1,16 @@
 # EvidenceForge Implementation Plan
 
-**Status:** Phase 2 - Scalability (In Progress - 2.4-2.9 remaining)
+**Status:** Phase 2 - Scalability (In Progress - 2.5-2.9 remaining)
 **Started:** 2026-03-11
-**Last Updated:** 2026-03-12
+**Last Updated:** 2026-03-12 (Phase 2.4 complete)
 **Target MVP Completion:** 7-10 weeks from start
 
 **Recent Completions:**
 - ✅ Phase 2.1: Parallel Generation with Threaded Emitters
 - ✅ Phase 2.2: 5 New Log Formats (eCAR, syslog, bash_history, snort_alert, web_access)
 - ✅ Phase 2.3: Progress Reporting
 - ✅ Phase 2.10: OS-Aware Activity Generation (Windows + Linux support)
+- ✅ Phase 2.4: Enhanced Scenario Schema (work hours parsing, model expansion, timezone tests, validation, docs)
 
 ---
 
@@ -219,15 +220,45 @@
   - [x] Show ETA based on moving average
 - [x] Test: Progress updates correctly during generation
 
-### 2.4 Enhanced Scenario Schema
-
-- [ ] Expand `models/scenario.py` with full MVP schema
-  - [ ] Timezone configuration (default + per-system patterns)
-  - [ ] Persona structure with expanded_activities (prepare for LLM expansion)
-  - [ ] Storyline with event_sequence (prepare for LLM expansion)
-  - [ ] Complete environment specification
-- [ ] Timezone handling in utilities
-- [ ] Test: Timezone conversions (UTC internal → system timezone output)
+### 2.4 Enhanced Scenario Schema ✅ COMPLETE
+
+**Goal:** Add optional schema fields for LLM expansion (Phase 3.1) and persona generation (Phase 2.6)
+
+- [x] Add `parse_work_hours()` utility to `utils/time.py`
+  - [x] Parse "9am-5pm" format to hour ranges
+  - [x] Support lunch breaks: "9am-5pm (lunch 12pm-1pm)"
+  - [x] Support half-hours: "8:30am-5:30pm"
+  - [x] Calculate peak hours (mid-morning, mid-afternoon)
+  - [x] Add 10+ test cases in `tests/unit/test_time_parsing.py` (17 tests, all passing)
+- [x] Expand Persona model in `models/scenario.py` (optional fields, backward compatible)
+  - [x] Timezone configuration (default + per-system patterns) - Already exists!
+  - [x] Add optional `expanded_activities` field (for Phase 3.1 LLM to populate)
+  - [x] Add optional `work_hours_parsed` field (auto-populated from work_hours)
+  - [x] Add optional `activity_intensity` field (per-activity overrides)
+  - [x] Add `@model_validator` to auto-populate work_hours_parsed
+- [x] Expand StorylineEvent model in `models/scenario.py` (optional fields, backward compatible)
+  - [x] Add optional `event_sequence` field (sub-events for complex attacks)
+  - [x] Add optional `duration` field (event duration like "30m")
+  - [x] Add optional `retry_on_failure` field
+  - [x] Add optional `success_probability` field (0.0-1.0)
+- [x] Add timezone tests
+  - [x] Create `tests/unit/test_timezone_handling.py` (14 tests: pattern matching, DST, conversions)
+  - [x] Create `tests/integration/test_scenario_timezone.py` (4 tests: multi-timezone scenarios)
+  - [x] Test UTC → local timezone conversions
+  - [x] Test pattern-based timezone overrides
+  - [x] Test work hours parsing
+- [x] Update validation in `validation/schema.py`
+  - [x] Validate expanded_activities structure if present
+  - [x] Validate event_sequence structure if present
+  - [x] 6 new validation tests added to `tests/unit/test_validation.py`
+- [x] Documentation
+  - [x] Create `docs/scenario-reference.md` with full schema reference
+  - [x] Document optional fields for Phase 3.1 LLM expansion
+  - [x] Document backward compatibility guarantees
+- [x] Verification
+  - [x] Test backward compatibility with existing scenarios (minimal.yaml, attack.yaml)
+  - [x] Verify work_hours_parsed auto-population works
+  - [x] All 105 Phase 2.4 tests pass
 
 ### 2.5 Network Visibility Architecture
 
@@ -321,8 +352,8 @@
 **Architecture Note:** Native logs (Windows Event Security, syslog) are ALWAYS present per OS type. eCAR is OPTIONAL and may be present on all, some, or no systems (EDR/XDR is not universally deployed).
 
 **Phase 2 Status:**
-- ✅ Complete: 2.1 (Parallel Generation), 2.2 (7 Log Formats), 2.3 (Progress Reporting), 2.10 (OS-Aware Generation)
-- 🚧 In Progress: 2.4-2.9 (Enhanced Schema, Network Visibility, Persona Generation, LLM Integration, Medium Datasets)
+- ✅ Complete: 2.1 (Parallel Generation), 2.2 (7 Log Formats), 2.3 (Progress Reporting), 2.4 (Enhanced Schema), 2.10 (OS-Aware Generation)
+- 🚧 Pending: 2.5-2.9 (Network Visibility, Persona Generation, LLM Integration, Medium Datasets, Phase 2 Testing)
 
 **Phase 2 Milestone (Partial):** Can generate datasets across 7 formats (Windows Event Security, Zeek, eCAR, syslog, bash_history, snort_alert, web_access) in parallel with threaded emitters. Windows and Linux systems generate appropriate OS-specific logs. Native logs (Windows Event/syslog) always present; eCAR optional EDR/XDR layer.
 

docs/scenario-reference.md

@@ -0,0 +1,219 @@
+# Scenario Schema Reference
+
+This document describes the EvidenceForge scenario file schema, including Phase 2.4 enhanced fields.
+
+## Overview
+
+Scenario files are YAML documents that define the environment, users, systems, personas, and storyline for log generation. All fields marked "Phase 2.4+" are optional and backward compatible with Phase 1 scenarios.
+
+## Top-Level Structure
+
+```yaml
+version: "1.0"
+name: scenario-name          # Alphanumeric, dash, underscore
+description: |
+  Multi-line scenario description
+environment: ...
+personas: [...]               # Optional
+time_window: ...
+baseline_activity: ...
+storyline: [...]              # Optional
+output: ...
+```
+
+## Environment
+
+```yaml
+environment:
+  description: "Corporate office network"
+  timezone:
+    default: "America/New_York"
+    systems:                  # Optional pattern-based overrides
+      "EU-*": "Europe/London"
+      "AP-*": "Asia/Tokyo"
+  users: [...]
+  systems: [...]
+  groups: [...]               # Optional
+```
+
+### Timezone Configuration
+
+All internal timestamps are stored in UTC. The timezone configuration controls output formatting.
+
+- **default**: Applied to all systems unless overridden (default: `"UTC"`)
+- **systems**: Pattern-based overrides using fnmatch glob syntax (`*`, `?`, `[seq]`)
+  - First matching pattern wins
+  - Unmatched hostnames use the default
+
+Valid timezone names are any [pytz timezone](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) (e.g., `America/New_York`, `Europe/London`, `Asia/Tokyo`, `UTC`).
+
+### Users
+
+```yaml
+users:
+  - username: jsmith           # Required: alphanumeric, dash, underscore
+    full_name: "Jane Smith"    # Required
+    email: jane@example.com    # Required
+    groups: ["developers"]     # Optional
+    enabled: true              # Optional (default: true)
+    persona: developer         # Optional: reference to persona name
+    primary_system: WS-01      # Optional: reference to system hostname
+```
+
+### Systems
+
+```yaml
+systems:
+  - hostname: WS-01            # Required: RFC 1123 compliant
+    ip: "10.0.1.10"            # Required: IPv4 or IPv6
+    os: "Windows 10"           # Required
+    type: workstation          # Required: workstation|server|domain_controller
+    assigned_user: jsmith      # Optional: reference to username
+    services: ["IIS"]          # Optional
+```
+
+## Personas
+
+Personas define user behavior patterns for activity generation.
+
+```yaml
+personas:
+  - name: developer            # Required: unique identifier
+    description: "Software developer who codes and browses"  # Required
+    typical_activities:        # Optional list of activity strings
+      - coding
+      - web_browsing
+    work_hours: "9am-5pm"     # Optional (default: "9am-5pm")
+    application_usage:         # Optional
+      - vscode
+      - chrome
+    risk_profile: low          # Optional: low|medium|high (default: "medium")
+```
+
+### Work Hours Format
+
+The `work_hours` field supports these formats:
+- `"9am-5pm"` - Basic range
+- `"8:30am-5:30pm"` - Half-hour precision
+- `"9am-5pm (lunch 12pm-1pm)"` - With lunch break
+- `"8:30am-5:30pm (lunch 12:30pm-1:30pm)"` - Both combined
+
+Work hours are automatically parsed into a `work_hours_parsed` dict containing:
+- `start`: Start hour as float (e.g., 9.0, 8.5)
+- `end`: End hour as float (e.g., 17.0, 17.5)
+- `lunch`: Tuple of (start, end) if specified, else null
+- `hours`: List of active integer hours (excluding lunch)
+- `peak_hours`: Mid-morning and mid-afternoon hours
+
+### Phase 2.4+ Optional Fields
+
+These fields are for future LLM expansion (Phase 3.1) and are not required:
+
+```yaml
+personas:
+  - name: developer
+    # ... Phase 1 fields above ...
+
+    expanded_activities:       # Phase 2.4+: LLM-populated activity sequences
+      - activity_type: process_code
+        sequence:
+          - action: open_ide
+            app: VS Code
+          - action: edit_files
+            duration_minutes: 30
+        temporal_pattern: morning_focus
+        frequency: daily
+
+    activity_intensity:        # Phase 2.4+: Per-activity events/hour overrides
+      process_code: 20
+      connection_web: 5
+```
+
+**expanded_activities** items must have:
+- `activity_type` (required): Maps to baseline activity types
+- `sequence` (optional): List of action steps
+- `temporal_pattern` (optional): When this activity typically occurs
+- `frequency` (optional): How often (hourly, daily, weekly)
+
+## Time Window
+
+```yaml
+time_window:
+  start: "2024-01-15T10:00:00Z"  # Required: ISO 8601 UTC
+  end: "2024-01-15T18:00:00Z"    # Either end OR duration required
+  duration: "8h"                   # Supports: "10h", "3d", "2h30m"
+```
+
+## Baseline Activity
+
+```yaml
+baseline_activity:
+  description: "Normal office activity"
+  intensity: medium              # low|medium|high (events/user/hour)
+  variation: low                 # low|medium|high (timing variation)
+```
+
+Intensity mapping: low=5, medium=15, high=40 events/user/hour.
+
+## Storyline
+
+Storyline events define specific actions at specific times.
+
+```yaml
+storyline:
+  - time: "+2h30m"             # Required: ISO 8601, relative offset, or seconds
+    actor: attacker            # Required: username or "attacker"
+    system: WS-01              # Required: system hostname
+    activity: "lateral movement"  # Required: activity description
+    details:                   # Optional: activity-specific details
+      target_ip: "10.0.1.20"
+      method: "pass-the-hash"
+```
+
+### Phase 2.4+ Optional Fields
+
+```yaml
+storyline:
+  - time: "+2h30m"
+    # ... Phase 1 fields above ...
+
+    event_sequence:            # Phase 2.4+: Multi-step sub-events
+      - sub_event_type: process
+        delay_seconds: 5
+        details:
+          process_name: powershell.exe
+      - sub_event_type: file
+        delay_seconds: 10
+        details:
+          file_path: C:\temp\payload.exe
+
+    duration: "30m"            # Phase 2.4+: Event duration
+    success_probability: 0.8   # Phase 2.4+: 0.0-1.0
+    retry_on_failure: true     # Phase 2.4+: Retry flag
+```
+
+**event_sequence** items must have:
+- `sub_event_type` (required): Type of sub-event (e.g., process, file, network)
+- `delay_seconds` (optional): Delay before this sub-event
+- `details` (optional): Sub-event-specific details
+
+## Output
+
+```yaml
+output:
+  logs:
+    - format: windows_event_security
+    - format: zeek_conn
+    - format: ecar
+  destination: ./output
+  compression: false           # Optional (default: false)
+```
+
+Supported formats: `windows_event_security`, `zeek_conn`, `ecar`, `syslog`, `bash_history`, `snort_alert`, `web_access`.
+
+## Backward Compatibility
+
+All Phase 2.4+ fields are optional with null defaults. Existing Phase 1 scenarios work without modification:
+- `expanded_activities`, `work_hours_parsed`, `activity_intensity` default to null
+- `event_sequence`, `duration`, `retry_on_failure`, `success_probability` default to null
+- `work_hours_parsed` is auto-populated from the `work_hours` string if not explicitly provided

src/log_generator/generation/emitters/base.py

@@ -238,9 +238,12 @@ def _flush_unlocked(self) -> None:
         if not self.buffer:
             return
 
-        # Ensure header is written first
+        # Ensure header is written first (or mark as done if no header)
         if not self._header_written:
             self._write_header_unlocked()
+            # Mark header as written even if no header template exists,
+            # so subsequent flushes use append mode instead of truncating
+            self._header_written = True
 
         # Ensure output directory exists
         self.output_path.parent.mkdir(parents=True, exist_ok=True)