Merge pull request #373 from Cisco-Talos/dev

feat: release EvidenceForge 1.14.0

Author: DavidJBianco

CHANGELOG.md

@@ -6,6 +6,34 @@ Detailed development history for the EvidenceForge project. Transferred from TOD
 
 ## Unreleased
 
+## v1.14.0 (2026-08-04)
+
+This minor release adds canonical multi-alert IDS attachments and exact
+cross-source evaluation integrity, then hardens the generator against realism
+findings discovered through repeated blind assessment.
+
+**Correlated IDS evidence**
+
+- Added typed, transport-owned IDS alert attachments with multi-SID support,
+  sensor-local routing, filtering policy, validation, configuration overlays,
+  documentation, and source-native Snort rendering (`2be53773`, `96f5afe1`).
+- Added exact IDS integrity evaluation across ground truth, transport evidence,
+  visibility, sensors, and rendered alerts, including acceptance gates and
+  regression coverage for correlation failures (`e023286b`).
+
+**Cross-source and source-native realism**
+
+- Corrected eCAR session timing and module principal ownership, machine-account
+  application transport semantics, versioned Sysmon manifests, and
+  deterministic leaf-certificate issuance (`3dfbd680`).
+- Replaced exact sudo ceilings and public-client User-Agent monoculture with
+  host/source-sticky distributions, and added sensor-local timing and bounded
+  capture-accounting texture for distributed network observations
+  (`3dfbd680`).
+- Prevented impossible baseline signatures and payload-content alerts on opaque
+  TLS while retaining metadata-visible TLS detections and cleartext payload
+  coverage (`3dfbd680`).
+
 ## v1.13.1 (2026-07-30)
 
 This patch release hardens agent-facing security boundaries and CI dependency

README.md

@@ -25,7 +25,7 @@ Most synthetic log generators produce isolated, single-format data that experien
 
 - **Deterministic engine, LLM-assisted authoring.** Scenario creation uses Claude Code Skills for interactive, research-backed attack planning. Log generation is fully deterministic — no LLM calls, no API costs, reproducible output every time.
 
-- **Built-in quality evaluation.** A 4-pillar scoring framework (20 sub-scores) measures parseability, plausibility, causality, and timing. Know exactly how good your data is before using it.
+- **Built-in quality evaluation.** A 4-pillar scoring framework (21 sub-scores) measures parseability, plausibility, causality, and timing. Know exactly how good your data is before using it.
 
 ## Quick Start
 
@@ -82,7 +82,7 @@ For scripted or non-interactive use:
 |---------|-------------|
 | `eforge generate <scenario.yaml> -o <dir>` | Generate logs from a scenario file |
 | `eforge validate <scenario.yaml>` | Validate scenario schema and cross-references |
-| `eforge eval <output_dir> -s <scenario.yaml>` | Evaluate data quality (4 pillars, 20 sub-scores) |
+| `eforge eval <output_dir> -s <scenario.yaml>` | Evaluate data quality (4 pillars, 21 sub-scores) |
 | `eforge info [field]` | Show installation info, config paths, and data inventories. Pass a dot-path field for a specific value (e.g., `eforge info personas`). Use `--fields` to list available fields, `--json` for machine output. |
 | `eforge validate-config` | Validate config files for cross-reference integrity. Use `--json` for machine output. |
 | `eforge install-skills [--agent all\|claude\|chatgpt\|codex] [--global]` | Install project-local or user-wide agent skills; defaults to all agents (`codex` aliases `chatgpt`) |
@@ -128,7 +128,7 @@ Every generated scenario includes a `GROUND_TRUTH.md` file. Attack scenarios doc
 - **Ground truth documentation** — Every run generates a GROUND_TRUTH.md; attack scenarios include narrative, timeline, and IOCs
 - **Parallel generation** — Threaded emitters write all formats simultaneously with temporal consistency
 - **Scenario validation** — Cross-reference checking, uniqueness constraints, and network topology validation
-- **Data quality evaluation** — 4-pillar scoring framework (20 sub-scores) with acceptance criteria
+- **Data quality evaluation** — 4-pillar scoring framework (21 sub-scores) with acceptance criteria
 - **Multi-timezone support** — Pattern-based timezone overrides per system hostname
 
 ## Supported Log Formats

TODO.md

@@ -1,6 +1,6 @@
 # EvidenceForge Implementation Plan
 
-**Status:** Phase 8.5 (Dual src/dst HostContext) COMPLETE; Pre-MVP quality fixes ongoing
+**Status:** Phase 8.5 (Dual src/dst HostContext) COMPLETE; post-1.0 quality improvements ongoing
 **Started:** 2026-03-11
 **Last Roadmap Review:** 2026-05-26
 
@@ -21,11 +21,12 @@ ground truth documentation.
 **Phase 2: Scalability.** Parallel threaded emitters, 7 log formats, persona
 temporal distribution, network visibility modeling, and multi-OS support.
 
-**Phase 3: MVP Release.** Skill-based scenario/generate/validate/evaluate
+**Phase 3: Initial Product Release.** Skill-based scenario/generate/validate/evaluate
 workflow, prebuilt personas, skill installation, and scenario reference docs.
 
 **Phase 4: Data Quality Evaluation.** `eforge eval` with deterministic scoring,
-source parsers, and acceptance criteria.
+source-instance-aware parsers, inferred narrative pivots, acceptance criteria,
+and exact correlated-IDS integrity gating.
 
 **Phase 5: Data Realism Improvements.** Major generator-level realism fixes for
 identity, protocol, process, temporal, and baseline noise patterns.
@@ -40,7 +41,7 @@ assessment history belongs in worklogs and changelog entries, not this roadmap.
 
 ---
 
-## Pre-MVP Quality Roadmap
+## Quality Roadmap
 
 Current goal: fix analyst-rejection issues and finish remaining quality work
 without turning `TODO.md` back into a high-conflict work journal.
@@ -212,15 +213,24 @@ further per-loop or per-PR details in worklogs or PR descriptions.
 
 ---
 
-## Post-MVP Enhancements
+## Future Enhancements
 
 ### Short-Term
 
 - [ ] Configurable work-week schedules and per-persona day-of-week overrides.
 - [ ] Storyline cadence field: `human`, `automated`, or periodic interval with
   jitter.
 - [ ] Cloud/SaaS log formats: Azure AD, AWS CloudTrail, GCP audit logs, and M365.
-- [ ] `snort_alert` typed event spec for IDS signature declarations.
+- [x] Correlated multi-SID IDS attachments on typed transport-owning events,
+  including connections, beacons, remote sessions, DHCP, scans, and DNS activity,
+  with sensor-local Snort-style alert filtering and reporting.
+- [ ] Extend correlated IDS attachments to typed `email_message` and `email_read`
+  events so asserted SIDs follow the real mail transports produced by modeled
+  routing and sensor placement. IDS sensors do not currently decrypt traffic;
+  before implementation, decide whether STARTTLS and implicit TLS suppress every
+  candidate or permit signatures classified as detectable from flow,
+  pre-encryption, or TLS metadata. Plaintext mail is eligible only when a
+  storyline or background path explicitly asserts a signature.
 - [ ] HTTP proxy server support for Squid, Blue Coat, and Zscaler.
 - [ ] Checkpointing and resume for long-running generation.
 - [ ] Additional skills: create-persona, create-log-format, create-network, and

commands/eforge/config.md

@@ -109,6 +109,7 @@ Also read the relevant reference doc for field schemas and conventions:
 | Evaluation rules | `references/config-evaluation.md` (read-only reference — not user-customizable) |
 | Cross-file dependencies | `references/config-dependency-graph.md` |
 | Validation checks | `references/config-validation.md` |
+| IDS signatures and alert policy | `references/config-ids.md` |
 
 ## Step 4: Interview for Completeness
 
@@ -126,6 +127,13 @@ duplicate names. Keep profile contents synthetic and behavior-shaped: URI shapes
 method/status/byte ranges, User-Agent pools, and deterministic tokens are fine;
 do not encode live malware IOC paths, domains, or exact campaign payloads.
 
+**Changing IDS cadence:** Edit the project overlay at
+`.eforge/config/activity/ids_signatures.yaml`; entries merge by `sid`. Add or
+replace `alert_policy` with `detection_filter`, `event_filter`, or both, or set it
+to `every`. Do not copy the whole packaged signature unless its metadata also
+needs to change. Read `references/config-ids.md`, run `eforge validate-config`,
+and start a fresh CLI process after the edit so cached signature data is reloaded.
+
 **Adding an application:** Which OS(es)? Categories? Which personas? Image path? PE metadata? Command templates? Parent process? Children? Network traffic?
 
 **Creating a persona:** Role description? Typical activities? Work hours (format: "9am-5pm (lunch 12pm-1pm)")? Risk profile (low/medium/high)? Browsing intensity (light/normal/heavy)? Applications? Custom traffic? Linux user?

commands/eforge/evaluate.md

@@ -11,7 +11,7 @@ description: >
 
 # EvidenceForge Data Quality Evaluator
 
-You are helping the user evaluate the quality of generated synthetic security log datasets using EvidenceForge's evaluation framework. The eval command scores datasets across **4 pillars** with 20 sub-scores, all deterministic and statistical. Your job is to run the eval, interpret the results, review sample records for realism, and provide actionable improvement suggestions.
+You are helping the user evaluate the quality of generated synthetic security log datasets using EvidenceForge's evaluation framework. The eval command scores datasets across **4 pillars** with 21 sub-scores, all deterministic and statistical. Your job is to run the eval, interpret the results, review sample records for realism, and provide actionable improvement suggestions.
 
 ## Quick Start
 
@@ -25,6 +25,19 @@ Default to `eforge` for all CLI execution. If `eforge` is not found and you are
 in an EvidenceForge source checkout, retry the same command with
 `uv run eforge ...`.
 
+Canonical IDS reconciliation is automated by the zero-weight
+`plausibility.ids_integrity` hard gate. It compares every per-sensor `(gid, sid)`
+count and ordered normalized digest in `GROUND_TRUTH.json` with parsed Snort
+rows, including the sensor-visible UTC timestamp, signature metadata, complete
+tuple, ephemeral source port, and NAT/PAT projection. It also reconciles policy
+filtering and observation totals and requires every emitted row to have an
+authorized authored, built-in, or raw origin. Legacy datasets without
+`ids_evaluation` skip this check with a warning, but a scenario containing
+authored `ids_alerts` fails when the summary is missing or invalid.
+For DHCP, count only the authored transaction, not later automatic renewals; for
+DNS tunnel activity, exclude generated background cover queries. For web scans,
+account for automatic and authored SIDs and the authored-wins duplicate rule.
+
 If they don't have generated output yet, suggest using `/eforge generate` first.
 
 For detailed field documentation and known limitations of each log format, use the `/eforge:references:evidence-formats` skill.
@@ -133,12 +146,13 @@ For each pillar, explain what the score means in practical terms:
 - Cross-Source Field Agreement: When the same event appears in multiple log sources, do shared fields agree? Uses pivot-key joins defined in `cross_source_pairs.yaml` plus built-in email checks — pairs include Windows 4688 ↔ eCAR PROCESS/CREATE (same PID+host → same process name), zeek_conn ↔ Cisco ASA (same 4-tuple), web_access/proxy ↔ zeek_http (same client+URI+10s bucket → same status/method), zeek_ssl ↔ zeek_x509 (cert chain fuids → server_name ∈ SAN), and email checks where SMTP UIDs join to conn.log, visible SMTP FUIDs join to files.log, and plaintext SMTP subject metadata agrees with `ARTIFACTS_MANIFEST.json` email records. A score below 100 means real field disagreements were found.
 - User Behavioral Diversity: Do different users behave differently, or are they cookie-cutter clones?
 - Benign Anomaly Rate: Is there a realistic 1–5% rate of anomalous-but-benign events? Zero anomalies is as implausible as 50%.
+- IDS Correlation Integrity: Do sensor-local Snort rows exactly match canonical counts, ordered digests, origins, and observation totals? This is a 100% hard gate with zero scoring weight.
 
 **Pillar 3: Causality (weight 0.25)**
 - Causal Ordering: Are logon→process→logoff and lock→reauth→unlock sequences correctly ordered? DNS before TCP? Kerberos/DC TGT/TGS before domain logons? NTLM/DC validation and Windows audit/process-access companions after their owning evidence?
 - Storyline Event Presence: Are all expected-visible storyline events visible in at least one log source? For non-`complete` observation profiles with a manifest, source rows marked `dropped`, `filtered`, or `out_of_window` are excluded from this coverage denominator.
 - Indicator Accuracy: Do traces carry the correct IPs, usernames, hostnames from the scenario?
-- Pivot Linkability: Can a hunter pivot between consecutive expected-visible attack steps using shared field values?
+- Pivot Linkability: Can a hunter pivot along inferred narrative edges built from shared typed indicators such as hosts, IPs, accounts, domains, URLs, files/hashes, and artifact/message IDs? Unrelated interleaved steps are not connected, generic ports/protocols are excluded, and isolated events are reported separately.
 - Storyline Temporal Integrity: Are expected-visible attack events in the right relative order at the right times?
 - Storyline Trace Coverage: For each expected-visible log format group on each involved host, does the storyline leave a trace?
 

commands/eforge/generate.md

@@ -25,6 +25,27 @@ Default to `eforge` for all CLI execution. If `eforge` is not found and you are
 in an EvidenceForge source checkout, retry the same command with
 `uv run eforge ...`.
 
+When a scenario uses canonical `ids_alerts`, do not infer missing Snort rows from
+the authored attachment count alone. Check `GROUND_TRUTH.json`/`.md` for each
+SID's effective policy and candidate/emitted/policy-filtered sensor totals, then
+check `OBSERVATION_MANIFEST.json` for collection drops, clipping, and policy
+`filtered` status. No IDS sensor, an invisible proxy leg, a denied/cache-hit
+origin request, observation missingness, or policy cadence can all legitimately
+reduce output. Candidate spooling is disk-backed and removed on success or
+failure; a leftover EvidenceForge IDS spool indicates an interrupted process and
+may be removed once no generation process is using it.
+`GROUND_TRUTH.json` also includes the bounded `ids_evaluation` acceptance
+contract: per-sensor/SID counts, origin totals, observation totals, and an
+ordered normalized alert digest. `GROUND_TRUTH.md` renders the same information
+in its IDS Evaluation Summary. Generation accumulates this summary while Snort
+candidates finalize and does not retain alert-volume-sized state.
+
+Interpret candidates by owned transport: SSH/RDP use only their session
+connection; DHCP uses only the authored transaction and does not pass assertions
+to automatic renewals; scans and DNS families fan out across authored
+probes/requests/queries. DNS-tunnel cover traffic is not attached. IDS sensors
+do not decrypt traffic, and email attachments remain unsupported.
+
 If they don't have a scenario file yet, suggest using `/eforge scenario` to create one first.
 
 ## Command Reference

commands/eforge/references/config-evaluation.md

@@ -30,6 +30,14 @@ correctness gates such as parseability, value plausibility, field agreement, and
 ordering remain strict. Adjusted sub-scores expose `raw_score` in JSON and show `raw:<score>` in
 the text report.
 
+`plausibility.ids_integrity` is a special zero-weight 100% hard gate. It validates
+the bounded `GROUND_TRUTH.json.ids_evaluation` contract against parsed,
+sensor-provenanced Snort rows and `OBSERVATION_MANIFEST.json`. Its zero weight
+keeps the overall numeric score unchanged while any count, digest, origin, tuple,
+timestamp, filtering, or observation contradiction fails acceptance. Legacy
+ground truth without the section skips the gate unless the supplied scenario
+authors `ids_alerts`, in which case absence is a failure.
+
 ### Structure
 
 ```yaml
@@ -87,6 +95,7 @@ pillars:
 | plausibility | `field_agreement` | no |
 | plausibility | `user_diversity` | no |
 | plausibility | `anomaly_rate` | no |
+| plausibility | `ids_integrity` | yes (100%; zero weight) |
 | causality | `causal_ordering` | yes |
 | causality | `event_presence` | yes |
 | causality | `indicator_accuracy` | no |

commands/eforge/references/config-ids.md

@@ -0,0 +1,69 @@
+# IDS Signature Configuration
+
+## File and overlay
+
+Package defaults live in `activity/ids_signatures.yaml`. Project changes belong
+in `.eforge/config/activity/ids_signatures.yaml`. Entries merge by `sid`, so an
+overlay can add a signature or replace selected fields, including its default
+alert policy. Reset/reload config state (or start a new CLI process) after edits,
+then run `eforge validate-config`.
+
+```yaml
+signatures:
+  - sid: 2002910
+    alert_policy:
+      event_filter:
+        type: both
+        track: by_src
+        count: 5
+        seconds: 60
+```
+
+Required signature fields remain `sid`, `rev`, `message`, `classification`,
+`priority`, and `proto`; the existing `dst_port`, `direction`, DNS templates,
+target service/OS, and baseline eligibility fields keep their current meanings.
+
+## `alert_policy`
+
+`alert_policy` is optional. Omission and `every` both admit every sensor-visible
+candidate. A policy object supports:
+
+```yaml
+alert_policy:
+  detection_filter:
+    track: by_src       # by_src | by_dst
+    count: 5            # positive integer
+    seconds: 60         # positive integer; half-open window
+  event_filter:
+    type: limit         # limit | threshold | both
+    track: by_src
+    count: 1
+    seconds: 300
+```
+
+`detection_filter` suppresses the first `count` rule matches and admits later
+matches while the rolling window stays above that threshold. `event_filter` is
+then applied to admitted matches: `limit` emits the first `count` per window,
+`threshold` emits every `count`th match and resets, and `both` emits once when the
+count is reached and suppresses until expiry. A timestamp exactly `seconds`
+after the window start begins a new window.
+
+Scenario attachment policies replace, rather than merge with, this default.
+Use `policy: every` on an attachment to explicitly bypass a signature default.
+EvidenceForge models alert output only: it does not parse complete Snort rules,
+apply `rate_filter`, CIDR suppressions, or IPS actions.
+
+Signature defaults are inherited by attachments on typed `connection`,
+`beacon`, `ssh_session`, `rdp_session`, `dhcp_lease`, `port_scan`, `web_scan`,
+`dns_query`, `dga_queries`, and `dns_tunnel` events. Defaults never make an
+unattached tuple alert. The IDS model does not decrypt traffic, and mail-event
+attachments remain deferred.
+
+## Validation
+
+`eforge validate-config` rejects empty policies, unknown keys, invalid
+filter/type/track values, booleans/fractions/non-positive values, and integers
+above 2,147,483,647. `eforge validate <scenario>` rejects unknown or duplicate
+attachment SIDs and conflicting effective policies for one SID. Protocol, port,
+and direction mismatches are advisory warnings because nonstandard deployments
+can be intentional.