fix: harden cross-source log realism

Author: DavidJBianco

docs/worklog/2026-08-12-iteration-test-loops-60-69.md

@@ -0,0 +1,98 @@
+# Iteration-Test Assessment Loops 60–69
+
+## Scope
+
+Completed ten counted `eforge-assess` loops (60 through 69) against
+`scenarios/iteration-test/scenario.yaml`. Each loop used deterministic scenario/config validation,
+the default non-slow regression suite before generation, forced generation into a new immutable
+loop directory, quantitative evaluation, targeted hard probes, four fresh isolated blind experts,
+and a family-level fix selected from the blind findings. Loop 68 and loop 69 exceeded the mandatory
+disagreement/spread threshold and received fresh report-only deliberations.
+
+Blind agents were restricted to each loop's `output/data` plus the expert briefing/persona
+references. They could not read the scenario, ground truth, evaluation, probes, source, parent
+directories, or prior reports. Dataset integrity manifests confirm the reviewed output did not
+change during each panel.
+
+## Results
+
+| Loop | Records | Eval | Threat / Detection / Network / Host | Initial avg | Deliberation |
+|---:|---:|---:|---|---:|---:|
+| 60 | 80,096 | 97.783 | 66 / 66 / 74 / 94 | 75.00 | — |
+| 61 | 82,379 | 97.086 | 68 / 89 / 82 / 95 | 83.50 | — |
+| 62 | 82,690 | 97.241 | 77 / 86 / 85 / 96 | 86.00 | — |
+| 63 | 82,690 | 97.241 | 77 / 86 / 85 / 96 | 86.00 | — |
+| 64 | 82,676 | 97.241 | 77 / 86 / 73 / 96 | 83.00 | — |
+| 65 | 82,817 | 97.261 | 90 / 86 / 73 / 95 | 86.00 | — |
+| 66 | 80,763 | 97.581 | 74 / 82 / 74 / 91 | 80.25 | — |
+| 67 | 81,573 | 97.403 | 85 / 93 / 71 / 98 | 86.75 | — |
+| 68 | 78,872 | 97.006 | 92 / 70 / 24 / 99 | 71.25 | 91.50 |
+| 69 | 77,567 | 97.291 | 86 / 88 / 28 / 74 | 69.00 | 82.50 |
+
+Loop 69's initial five-loop rolling mean was 78.65. The falling initial mean in loops 68–69 did
+not indicate broad regression: network specialists increasingly judged that subsystem Real, while
+host specialists found sharper endpoint identity contradictions. Deliberation explicitly resolved
+this as scope weighting.
+
+## Family-Level Changes
+
+- **Loop 60:** repaired Windows background-process lifecycle ownership, including durable
+  `taskhostw.exe` identity and dependent-aware termination.
+- **Loop 61:** made scenario IP ownership the canonical source for internal PTR identity and private
+  reverse-zone authority.
+- **Loop 62:** made retained historical process lifetimes participate in host-local PID reservation.
+- **Loop 63:** added data-driven TLS SNI predicates so domain-specific IDS rules attach only to an
+  exact eligible flow.
+- **Loop 64:** made the Linux sudo bundle own the visible `/usr/bin/sudo` lifecycle and PAM PID.
+- **Loop 65:** attached sudo to a live user shell/session, modeled its elevated child, and terminated
+  child then sudo after PAM close.
+- **Loop 66:** allowed out-of-order baseline planning to deterministically bootstrap/reuse the sudo
+  owner session.
+- **Loop 67:** leased one stable TTY/session shell per host/user/TTY, serialized foreground sudo
+  commands, made resolver recovery stateful, and constrained image loads to process lifetime.
+- **Loop 68:** reused historical Linux sessions, unified PAM/eCAR login PID ownership, leased TTYs
+  exclusively, restricted `wsqmcons.exe` to rare workstation-only activity, and improved multipart
+  curl ownership metadata.
+- **Loop 69 post-review:** resolved Security authentication/Kerberos provider PIDs through the
+  host's canonical LSASS identity; registered pre-window sudo sessions as carried state instead of
+  visible boundary logins; reused the PAM-owned local-login process for shell ancestry; and made
+  child creates a termination floor for the eCAR parent process.
+
+## Final Loop Findings
+
+The loop-69 hard probes verified zero duplicate eCAR event IDs, exclusive TTY ownership across 67
+successful sudo chains, no module-after-termination rows, and workstation-only `wsqmcons.exe`.
+Twenty-seven of 31 successful PAM local-login records matched the exact eCAR `/bin/login` PID,
+substantially improving loop 68. The blind panel then isolated the remaining two owner seams:
+duplicate login processes/boundary session initialization and literal PID 600 across selected
+Windows Security authentication families. It also confirmed one 28 ms eCAR
+parent-termination-before-child inversion.
+
+Network evidence is the strongest subsystem. The loop-69 network reviewer found coherent
+sensor-local identities for 1,866 shared flows, fully contained protocol/file lifecycles, exact
+loss-aware proxy byte reconciliation, realistic DHCP T/2 renewals, and complete ASA connection/NAT
+lifecycle behavior. The initial network synthetic-confidence score was 28.
+
+## Verification
+
+- Scenario validation: valid with only expected advisory warnings.
+- Config validation: 89 configuration files valid, zero errors.
+- Every generated loop: automated evaluation PASS.
+- Loop-69 quantitative evaluation: 97.29138173504755 over 77,567 records.
+- Focused post-review regression tests: passed.
+- Ruff lint and format checks: passed.
+- Full post-review regression suite: 5,516 passed, 20 skipped in 359.16 seconds; no generated loop
+  was mutated after its blind review began.
+
+## Remaining Work
+
+1. Regenerate to verify the post-loop-69 provider-PID and carried-session fixes against output-level
+   probes and a new blind panel.
+2. Add coherent source-process visibility/drop semantics for RDP and PsExec initiating endpoints.
+3. Expand public scanner actor and TCP-fingerprint populations with a long one-off tail.
+4. Correct the isolated explicit-credential source-context leak and addressless KDC 4771 request.
+5. Continue reducing local-console density on Linux server roles.
+
+Artifacts are under `scenarios/iteration-test/blind-test/loop-60` through `loop-69`; the aggregate
+report is `scenarios/iteration-test/blind-test/REPORT.md`, and the trend visualization is
+`scenarios/iteration-test/blind-test/assessment-effectiveness-dashboard-last-20-loops.svg`.

src/evidenceforge/config/activity/edr_pools.yaml

@@ -11,6 +11,13 @@
 # {hex} = 8-character hexadecimal token, {process_prefetch_name} = owning
 # process basename uppercased for Windows Prefetch artifacts.
 
+# Additional application/service principals that must never receive generic interactive-user
+# file churn. Core operating-system accounts remain protected in code for compatibility.
+linux_service_users:
+  - "dovecot"
+  - "meridian-app"
+  - "postfix"
+
 group_policy_extension_guids:
   - "35378EAC-683F-11D2-A89A-00C04FBBCFA2"
   - "42B5FAAE-6536-11D2-AE5A-0000F87571E3"
@@ -133,7 +140,7 @@ file_side_effect_profiles:
 
   - name: linux_dbus_state
     executables: ["dbus-daemon"]
-    actions: ["modify"]
+    actions: ["read"]
     paths_linux:
       - "/var/lib/dbus/machine-id"
       - "/run/dbus/system_bus_socket"

src/evidenceforge/config/activity/extra_syslog_messages.yaml

@@ -479,6 +479,7 @@ programs:
   - app: systemd-resolved
     distro: ubuntu
     weight: 1
+    max_per_host_window: 4
     params:
       feature_set:
         - UDP
@@ -501,10 +502,8 @@ programs:
         - "1232"
         - "1472"
     messages:
-      - "Using degraded feature set {degraded_feature_set} instead of UDP+EDNS0 for DNS server {dns_server} after transaction {0}."
-      - "Grace period over, resuming full feature set UDP+EDNS0 for DNS server {dns_server} after probe {0}."
-      - "Flushed {cache_bucket} cache scope {scope} after DNS server {dns_server} changed features."
-      - "Transaction {0} switched to {feature_set} with advertised UDP packet size {packet_size}."
+      - "Using degraded feature set {degraded_feature_set} instead of UDP+EDNS0 for DNS server {dns_server}."
+      - "Grace period over, resuming full feature set (UDP+EDNS0) for DNS server {dns_server}."
 
   - app: thermald
     system_types: [workstation]

src/evidenceforge/config/activity/ids_signatures.yaml

@@ -93,13 +93,13 @@ signatures:
       application_protocol: tls
       inspection: metadata
       semantic_claim: response_content
-  - {sid: 2016360, rev: 1, message: "ET INFO Observed Discord Domain (discordapp.com)", classification: "misc-activity", priority: 3, proto: tcp, dst_port: 443, direction: out}
-  - {sid: 2023882, rev: 2, message: "ET INFO Observed Telegram Domain (t.me)", classification: "misc-activity", priority: 3, proto: tcp, dst_port: 443, direction: out}
-  - {sid: 2025712, rev: 2, message: "ET INFO External IP Lookup Domain (ipify.org)", classification: "misc-activity", priority: 2, proto: tcp, dst_port: 443, direction: out}
-  - {sid: 2024897, rev: 3, message: "ET INFO External IP Lookup (ipinfo.io)", classification: "misc-activity", priority: 2, proto: tcp, dst_port: 443, direction: out}
+  - {sid: 2016360, rev: 1, message: "ET INFO Observed Discord Domain (discordapp.com)", classification: "misc-activity", priority: 3, proto: tcp, dst_port: 443, direction: out, predicate: {phase: established, application_protocol: tls, tls_server_names: [discordapp.com, "*.discordapp.com"]}}
+  - {sid: 2023882, rev: 2, message: "ET INFO Observed Telegram Domain (t.me)", classification: "misc-activity", priority: 3, proto: tcp, dst_port: 443, direction: out, predicate: {phase: established, application_protocol: tls, tls_server_names: [t.me, "*.t.me"]}}
+  - {sid: 2025712, rev: 2, message: "ET INFO External IP Lookup Domain (ipify.org)", classification: "misc-activity", priority: 2, proto: tcp, dst_port: 443, direction: out, predicate: {phase: established, application_protocol: tls, tls_server_names: [ipify.org, "*.ipify.org"]}}
+  - {sid: 2024897, rev: 3, message: "ET INFO External IP Lookup (ipinfo.io)", classification: "misc-activity", priority: 2, proto: tcp, dst_port: 443, direction: out, predicate: {phase: established, application_protocol: tls, tls_server_names: [ipinfo.io, "*.ipinfo.io"]}}
   - {sid: 2028401, rev: 1, message: "ET JA3 Hash - Possible Malware - Various RAT", classification: "potentially-bad-traffic", priority: 1, proto: tcp, dst_port: 443, direction: out}
-  - {sid: 2024290, rev: 2, message: "ET INFO Observed AWS S3 Domain (s3.amazonaws.com)", classification: "misc-activity", priority: 3, proto: tcp, dst_port: 443, direction: out}
-  - {sid: 2025991, rev: 2, message: "ET INFO Observed Cloudflare DNS Over HTTPS Domain (cloudflare-dns.com)", classification: "misc-activity", priority: 3, proto: tcp, dst_port: 443, direction: out}
+  - {sid: 2024290, rev: 2, message: "ET INFO Observed AWS S3 Domain (s3.amazonaws.com)", classification: "misc-activity", priority: 3, proto: tcp, dst_port: 443, direction: out, predicate: {phase: established, application_protocol: tls, tls_server_names: [s3.amazonaws.com, "*.s3.amazonaws.com"]}}
+  - {sid: 2025991, rev: 2, message: "ET INFO Observed Cloudflare DNS Over HTTPS Domain (cloudflare-dns.com)", classification: "misc-activity", priority: 3, proto: tcp, dst_port: 443, direction: out, predicate: {phase: established, application_protocol: tls, tls_server_names: [cloudflare-dns.com, "*.cloudflare-dns.com"]}}
   - {sid: 2023672, rev: 2, message: "ET INFO Python-urllib Outbound Request", classification: "misc-activity", priority: 3, proto: tcp, dst_port: 443, direction: out}
   - {sid: 2022476, rev: 1, message: "ET INFO LibreSSL Outbound Connection", classification: "misc-activity", priority: 3, proto: tcp, dst_port: 443, direction: out}
 

src/evidenceforge/config/activity/system_processes.yaml

@@ -57,7 +57,10 @@ scheduled_tasks:
       - "cleanmgr.exe /autoclean /d C:"
     parent: svchost_local_system
   - image: "C:\\Windows\\System32\\wsqmcons.exe"
-    weight: 4
+    system_types: [workstation]
+    weight: 1
+    max_per_host_window: 1
+    cooldown_hours: 24
     command_templates:
       - "wsqmcons.exe"
     parent: svchost_local_system

src/evidenceforge/config/schemas.py

@@ -61,6 +61,7 @@ class IdsSignaturePredicateSpec(BaseModel, extra="forbid", frozen=True):
     http_methods: list[str] = Field(default_factory=list)
     http_statuses: list[int] = Field(default_factory=list)
     requires_http_body: bool = False
+    tls_server_names: list[str] = Field(default_factory=list)
     file_mime_types: list[str] = Field(default_factory=list)
     semantic_claim: Literal[
         "flow_metadata",
@@ -109,6 +110,19 @@ def normalize_file_mime_types(cls, values: list[str]) -> list[str]:
             raise ValueError("file_mime_types must not contain duplicates")
         return normalized
 
+    @field_validator("tls_server_names")
+    @classmethod
+    def normalize_tls_server_names(cls, values: list[str]) -> list[str]:
+        """Normalize exact or suffix-wildcard TLS server-name requirements."""
+        normalized = [value.strip().lower().rstrip(".") for value in values]
+        if any(
+            not value or (value.startswith("*.") and value.count("*") > 1) for value in normalized
+        ):
+            raise ValueError("tls_server_names must contain non-empty exact or *.suffix names")
+        if len(normalized) != len(set(normalized)):
+            raise ValueError("tls_server_names must not contain duplicates")
+        return normalized
+
     @model_validator(mode="after")
     def validate_semantic_combination(self) -> Self:
         """Reject predicates that cannot be evaluated coherently."""
@@ -121,6 +135,8 @@ def validate_semantic_combination(self) -> Self:
             raise ValueError("HTTP-specific fields require application_protocol='http'")
         if self.requires_http_body and self.payload_direction not in {"orig", "either"}:
             raise ValueError("requires_http_body needs orig/either payload_direction")
+        if self.tls_server_names and self.application_protocol != "tls":
+            raise ValueError("tls_server_names requires application_protocol='tls'")
         if self.file_mime_types and self.semantic_claim != "file_content":
             raise ValueError("file_mime_types requires semantic_claim='file_content'")
         if self.phase == "response" and not self.requires_response:

src/evidenceforge/events/network.py

@@ -72,6 +72,7 @@ class SignaturePredicate:
     http_methods: tuple[str, ...] = ()
     http_statuses: tuple[int, ...] = ()
     requires_http_body: bool = False
+    tls_server_names: tuple[str, ...] = ()
     file_mime_types: tuple[str, ...] = ()
     semantic_claim: SemanticClaim = "flow_metadata"
 
@@ -92,6 +93,8 @@ def __post_init__(self) -> None:
             raise ValueError("HTTP-specific IDS predicates require application_protocol='http'")
         if self.requires_http_body and self.payload_direction not in {"orig", "either"}:
             raise ValueError("HTTP request bodies require orig/either payload direction")
+        if self.tls_server_names and self.application_protocol != "tls":
+            raise ValueError("TLS server-name requirements need application_protocol='tls'")
         if self.file_mime_types and self.semantic_claim != "file_content":
             raise ValueError("IDS file MIME requirements need semantic_claim='file_content'")
 

src/evidenceforge/generation/actions/ids_alert.py

@@ -224,6 +224,7 @@ def _predicate_from_signature(
         http_methods=tuple(spec.http_methods),
         http_statuses=tuple(spec.http_statuses),
         requires_http_body=spec.requires_http_body,
+        tls_server_names=tuple(spec.tls_server_names),
         file_mime_types=tuple(spec.file_mime_types),
         semantic_claim=spec.semantic_claim,
     )
@@ -301,6 +302,14 @@ def ids_alert_matches_transaction(
         return False
     if predicate.requires_http_body and (http is None or int(http.request_body_len or 0) <= 0):
         return False
+    if predicate.tls_server_names:
+        server_name = str(getattr(ssl, "server_name", "") or "").lower().rstrip(".")
+        if not any(
+            server_name == expected
+            or (expected.startswith("*.") and server_name.endswith(expected[1:]))
+            for expected in predicate.tls_server_names
+        ):
+            return False
     if predicate.semantic_claim == "dns_query" and dns is None:
         return False
     if predicate.semantic_claim == "dns_response" and (dns is None or not has_response):