Skip to content

Commit 67cae86

Browse files
authored
ops(e2e): enable IS-060 datamarking in shadow mode for nightly validation (#216)
Stage 1 of the IS-060 PR-2 validation chain. The datamarking transform landed in PR #214 with `enabled: false` defaults — this PR turns it ON in shadow mode for the nightly e2e harness (config-e2e-judge.yaml, the canonical proxy config for both PR-gate and scheduled nightly). Configuration applied: - boundary_defense.enabled: true (parent — datamarking depends on zone wrapping) - boundary_defense.randomize_nonce: true (defense in depth) - boundary_defense.datamarking.enabled: true - boundary_defense.datamarking.shadow_mode: true (compute+metrics, forward original) - boundary_defense.datamarking.marker_strategy: randomized Shadow mode means: the transform runs, emits the four `llmtrace_spotlighting_*_total` counters, and emits the audit-trail `spotlighting_applied` finding — but the bytes forwarded upstream are unchanged. This isolates the "transform runs without breaking the proxy / upstream API" signal from the "transform changes the model's behaviour" signal. Acceptance criteria for this stage (validated post-merge): - Existing E2E PR Gate + E2E Tests stay green (no regression with shadow mode active). - The nightly's `Run full corpus` step completes without 4xx-rate shift vs prior nightly (`e2e_2026-05-15.json` is the comparison baseline). - `llmtrace_spotlighting_zones_total{shadow=\"true\"}` > 0 in the run. - `llmtrace_spotlighting_byte_delta_total{shadow=\"true\"}` > 0. - `llmtrace_spotlighting_failures_total` == 0. If all four hold after one nightly cycle, a follow-up ops PR flips shadow_mode to false (active mode) and begins the 3-5 night ASR delta collection per IS-060 PR-2's acceptance criteria. Refs: #214 (PR-2 implementation), #213 (PR-3 corpus expansion), docs/architecture/SPOTLIGHTING_INDIRECT_INJECTION.md §6.2 + §6.4 (evidence requirements + baseline).
1 parent 0069991 commit 67cae86

1 file changed

Lines changed: 19 additions & 0 deletions

File tree

tests/e2e/fixtures/config-e2e-judge.yaml

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,25 @@ health_check:
5454
timeout_ms: 5000
5555
retries: 3
5656

57+
# IS-060 — Boundary defense + datamarking transform.
58+
# Enabled in shadow mode for the first validation cycle: data zones are
59+
# wrapped with <llmtrace-boundary>...</llmtrace-boundary> tags AND the
60+
# datamarking pipeline computes the U+E000 substitution + emits metrics,
61+
# but shadow_mode=true means the original (unmarked) bytes still go
62+
# upstream. Validate one nightly cycle for zero 4xx delta and non-zero
63+
# byte_delta_total before flipping shadow_mode to false in a follow-up
64+
# ops PR. See PR #214 + docs/architecture/SPOTLIGHTING_INDIRECT_INJECTION.md.
65+
boundary_defense:
66+
enabled: true
67+
randomize_nonce: true
68+
inject_system_reminder: true
69+
datamarking:
70+
enabled: true
71+
shadow_mode: true
72+
marker_strategy:
73+
kind: randomized
74+
75+
5776
# See config-e2e.yaml for rationale on enforcement mode/min_severity.
5877
# analysis_depth=full so the pre-request enforcement decision sees BOTH
5978
# regex AND ML findings. With `fast` (regex only) the proxy_outcome

0 commit comments

Comments
 (0)