Streaming anomaly detection for semiconductor equipment telemetry with exactly-once alert output: 100% recall on four labeled fault classes, 1.7 s mean spike detection latency, byte-identical replay.
-8A2BE2){kind=icon}
- Batch anomaly reports on fab tool telemetry arrive hours after the excursion; this pipeline alerts in stream time, seconds to minutes from onset.
- Restarts and redeliveries normally mean duplicate or missing alerts; here every event affects the output exactly once, provably.
- Post-incident analysis needs trustworthy reprocessing; any offset range replays to byte-identical alerts.
Unplanned downtime on a single semiconductor process tool is commonly costed at around $100k per hour, and the excursions that cause it (a stuck pressure reading, a slow calibration drift, RF arcing, a heater loop that stops tracking power) are visible in sensor telemetry well before scrap accumulates. The operational failure is latency: when anomaly detection runs as a nightly batch job, the report describes yesterday's problem. The window in which an engineer could have acted is already closed.
This project is a streaming pipeline that closes that window mechanically rather than heroically. A seeded simulator emits multi-tool, multi-channel telemetry with labeled fault injections. Events flow through a broker abstraction (SQLite log locally, Kafka client for real deployments) into a processor running four sliding-window detectors: robust z-score on first differences (median/MAD, spike and step faults), flatline run-length (stuck sensors), dual-EWMA control bands (slow drift), and rolling Pearson correlation across physically coupled channels (faults invisible to any single channel). The twist is exactly-once output on top of an at-least-once broker: alert batches commit atomically with the offset-plus-state checkpoint in one SQLite transaction, alert ids are deterministic content hashes, and the checkpointed offset acts as a dedupe watermark that drops redelivered events before they can touch warm detector state.
Measured on the committed labeled benchmark (24 injected faults, 115,200 events, seed 42): 100% recall on all four fault classes with 94.7% alert precision; mean detection latency of 1.7 s for spike bursts, 29.4 s for stuck sensors, 194 s for correlation breaks, and 206 s for slow drift (a drift is by definition small early). Sustained throughput is 15,738 events/sec at the default window (18,019 at window 64) with p99 per-event processing latency of 241 microseconds, on 2 shared vCPUs. Replaying the full log and a 60,001-event segment twice each produced byte-identical alert files (sha256 verified, zero duplicate alert ids), and the live run hash equals the replay hash.
Labeled diagram: solid arrows are the data path, annotations mark failure boundaries.
flowchart LR
subgraph SIM["Simulator (seeded, labeled)"]
T1["Tools ETCH-01..CVD-02<br/>temperature, rf_power,<br/>pressure, vibration"]
end
subgraph BROKER["Broker (at-least-once boundary)"]
LOG["Append-only log<br/>LocalBroker: SQLite WAL<br/>KafkaBroker: Redpanda/Kafka"]
end
subgraph PROC["Processor (crash boundary)"]
DET["Detector banks per tool/channel<br/>robust_z, flatline, ewma_drift"]
CORR["corr_break per tool<br/>(temperature+rf_power)"]
CP["Checkpoint: offset + state<br/>dedupe watermark"]
end
subgraph SINK["Alert store (exactly-once boundary)"]
DB["SQLite: alerts + checkpoints<br/>ONE transaction"]
JSONL["canonical alerts.jsonl<br/>sha256-stable"]
end
RCLI["Replay CLI<br/>read_range, fresh state"]
T1 -->|produce| LOG
LOG -->|"consume (position != committed)"| DET
LOG --> CORR
DET --> CP
CORR --> CP
CP -->|atomic commit| DB
DB --> JSONL
LOG -.->|"offset range"| RCLI
RCLI -.->|"byte-identical output"| JSONL
Fault labels flow only to the evaluator, never to the processor.
| Component | Choice | Why here specifically |
|---|---|---|
| Language | Python 3.10+ | Detector math is windowed scalar work; stdlib deque + sqlite3 cover the hot path without a framework |
| Validation | pydantic 2 | Config errors surface at startup with field-level messages, not mid-stream |
| Broker (executed) | SQLite WAL log | Same contract as one Kafka partition (offsets, commits, ranged reads) with zero infra; what all committed numbers ran on |
| Broker (deployment) | kafka-python + Redpanda compose | Same Broker interface; single-partition topics preserve the total order replay relies on |
| State/alerts | SQLite, one file | Alert batch + checkpoint commit in one transaction; the exactly-once mechanism (ADR-002) |
| Numerics | numpy (simulator only) | Vectorized noise generation; detectors are dependency-free on purpose |
| Tests | pytest + pytest-cov | 56 tests, 95% coverage, including crash-recovery and redelivery regressions |
| Lint/CI | ruff + GitHub Actions | CI re-proves replay determinism on every push, not just tests |
Author-built: simulator, broker abstraction and both implementations, all four detectors, checkpoint/watermark processor, replay and evaluation tooling, benchmark harness, SVG chart generator. Library-provided: numpy RNG, pydantic validation, sqlite3, kafka-python client.
git clone https://github.com/panchalvedant13/stream-anomaly-sentinel.git
cd stream-anomaly-sentinel
python3 -m venv venv && . venv/bin/activate
pip install -e .[dev]
# 1. generate the labeled telemetry stream (deterministic, seed 42)
sentinel simulate # 115,200 events, 24 labeled faults
# 2. run the streaming processor
sentinel process # prints alert count + sha256
# 3. score alerts against ground truth
sentinel evaluate # per-fault recall, latency, precision
# 4. prove idempotency: two replays, identical bytes
sentinel replay --out data/replay --name pass1
sentinel replay --out data/replay --name pass2
sha256sum data/replay/pass1.alerts.jsonl data/replay/pass2.alerts.jsonlThe two sha256 lines must be identical, and both must equal the alerts_sha256 printed by step 2. Determinism is guaranteed within a platform (same CPU float behavior and libm); CI therefore proves replay identity on its own runner rather than pinning an absolute hash. Tests: pytest --cov=src/sentinel.
Methodology: drain the full committed benchmark topic (115,200 events, 16 tool/channel keys) through a fresh processor per configuration, single process, LocalBroker on SQLite. Wall time gives sustained events/sec; perf_counter_ns around each parse-detect-buffer step gives per-event latency, with the inline checkpoint commit every 500 events landing in the tail. Environment: 2 vCPU, 4GB shared container. Raw output: benchmark/results/throughput.json. Reproduce with sentinel bench.
Chart: measured throughput (left) and p50/p95/p99 processing latency (right) per window size.
| config | events/sec | p50 (us) | p95 (us) | p99 (us) | max (us) |
|---|---|---|---|---|---|
| window 64 | 18,019 | 17.5 | 106.4 | 252.4 | 43,557 |
| window 128 (default) | 15,738 | 16.3 | 118.2 | 241.2 | 445,625 |
| window 256 | 10,141 | 15.0 | 174.6 | 313.0 | 389,759 |
Honest degradation: throughput drops 44% from window 64 to 256 because MAD recomputation is O(n log n) in window size, and the worst-case single-event stall is the inline SQLite checkpoint commit at up to ~0.45 s; a deployment that cannot tolerate that pause should move the commit off the hot path.
Detection quality on the same run (benchmark/results/detection_quality.json):
| fault type | faults | detected | recall | mean latency (s) | max latency (s) |
|---|---|---|---|---|---|
| spike | 6 | 6 | 1.00 | 1.7 | 6.7 |
| stuck | 6 | 6 | 1.00 | 29.4 | 29.8 |
| corr_break | 6 | 6 | 1.00 | 194.2 | 277.1 |
| drift | 6 | 6 | 1.00 | 205.8 | 250.0 |
Alert precision: flatline 1.00, corr_break 1.00, robust_z 0.93, ewma_drift 0.90; overall 94.7% (54 of 57 alerts matched a labeled fault; the 3 false positives are EWMA baseline re-convergence firing minutes after a large drift cleared, plus one marginal z=5.68 tail crossing).
Full records in docs/adr/:
- ADR-001: robust statistics (median/MAD, EWMA) over learned autoencoder detectors, for this latency and explainability profile.
- ADR-002: SQLite checkpoint and alert store over Redis/Postgres for a single-node agent, with the trigger to move and the Kafka path for scale.
- No multivariate learned models until the robust detectors miss a documented fault class (recall under 90% with tuned thresholds on real telemetry). Today they would add opacity, not recall.
- No Flink/Spark until sustained input exceeds the single-node ceiling (~15k events/sec measured here; a 200-tool fab at 4 channels x 1 Hz is 800 events/sec).
- No multi-partition ordering: topics are single-partition per tool group by design; scale out is more agents, not a shuffle.
- No alert routing/paging integration; the store and JSONL export are the integration surface.
- No online threshold auto-tuning; thresholds are config, changed deliberately and versioned.
- No secrets in code or config files; broker credentials for the Kafka path come from environment variables only (
SENTINEL_*overrides follow the same rule). - Logs are structured JSON and carry aggregates only (counts, offsets, latencies); raw telemetry values never appear in logs, so log shipping cannot leak process data.
- OT/IT separation: the agent is designed to run on the OT side next to the tool data source; only the alert store contents (alert metadata, no raw traces) need to cross to IT, and the SQLite file makes that a one-way file transfer rather than an open port.
- Input validation at every boundary: config through pydantic, events re-parsed and type-checked, poisoned messages quarantined by offset without payload logging.
| Failure | Detection | Behavior | Recovery |
|---|---|---|---|
| Broker outage | consume raises / returns nothing | processor exits current batch cleanly; checkpoint is durable | restart when broker returns; resumes at watermark, no loss, no duplicates |
| Poisoned message | Event parse fails | counted, logged with offset only, skipped; partition never stalls | inspect quarantined offsets via replay --from-offset N |
| Checkpoint corruption | typed CheckpointCorrupt on load |
refuses to start with silent state loss | delete checkpoint row, replay from a known-good offset; deterministic alert ids absorb duplicates |
| Crash mid-batch | uncommitted work lost with its checkpoint (atomic) | at-least-once redelivery on restart | watermark skips already-committed offsets; regenerated alerts dedupe by id (tested) |
| Clock skew / late events | event time (ts) is carried in payload, processing order is offset order |
detectors key on stream time, cooldowns use event ts | a late event replays into the same deterministic alert id; wall clock is never load-bearing |
The replay proof kept passing while the numbers were quietly wrong. On the first full run the processor reported 20,000 events processed from a 19,200-event topic. Nothing crashed; the alert hashes even matched. The local broker was serving consume() from the group's committed offset, and since commits happened every 500 events, the uncommitted tail of every batch was silently redelivered and re-applied to warm detector windows. Fixed by giving brokers a real consumer position separate from the committed offset, Kafka-poll style (08b5074). Chasing the fallout exposed a nastier cousin: after every stuck-sensor fault, the diff window was full of exact zeros, the MAD collapsed as healthy data returned, and the first normal readings scored z > 2000, firing confident false spike alerts 65 seconds after each stuck fault cleared. The fix is a zero-fraction guard: no scoring while more than 35% of the window is exactly zero, because that regime belongs to the flatline detector (79c4339, regression-tested in test_robust_z_no_blowup_after_flatline).
- Kafka integration test job in CI using the committed Redpanda compose file (service containers), so the KafkaBroker path gets executed proof, not just interface parity.
- Async alert export and off-hot-path checkpointing to cut the ~0.45 s worst-case commit stall.
- Per-channel threshold commissioning helper: fit MAD/EWMA bands from a labeled healthy window and emit a config patch.
- Cross-tool fleet detector: the same excursion signature appearing on sibling chambers within a time window is a different, higher-severity event.
- Prometheus metrics endpoint (counters and latency histograms already exist internally).
MIT. This is a portfolio project; the telemetry is simulated and labeled, and every number above is reproducible from the committed code and seed.