Status: accepted Date: 2026-07-31
The pipeline must flag equipment anomalies in sensor telemetry within seconds of onset, on modest hardware, and every alert lands in front of an equipment engineer who will decide whether to interrupt a running tool. Interrupting a tool costs real money; so does ignoring a real excursion. That gives three hard requirements: low per-event latency, explainable alerts, and deterministic behavior under replay.
Candidate approaches considered:
- Sliding-window robust statistics: robust z-score on first differences (median/MAD), EWMA control bands, rolling Pearson correlation.
- Learned models: LSTM or autoencoder reconstruction error per channel, trained on healthy telemetry.
Use robust statistical detectors (option 1) for all four fault classes.
- Explainability is contractual, not cosmetic. "z = 7.1 against a MAD of 0.11 over the last 128 samples" is an argument an equipment engineer can check against the raw trace in under a minute. "Reconstruction error 0.83" is not, and in an OT environment an unexplainable alert quickly becomes an ignored alert.
- The fault classes that dominate downtime here (stuck sensor, spike burst, slow drift, decorrelation of coupled channels) have crisp statistical signatures. Measured on the labeled simulator: 100% recall on all four classes with 94.7% alert precision. A learned model has no headroom to improve recall and plenty of room to hurt precision.
- Median/MAD is robust to exactly the contamination that breaks mean/sigma: a 10-sigma spike inside the window inflates a standard deviation enough to mask the next spike, while the MAD barely moves.
- Latency and footprint: each detector update is O(window) worst case with MAD recomputed every 8 samples; measured p50 processing cost is about 16 microseconds per event on 2 shared vCPUs. No model server, no GPU, no training pipeline, no model registry.
- Determinism: the detectors are pure functions of checkpointable state, which is what makes byte-identical replay possible. Learned inference stacks make bit-level determinism across library upgrades genuinely hard.
- Faults whose signature is neither a jump, a level shift, a flatline, nor a pairwise decorrelation (e.g. subtle multivariate shape changes across five or more channels) will be missed. That is an accepted, documented gap.
- Thresholds are per-deployment configuration and need an SPC-style commissioning pass on real tools.
- Revisit trigger: when a documented fault class shows recall below 90% with tuned thresholds on real telemetry, add a learned detector for that class behind the same Detector interface, and keep the robust detectors as the explainable first line.