Streaming statistics optimized for heavy-tailed distributions. Computes accurate quantiles with bounded memory usage by adapting to your data's shape.
- Adaptive compression - Automatically adjusts t-digest compression algorithm parameters based on skewness, kurtosis, and observation count
- Exact moments - Mean, variance, skewness, and excess kurtosis computed online
- Memory efficient - Uses ~10KB for millions of data points (vs ~8MB for naive approaches)
- Accurate tail quantiles - <1% error on p99/p99.9 even with extreme outliers
- Merge support - Combine statistics from parallel computations
heavy-streams combines:
- T-digest for quantile estimation with bounded memory
- Online moments using numerically stable algorithms
- Adaptive compression that adjusts based on detected distribution shape
When the algorithm detects heavy tails (high kurtosis) or skewness, it automatically adjusts internal parameters to maintain accuracy in the tails. Read the original t-digest paper for more detail and intuition (https://arxiv.org/abs/1902.04023)
TODO...
Generate and view API documentation:
cargo doc --open --no-depsDual-licensed under MIT and Apache 2.0.