What are the rules on which zap's sampling deduplication is based?

[ZigHuang]

I tried reading this part of the code, but didn't find it

[mway]

Zap's sampling logic is defined in zapcore/sampler.go as part of the sampler core. At a high level, the way that sampling works is that - for a given log message and its level - determine how many times we've seen that message at that log level within a given period, and evaluate whether we should drop the log based on the sampling config.

The way that it's implemented is:

1. There are stateful counters per sampler core. The counters establish a number of per-level buckets - currently 4096 buckets for each log level - that we use for tracking periodic state.

2. zap.SamplingConfig defines two key fields, Initial and Thereafter:

type SamplingConfig struct {
  Initial    int // ...
  Thereafter int // ...
        // ...
}

Initial defines how many of the same level:msg combinations we allow before we start dropping logs. Thereafter defines the ratio of sampling - i.e., 1 of every Thereafter logs will be sampled (kept), and the rest will be dropped.

3. When a log message is received, we use the modulus of a hash of the log message to determine which counter bucket to use. (Note that if you have highly dynamic log messages, it's entirely possible for there to be hash collisions.) As a simple illustration:

logger.Info("foo")  // --> info buckets ->  "foo" hashes to info bucket 123
logger.Info("bar")  // --> info buckets ->  "bar" hashes to info bucket 456
logger.Warn("oops") // --> warn buckets -> "oops" hashes to warn bucket 123

Thus no log level can be conflated with the other log levels, and messages within a given level can only be conflated if they are hashed into the same bucket for that level.

4. The leveled log message in question is evaluated against the matched counter based on the SamplingConfig discussed earlier. If we haven't seen this hash modulus at this level more than Initial times within the current sampling period, we sample (keep) the log; otherwise, we drop it unless this is a Thereafterth time we're seeing this level:message since we reached Initial.

The terminology can be a bit confusing, but hopefully that helps clear things up!