|
| 1 | +# Signature Observability and Throttling |
| 2 | + |
| 3 | +## Overview |
| 4 | + |
| 5 | +Every signature the node produces or checks goes through two services: the identity |
| 6 | +provider (which resolves *signers*) and the deserializer (which resolves *verifiers*). |
| 7 | +Before this feature they were silent — a caller hammering `GetSigner`, or feeding a |
| 8 | +stream of forged signatures to `OwnerVerifier`, looked exactly like healthy traffic, and |
| 9 | +the only evidence was CPU time. |
| 10 | + |
| 11 | +Panurus now: |
| 12 | + |
| 13 | +1. **Instruments** both services. Every operation produces one event — what was |
| 14 | + done, for which principal, with what outcome, and how long it took. |
| 15 | +2. **Exports** those events as Prometheus metrics and as a privacy-safe audit log. |
| 16 | +3. **Escalates** automatically: a principal whose request rate, error ratio, or |
| 17 | + invalid-signature ratio crosses a threshold is moved to a reduced quota and, if it |
| 18 | + keeps going, blocked for a while. |
| 19 | + |
| 20 | +Instrumentation is always on and costs a no-op call when nothing is wired to it. The |
| 21 | +throttle policy defaults to **monitor** — it evaluates and reports but never denies — |
| 22 | +so enabling enforcement is a deliberate decision. |
| 23 | + |
| 24 | +## Principals |
| 25 | + |
| 26 | +Every event is attributed to a **principal**: the identity hash, |
| 27 | +`driver.Identity.UniqueID()`, of the identity the operation was performed for. |
| 28 | + |
| 29 | +Raw identity bytes are never used as a metric label, never logged, and never used as a |
| 30 | +throttle key. The hash is stable, it is already the signer cache key, and it keeps |
| 31 | +identity material (which for X.509 identities includes the certificate) out of logs and |
| 32 | +out of a metrics database. |
| 33 | + |
| 34 | +Operations that cannot be attributed to a single identity — `AreMe` over a batch, for |
| 35 | +instance — are reported with an empty principal and are never throttled. Charging a |
| 36 | +batch to one of its members would let unrelated callers throttle each other. |
| 37 | + |
| 38 | +## Where the gate applies |
| 39 | + |
| 40 | +**Instrumentation is installed everywhere. The gate is consulted at exactly one place: |
| 41 | +`token.SignatureService`, the client-facing entry point.** |
| 42 | + |
| 43 | +This is a deliberate asymmetry. `GetOwnerVerifier` and friends are also called by driver |
| 44 | +*validators*, while validating a transaction. Denying a verifier resolution there would |
| 45 | +make the validity of a transaction depend on the local call history of whichever node |
| 46 | +happened to validate it — two nodes would disagree about the same transaction. So the |
| 47 | +validators' deserializers are instrumented (their events feed the metrics and the |
| 48 | +policy) but never gated. |
| 49 | + |
| 50 | +`AreMe` and `IsMe` are not gated either, even at the client boundary: they answer a |
| 51 | +question about local state and have no way to express "refused". Returning `false` for an |
| 52 | +identity that is in fact ours would be a wrong answer, not a denial. |
| 53 | + |
| 54 | +## Metrics |
| 55 | + |
| 56 | +All metrics are TMS-scoped: the `network`, `channel` and `namespace` labels are added by |
| 57 | +the TMS metrics provider, ahead of the labels listed below. |
| 58 | + |
| 59 | +| Metric | Type | Labels | What it tells you | |
| 60 | +|--------|------|--------|-------------------| |
| 61 | +| `identity_signature_operations_total` | counter | `op`, `role`, `outcome` | The main series. A rising `outcome="invalid"` on `op="verify"`, or a rising `outcome="throttled"`, is a misbehaving caller. | |
| 62 | +| `identity_signature_operation_duration_seconds` | histogram | `op` | Latency per operation. Signature work is CPU-bound, so this is where a resource-exhaustion attempt shows up. | |
| 63 | +| `identity_signer_cache_lookups_total` | counter | `result` (`hit`/`miss`) | A collapsing hit ratio means signer material is re-derived on every call. | |
| 64 | +| `identity_throttle_escalations_total` | counter | `level`, `reason` | Every level change, including de-escalations back to `normal`. | |
| 65 | +| `identity_throttled_principals` | gauge | `level` | How many principals are currently at `soft` / `blocked`. | |
| 66 | +| `identity_signer_resolutions_total` | counter | `outcome` (`cache`/`routed`/`fallback`) | How signers are being obtained. | |
| 67 | +| `identity_get_signer_duration_seconds` | histogram | `path` | Latency of signer resolution per path. | |
| 68 | + |
| 69 | +`op` is one of `get_signer`, `register_signer`, `register_identity_descriptor`, `is_me`, |
| 70 | +`get_audit_info`, `bind`, `owner_verifier`, `issuer_verifier`, `auditor_verifier`, |
| 71 | +`sign`, `verify`, `escalation`. `role` is `owner`, `issuer`, `auditor` or `unknown`. |
| 72 | +`outcome` is `ok`, `error`, `invalid` or `throttled`. |
| 73 | + |
| 74 | +`invalid` and `error` are deliberately separate. A `Verifier` that returns an error has |
| 75 | +*rejected a signature*; a `GetSigner` that returns an error hit a missing signer or a |
| 76 | +storage failure. The first is a security signal, the second is an operational one. |
| 77 | + |
| 78 | +Cardinality is bounded by those closed sets. No metric carries a per-identity label, |
| 79 | +since the number of identities a deployment sees is unbounded. |
| 80 | + |
| 81 | +### Suggested alerts |
| 82 | + |
| 83 | +```promql |
| 84 | +# A principal is presenting rejected signatures. |
| 85 | +rate(identity_signature_operations_total{op="verify",outcome="invalid"}[5m]) > 1 |
| 86 | +
|
| 87 | +# The policy is actively refusing work. |
| 88 | +identity_throttled_principals{level="blocked"} > 0 |
| 89 | +
|
| 90 | +# Enforcement is on and someone is hitting it. |
| 91 | +rate(identity_signature_operations_total{outcome="throttled"}[5m]) > 0 |
| 92 | +``` |
| 93 | + |
| 94 | +## The audit log |
| 95 | + |
| 96 | +Alongside the metrics, each event is written as a single structured line to the |
| 97 | +`panurus.driver.<driver>.signature` logger: |
| 98 | + |
| 99 | +``` |
| 100 | +sig-audit op=get_signer principal=abcd1234 role=owner outcome=ok path=cache cache=hit duration_ms=1.500 |
| 101 | +sig-audit op=verify principal=abcd1234 role=owner outcome=invalid duration_ms=0.412 err=[signature mismatch] |
| 102 | +sig-audit op=escalation principal=abcd1234 role=unknown outcome=ok level=blocked reason=invalid_signature_rate |
| 103 | +``` |
| 104 | + |
| 105 | +Log level follows the outcome, so a deployment can keep the interesting lines without |
| 106 | +the volume of the routine ones: |
| 107 | + |
| 108 | +| Outcome | Level | |
| 109 | +|---------|-------| |
| 110 | +| `ok` | debug | |
| 111 | +| `error`, `invalid`, `throttled` | warn | |
| 112 | +| `escalation` (policy state) | info | |
| 113 | + |
| 114 | +Fields that do not apply are omitted; an unattributed operation is written as |
| 115 | +`principal=none`. As with the metrics, the `principal` field is an identity hash — the |
| 116 | +audit log never contains identity bytes. |
| 117 | + |
| 118 | +## The escalation policy |
| 119 | + |
| 120 | +The policy keeps, per principal, a token bucket and a sliding window (one minute by |
| 121 | +default, in ten-second steps) of operation counts. A principal moves up a level when: |
| 122 | + |
| 123 | +- it exhausts its token bucket (`reason=rate`), or |
| 124 | +- the fraction of its operations that failed crosses `errorRateThreshold` |
| 125 | + (`reason=error_rate`), or |
| 126 | +- the fraction of its verifications that were rejected crosses |
| 127 | + `invalidSignatureRateThreshold` (`reason=invalid_signature_rate`). |
| 128 | + |
| 129 | +Ratios are only evaluated once the window holds at least `minSamples` observations: one |
| 130 | +failure out of three calls is not an attack. |
| 131 | + |
| 132 | +The levels are: |
| 133 | + |
| 134 | +| Level | Effect | |
| 135 | +|-------|--------| |
| 136 | +| `normal` | Full quota. | |
| 137 | +| `soft` | Rate and burst multiplied by `quotaReductionFactor`, for at least `softDuration`. A soft-limited principal is slowed, never stopped: the reduced bucket always keeps room for one token. | |
| 138 | +| `blocked` | Metered operations refused for `blockDuration`, then released back to `soft` (`reason=block_expired`). | |
| 139 | + |
| 140 | +A principal that goes `deescalateAfter` without a violation is restored one level at a |
| 141 | +time (`reason=quiet_period`). Each transition resets the window, so the counters that |
| 142 | +caused an escalation cannot immediately trigger the next one. Per-principal state is |
| 143 | +dropped after `idleTTL` of inactivity — except for principals above `normal`, whose |
| 144 | +state *is* the record that they are throttled. |
| 145 | + |
| 146 | +## Handling a denial |
| 147 | + |
| 148 | +A denied operation returns an error wrapping the `token.SignatureThrottled` sentinel. |
| 149 | +Callers should treat it as "back off", distinct from "unknown identity" or "invalid |
| 150 | +signature": |
| 151 | + |
| 152 | +```go |
| 153 | +signer, err := signatureService.GetSigner(ctx, id) |
| 154 | +if errors.Is(err, token.SignatureThrottled) { |
| 155 | + // back off and retry later, shed the request, or surface a 429-style response |
| 156 | +} |
| 157 | +``` |
| 158 | + |
| 159 | +## Configuration |
| 160 | + |
| 161 | +The policy is read per TMS from `token.tms.<name>.identity.throttle`. The whole section |
| 162 | +is optional; a missing section means the defaults below. |
| 163 | + |
| 164 | +```yaml |
| 165 | +token: |
| 166 | + tms: |
| 167 | + <name>: |
| 168 | + identity: |
| 169 | + throttle: |
| 170 | + mode: monitor |
| 171 | + rate: 200 |
| 172 | + burst: 400 |
| 173 | + window: 1m |
| 174 | + minSamples: 50 |
| 175 | + errorRateThreshold: 0.5 |
| 176 | + invalidSignatureRateThreshold: 0.2 |
| 177 | + quotaReductionFactor: 0.25 |
| 178 | + softDuration: 5m |
| 179 | + blockDuration: 1m |
| 180 | + deescalateAfter: 5m |
| 181 | + idleTTL: 10m |
| 182 | +``` |
| 183 | +
|
| 184 | +| Field | Default | Description | |
| 185 | +|-------|---------|-------------| |
| 186 | +| `mode` | `monitor` | `off` — nothing metered, nothing denied. `monitor` — evaluate and report, never deny. `enforce` — deny throttled principals. | |
| 187 | +| `rate` | `200` | Metered signature operations per second per principal. A negative value disables the policy, like `off`. | |
| 188 | +| `burst` | `400` | Bucket capacity, absorbing short spikes without raising the sustained rate. Values below `rate` are raised to `rate`. | |
| 189 | +| `window` | `1m` | Evaluation period for the ratio thresholds. | |
| 190 | +| `minSamples` | `50` | Minimum observations in a window before a ratio can escalate a principal. | |
| 191 | +| `errorRateThreshold` | `0.5` | Failing-operation fraction that escalates. `1` or more disables this trigger. | |
| 192 | +| `invalidSignatureRateThreshold` | `0.2` | Rejected-verification fraction that escalates. Stricter than the error threshold: a healthy caller does not present bad signatures. | |
| 193 | +| `quotaReductionFactor` | `0.25` | Multiplier applied to `rate` and `burst` at level `soft`. Must be in `(0,1]`. | |
| 194 | +| `softDuration` | `5m` | Minimum time on a reduced quota. | |
| 195 | +| `blockDuration` | `1m` | How long a blocked principal is refused before release back to `soft`. | |
| 196 | +| `deescalateAfter` | `5m` | Violation-free period required to restore a level. | |
| 197 | +| `idleTTL` | `10m` | How long per-principal state is kept after its last operation. | |
| 198 | + |
| 199 | +Out-of-range values are rejected at startup rather than clamped: a configuration asking |
| 200 | +for a `quotaReductionFactor` of `3` has a mistake in it, and silently treating it as `1` |
| 201 | +would leave the operator believing a policy is in force that is not. |
| 202 | + |
| 203 | +### Rolling out enforcement |
| 204 | + |
| 205 | +1. Deploy with the default `monitor` mode. |
| 206 | +2. Watch `identity_throttle_escalations_total` and the audit log's `op=escalation` |
| 207 | + lines. In monitor mode these are exactly the denials that `enforce` would have made. |
| 208 | +3. Adjust `rate`, `burst` and the two thresholds until only traffic you would want to |
| 209 | + refuse escalates. |
| 210 | +4. Switch `mode` to `enforce`. |
| 211 | + |
| 212 | +## Implementation notes |
| 213 | + |
| 214 | +For contributors: |
| 215 | + |
| 216 | +- `token/services/identity/sigobserve` — the event vocabulary (`Event`, `Op`, `Outcome`, |
| 217 | + `Observer`, `Gate`), the `Timer` that times an operation without allocating, the |
| 218 | + `InstrumentSigner`/`InstrumentVerifier` decorators, and the audit logger. It depends on |
| 219 | + nothing but `token/driver`, which is what lets `token` import it without a cycle. |
| 220 | +- `token/services/identity/throttle` — the escalation policy. It is both an `Observer` |
| 221 | + (it watches outcomes) and a `Gate` (it decides). |
| 222 | +- `token/services/ratelimit` — the per-key token buckets the policy meters with. |
| 223 | +- `token/services/identity/sigpolicy` — assembles metrics + audit log + policy into one |
| 224 | + `Stack`, so the wiring lives in one place instead of in every driver. |
| 225 | +- `token/services/identity/metrics.go` — the Prometheus sink. Every metric must declare |
| 226 | + `network`, `channel`, `namespace` as its leading labels; the TMS provider prepends |
| 227 | + those values, and a metric that omits them makes Prometheus reject the series. |
0 commit comments