Skip to content
Open
39 changes: 36 additions & 3 deletions reference/analytics/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,10 +55,8 @@ Example raw entry:
"metric": "bytes-sent",
"path": "search_by_conditions",
"type": "operation",
"median": 202,
"mean": 202,
"p95": 202,
"p90": 202,
"distribution": [202],
"count": 1
},
{
Expand Down Expand Up @@ -169,6 +167,41 @@ Harper automatically tracks the following metrics for all services. Applications
| `bytes-received` | node.database | `replication` | `blob` | bytes | Bytes received for blob replication |
| `replication-latency` | node.database.table | | `ingest` | ms | Time difference from source commit timestamp to local time |

### Storage Metrics

<VersionBadge version="v5.2.0" />

| `metric` | `path` | `method` | `type` | Unit | Description |
| ------------------------- | ------ | -------- | ------ | ---- | ------------------------------------------------------------------- |
| `transaction-commit-time` | | | | ms | RocksDB write-commit duration, submit to settle, per commit attempt |

`transaction-commit-time` is recorded on the RocksDB asynchronous commit path only; it is not
emitted for LMDB-backed databases, and not for the synchronous `commitSync()` path used during
transaction-log replay. Each sample covers one commit attempt, not one logical write transaction — a
transient-conflict retry re-issues the commit and records its own sample, so `count` can exceed the
number of logical writes. A sample is only recorded once an attempt settles, so a commit that is
still outstanding contributes nothing yet. Raw entries (`hdb_raw_analytics`) carry `mean`,
`distribution`, and `count`; percentiles (`median`, `p90`, `p95`, `p99`, `p999`) are only available on
the per-minute aggregate (`hdb_analytics`) once raw entries are rolled up — query the aggregate table
for percentile-based alerting. It shares a timebase with the RocksDB storage engine's overload guard,
which times only the commit attempt it arms: when a tracked outstanding commit on a thread exceeds
`storage.maxTransactionQueueTime` (default 45s), Harper rejects new record updates and publishes on
that thread with `Outstanding write transactions have too long of queue, please try again later`
(HTTP 503) — deletes and writes applied from a canonical source (e.g. replication or a caching
source) bypass this check. The guard tracks at most one outstanding commit per thread: a retry
issued while the prior attempt still holds that slot (a coordinated retry, or an early backoff
retry) recommits before the slot clears and is never armed, so a wedge there won't trip the 503; a
later, backoff-delayed retry recommits after the slot clears and is tracked like a fresh attempt.

A rising `p99`/`p999` signals commits are taking longer to drain — from write volume, large
transactions, or a saturated storage volume — and is a useful early warning to shed or throttle write
load. But the metric shares only a timebase with the guard, not its population: a wedged commit
contributes no sample until it settles, and an early retry that wedges can go untracked by the guard
entirely (see above). Don't treat this distribution as a leading indicator on its own — watch the
server log for the "Rejecting writes on this thread" error the guard emits when it does trip, and
don't rely solely on percentiles trending toward `storage.maxTransactionQueueTime`. Tune the
threshold against a baseline for your workload.

### Resource Usage Metrics

| `metric` | Key attributes | Other | Unit | Description |
Expand Down
6 changes: 6 additions & 0 deletions release-notes/v5-lincoln/5.2.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,3 +43,9 @@ Components can now declare `host` and `urlPath` in `config.yaml`, or pass them t
### Web Application Firewall

Harper Pro now includes a Web Application Firewall that evaluates rule-based IP/CIDR, method, path, header, and query conditions before authentication and application routing. Rules support block, log, and score actions; cluster-wide monitor and off modes; per-rule shadowing; node activation gates; live replicated updates; and RE2-backed regular expressions. See [Web Application Firewall](/reference/v5/web-application-firewall/overview).

## Analytics

### `transaction-commit-time` metric

A new `transaction-commit-time` storage metric records the submit-to-settle duration of RocksDB write commits, giving operators a distribution to watch alongside the `storage.maxTransactionQueueTime` overload guard — see [Storage Metrics](/reference/v5/analytics/overview#storage-metrics).