diff --git a/reference/analytics/overview.md b/reference/analytics/overview.md index 5a6309b6..86997e13 100644 --- a/reference/analytics/overview.md +++ b/reference/analytics/overview.md @@ -171,14 +171,71 @@ Harper automatically tracks the following metrics for all services. Applications ### Resource Usage Metrics -| `metric` | Key attributes | Other | Unit | Description | -| ------------------------- | ------------------------------------------------------------------------------------------------ | ------------------- | ------- | --------------------------------------------------------------------------------- | -| `database-size` | `size`, `used`, `free`, `audit` | `database` | bytes | Database file size breakdown | -| `main-thread-utilization` | `idle`, `active`, `taskQueueLatency`, `rss`, `heapTotal`, `heapUsed`, `external`, `arrayBuffers` | `time` | various | Main thread resource usage: idle/active time, queue latency, and memory breakdown | -| `resource-usage` | (see below) | | various | Node.js process resource usage (see [resource-usage](#resource-usage-metric)) | -| `storage-volume` | `available`, `free`, `size` | `database` | bytes | Storage volume size breakdown | -| `table-size` | `size` | `database`, `table` | bytes | Table file size | -| `utilization` | | | % | Percentage of time the worker thread was processing requests | +| `metric` | Key attributes | Other | Unit | Description | +| ------------------------------- | ------------------------------------------------------------------------------------------------ | ------------------- | ------- | --------------------------------------------------------------------------------------------------------------------- | +| `database-size` | `size`, `used`, `free`, `audit` | `database` | bytes | Database file size breakdown | +| `main-thread-utilization` | `idle`, `active`, `taskQueueLatency`, `rss`, `heapTotal`, `heapUsed`, `external`, `arrayBuffers` | `time` | various | Main thread resource usage: idle/active time, queue latency, and memory breakdown | +| `read-transaction-queue-depth` | `depth`, `maxDepth` | | count | Open tracked read transactions (see [transaction queue depth](#transaction-queue-depth-metrics)) | +| `resource-usage` | (see below) | | various | Node.js process resource usage (see [resource-usage](#resource-usage-metric)) | +| `storage-volume` | `available`, `free`, `size` | `database` | bytes | Storage volume size breakdown | +| `table-size` | `size` | `database`, `table` | bytes | Table file size | +| `transaction-commit-time` | `mean`, `median`, `p90`, `p95`, `p99`, `p999` | | ms | Duration from write commit submission to settlement (see [transaction queue depth](#transaction-queue-depth-metrics)) | +| `utilization` | | | % | Percentage of time the worker thread was processing requests | +| `write-transaction-queue-depth` | `depth`, `maxDepth` | | count | In-flight write transaction commits (see [transaction queue depth](#transaction-queue-depth-metrics)) | + +#### Transaction Queue Depth Metrics + + + +`write-transaction-queue-depth` and `read-transaction-queue-depth` expose how many transactions are +in flight against the storage engine per worker thread — a concurrency and throughput signal, not a +reliable predictor on their own of the `Outstanding write transactions have too long of queue, please +try again later` (HTTP 503) rejection: `maxDepth` amplitude reflects concurrent commits, not whether +any single one is approaching the +[`storage.maxTransactionQueueTime`](../database/storage-tuning.md#storagemaxtransactionqueuetime) +duration limit (default 45s) that actually trips the 503. + +`transaction-commit-time` records each commit's submit-to-settle duration on that same clock, and a +rising `p99`/`p999` (in the `hdb_analytics` aggregate table, where percentiles are computed — they +aren't present on `hdb_raw_analytics`) is a leading indicator of _gradual_ slowdowns approaching that +limit. It doesn't help with a single commit that hangs indefinitely, though: the metric only records +once a commit settles, so a genuinely wedged commit contributes no sample at all, while +`write-transaction-queue-depth`'s `depth` stays elevated on that thread for as long as the commit +remains outstanding. Harper also logs once per stuck commit when the 503 check itself fires, which is +the authoritative signal for that specific failure. + +| Field | Unit | Description | +| ---------- | ----- | ------------------------------------------------------- | +| `depth` | count | Instantaneous depth sampled at emit time | +| `maxDepth` | count | High-water mark since this thread's last emitted sample | + +- **`write-transaction-queue-depth`** counts write commits handed to the storage engine whose commit + promises have not yet settled — how many commits this thread is juggling concurrently. This is + in-flight, not durability: under `storage.writeAsync: true` a settled commit promise does not + guarantee the write has been synced to disk. +- **`read-transaction-queue-depth`** counts concurrently open tracked read transactions, including + ones opened with snapshot disabled. A high count can mean either many short-lived reads or a few + long-lived ones — the count alone can't distinguish them, so use it as a concurrency signal; a + duration-based metric would be needed to identify a single transaction held open long enough to + hold back compaction. + +Both metrics are gauges tracked only on the RocksDB write/read path, sampled per worker thread. On an +LMDB-backed database (`storage.engine: lmdb`), `depth` and `maxDepth` for both metrics always read `0` +— indistinguishable from a healthy, empty queue — regardless of actual read/write load. All per-thread +analytics reporting, including these gauges, piggybacks on the thread having recorded some other +analytics-eligible activity in the period — a thread with no recordable activity in a given second +emits no row at all rather than an explicit `depth: 0`. Absence of a sample is not the same as a +healthy reading, particularly for `read-transaction-queue-depth` on an otherwise-quiet thread holding a +single long-lived read. + +The raw per-thread entries in `hdb_raw_analytics` retain each thread's true instantaneous `depth` and +per-period `maxDepth`; treat those as the reliable source for spike detection. The aggregate +`hdb_analytics` table is not a sum of per-thread peaks — each thread's `maxDepth` is first averaged +across its raw samples for the period, then those per-thread averages are summed — so a brief +single-thread spike is diluted rather than preserved. Always alert on `hdb_raw_analytics.maxDepth` (or +lower the sampling/aggregation period) rather than relying on the aggregate table to catch short +spikes. Tune the concrete alert threshold against a baseline for your workload, since absolute depth +scales with worker-thread count and per-transaction size. #### `resource-usage` Metric diff --git a/reference/configuration/options.md b/reference/configuration/options.md index 482c7d9d..326f6997 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -253,7 +253,7 @@ storage: - `compression` — LZ4 record compression; _Default_: `true` (enabled by default since v4.3.0). Sub-options: `dictionary`, `threshold` - `compactOnStart` — Compact all non-system databases on startup; _Default_: `false` (Added in: v4.3.0) - `compactOnStartKeepBackup` — Retain compaction backups; _Default_: `false` -- `maxTransactionQueueTime` — Max write queue time before 503; _Default_: `45s` +- `maxTransactionQueueTime` — Max time a single write commit may stay unsettled before Harper starts rejecting writes with 503; see [Storage Tuning](../database/storage-tuning.md#storagemaxtransactionqueuetime); _Default_: `45s` - `noReadAhead` — Advise OS against read-ahead; _Default_: `false` - `prefetchWrites` — Prefetch before write transactions; _Default_: `true` - `path` — Database files directory; _Default_: `/database` diff --git a/reference/database/storage-tuning.md b/reference/database/storage-tuning.md index d99153b8..e9964657 100644 --- a/reference/database/storage-tuning.md +++ b/reference/database/storage-tuning.md @@ -37,7 +37,7 @@ Type: `string` (duration) Default: `45s` -The maximum estimated time a write may wait in the commit queue before Harper rejects new writes with HTTP 503. Acts as backpressure when downstream disk I/O cannot keep up with incoming writes. +The maximum time a single write commit may remain unsettled before Harper starts rejecting new application-originated writes on that thread with HTTP 503. This is a per-commit duration check, not a queue-length threshold — it acts as backpressure when downstream disk I/O cannot keep up with incoming writes. Deletes and writes applied from a canonical source (e.g. replication or a caching source) bypass this check. Lower this in latency-sensitive systems where it is better to shed load early than to let request queues grow. Raise it when occasional disk-write bursts are expected and the application can tolerate longer commit latency. diff --git a/release-notes/v5-lincoln/5.2.md b/release-notes/v5-lincoln/5.2.md index dde9cb59..d95b12c9 100644 --- a/release-notes/v5-lincoln/5.2.md +++ b/release-notes/v5-lincoln/5.2.md @@ -26,6 +26,12 @@ Components can now declare recurring jobs in their configuration with a new buil The `set_configuration` operation now accepts `"replicated": true` to apply a configuration change to all cluster nodes in a single Operations API call, with per-node outcomes reported in the response's `replicated` array. Only cluster-appropriate parameters should be replicated — see [Configuration Operations](/reference/v5/configuration/operations#set-configuration). +## Analytics + +### Transaction queue depth metrics + +New `write-transaction-queue-depth` and `read-transaction-queue-depth` metrics report per-thread write-commit and open-read-transaction concurrency against the storage engine, as a throughput/concurrency signal. A new `transaction-commit-time` metric records per-commit submit-to-settle duration, a leading indicator of gradual slowdowns toward the write-queue HTTP 503 rejection — see [Transaction Queue Depth Metrics](/reference/v5/analytics/overview#transaction-queue-depth-metrics). + ## CLI ### Explicit Authentication for Operations API Commands