Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 1 addition & 4 deletions docs/spectator/core/meters/counter.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,7 @@ to reason about the measurement and allows for aggregating the counter across in
In Atlas, the `:per-step` operator can be used to convert them back into a count-per-step on a
graph.

!!! Note
For high performance code, such as incrementing in a tight loop that lasts less than a
reporting interval, increment a local variable and add the final value to the counter after
the loop has completed.
For high-volume use, see [Performance Tips](../performance.md).

## Languages

Expand Down
95 changes: 95 additions & 0 deletions docs/spectator/core/performance.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
# Performance Tips

A few patterns are worth keeping in mind when instrumenting hot paths. These apply across
all Spectator client libraries; language-specific helpers are noted where they exist.

## Cache the meter reference

`registry.counter("server.numRequests")` (and equivalents for timer, gauge, etc.) performs
a lookup in the registry on every call. For meters that don't have dynamic tag values,
look the meter up once and reuse the reference:

```java
// Good: lookup happens once.
private final Counter requests = registry.counter("server.numRequests");

public void handle() {
requests.increment();
}
```

```java
// Avoid in hot paths: lookup on every call.
public void handle() {
registry.counter("server.numRequests").increment();
}
```

For meters where some tag values are dynamic (e.g. a status code), cache the base `Id` and
derive per-request meters with `with_tag` / `withTag`:

```java
private final Id requestsId = registry.createId("server.numRequests");

public void handle(Response res) {
registry.counter(requestsId.withTag("status", res.statusCode())).increment();
}
```

## Avoid instrumentation in tight loops

If you need to update a meter inside a tight loop where each iteration is cheap, the
instrumentation overhead can dominate. Accumulate locally and apply the delta once:

```java
long localCount = 0;
for (Item item : items) {
if (item.isFoo()) localCount++;
}
requests.increment(localCount);
```

Java's [BatchUpdater](#java-batchupdater) automates this pattern for `Counter`,
`Timer`, and `DistributionSummary`.

## Prefer basic meters over percentile variants

[Percentile Timer](meters/timer.md#percentile-timer) and
[Percentile Distribution Summary](meters/dist-summary.md#percentile-distribution-summary)
maintain a set of bucket counters. Storage cost is up to ~300x that of the basic Timer or
Distribution Summary. Use them only for one or two key indicators per application, set an
appropriate range, and keep tag cardinality bounded.

## Keep tag cardinality bounded

Every distinct combination of tag values produces a new time series. Avoid putting
high-cardinality values in tags — user IDs, request IDs, raw paths, etc. Use the
[Cardinality Limiter](../lang/java/patterns/cardinality-limiter.md) (Java) or apply a
similar mapping in other languages to cap the value set for any tag that could grow
unbounded.

## Java: BatchUpdater

For very high-volume updates within a single thread, `Counter`, `Timer`, and
`DistributionSummary` expose a `batchUpdater(batchSize)` method that buffers updates and
flushes them as a single operation. Trade-off: updates are delayed by up to `batchSize`
events before they appear on the underlying meter.

```java
try (Counter.BatchUpdater updater = requests.batchUpdater(1000)) {
for (Item item : items) {
process(item);
updater.increment();
}
}
```

The updater is `AutoCloseable`; the try-with-resources block guarantees a final flush. See
the [Counter](../lang/java/meters/counter.md), [Timer](../lang/java/meters/timer.md), and
[Distribution Summary](../lang/java/meters/dist-summary.md) pages for the per-meter
signatures.

The spectatord-backed clients (C++, Go, Node.js, Python) do not need an equivalent helper:
they buffer protocol lines client-side, and spectatord aggregates updates from many sources
before flushing to Atlas. Each language's usage page has a Line Buffer or Buffers section
for the relevant tuning knobs.
18 changes: 18 additions & 0 deletions docs/spectator/lang/java/meters/counter.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,3 +48,21 @@ events happen together.
}
}
```

## Batch Updates

For very high-volume updates within a single thread, `Counter` exposes a `batchUpdater` that
buffers updates and flushes them as a single operation. The trade-off is that updates are
delayed by up to `batchSize` events before they appear on the underlying counter.

```java
try (Counter.BatchUpdater updater = insertCounter.batchUpdater(1000)) {
for (Object obj : objs) {
impl.insert(obj);
updater.increment();
}
}
```

The updater is `AutoCloseable`; the try-with-resources block guarantees a final flush. See
[Performance Tips](../../../core/performance.md) for the cross-cutting guidance.
11 changes: 9 additions & 2 deletions docs/spectator/lang/java/meters/dist-summary.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,10 +26,17 @@ Then call record when an event occurs:
```
**Note:** If the amount recorded is less than 0 the value will be dropped.

## Batch Updates

For very high-volume updates within a single thread, `DistributionSummary` exposes a
`batchUpdater` that buffers updates and flushes them as a single operation. See
[Performance Tips](../../../core/performance.md) for the cross-cutting guidance and
[`Counter.batchUpdater`](counter.md#batch-updates) for an annotated example.

## Percentile Distribution Summary

concept and storage-cost caveats. Use a regular [Distribution Summary](dist-summary.md) unless
percentile estimates are required.
Use a regular [Distribution Summary](#distribution-summary) unless percentile estimates are
required.

Create one via the static builder:

Expand Down
9 changes: 8 additions & 1 deletion docs/spectator/lang/java/meters/timer.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,11 +105,18 @@ backend, the unit will be seconds.
[System.nanoTime()]: https://docs.oracle.com/en/java/javase/25/docs/api/java.base/java/lang/System.html#nanoTime()
[System.currentTimeMillis()]: https://docs.oracle.com/en/java/javase/25/docs/api/java.base/java/lang/System.html#currentTimeMillis()

## Batch Updates

For very high-volume updates within a single thread, `Timer` exposes a `batchUpdater` that
buffers updates and flushes them as a single operation. See
[Performance Tips](../../../core/performance.md) for the cross-cutting guidance and
[`Counter.batchUpdater`](counter.md#batch-updates) for an annotated example.

## Percentile Timer

**Note**: Percentile timers generate a metric per bucket in the histogram. Create instances
once per ID and reuse them as needed. Avoid adding tags with high cardinality as that increases
the cardinality of the metric. If at all possible, use a [Timer](timer.md) instead.
the cardinality of the metric. If at all possible, use a [Timer](#timer) instead.

To get started, create an instance using the [Registry](../registry/overview.md):

Expand Down
1 change: 1 addition & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -224,6 +224,7 @@ nav:
- Gauge: spectator/core/meters/gauge.md
- Max Gauge: spectator/core/meters/max-gauge.md
- Timer: spectator/core/meters/timer.md
- Performance Tips: spectator/core/performance.md
- Patterns:
- Age Gauge: spectator/patterns/age-gauge.md
- Monotonic Counter: spectator/patterns/monotonic-counter.md
Expand Down