increase() on per-session claude_code counters is wrong in BOTH directions (undercounts short sessions, 3.5x overcounts gapped ones)

[ryanrozich]

Problem

dashboards/unified-dashboard.json aggregates native claude_code_* counters with increase(). Those counters are cumulative per session (every series carries session_id) and the prometheus exporter sets metric_expiration: 15m. That combination breaks increase() in two opposite ways at once.

Mechanism 1 — undercount (short sessions)

A session whose first scraped sample is already nonzero has that sample treated as the baseline and subtracted. A single-sample series contributes ~0. Relay phase workers — a claude -p that runs ~10 minutes and exits — are exactly this case.

Mechanism 2 — overcount (sessions with an idle gap)

A session that goes idle for more than 15m has its series expire and later reappear. increase() reads that gap as a counter reset and adds the pre-gap value on top of the post-gap climb. Long-running interactive sessions do this repeatedly, and they are the biggest spenders.

Evidence (measured live)

Per-series, split by whether the series has any reset:

=== 24h window ===  (410 cost series, 32 with >=1 reset)
  NO-reset series : max_over_time=$1223.76   increase=$1054.13    increase =  86.1% of true  (14% LOW)
  RESET series    : max_over_time=$ 216.16   increase=$ 758.43    increase = 350.9% of true  (3.5x HIGH)

=== 6h window ===   (93 cost series, 9 with >=1 reset)
  NO-reset series : max_over_time=$ 357.94   increase=$ 317.24    increase =  88.6% of true
  RESET series    : max_over_time=$  84.53   increase=$ 264.48    increase = 312.9% of true

And on a single short relay worker (CTC-936, [1h], no gap — pure Mechanism 1):

max_over_time  {linear_key="CTC-936"}  0.6741176
increase()     {linear_key="CTC-936"}  0.4491615511466666      <- 33% LOW

max_over_time is correct in both regimes: it takes each session series' final cumulative value, which is that session's true total, and summing across series gives the true aggregate.

⚠️ Which way the numbers move depends on the window — do NOT assume one direction

An earlier framing of this ticket said totals would simply go up. That is wrong, and the correction matters because someone will diff a before/after screenshot.

• Whole-dashboard totals over 6h/24h will go DOWN (~20-26%). The handful of gap-inflated long sessions outweigh the many slightly-undercounted short ones.
• Per-ticket / per-relay-worker figures will go UP (~14-33%), because those are short, gapless sessions hit only by Mechanism 1.

Neither movement is a change in real spend. Both are corrections of a measurement error. When comparing burn across time, do not compare a pre-fix figure against a post-fix one.

Scope

34 increase() calls in unified-dashboard.json. 29 are on the six per-session native counters and must be converted:
claude_code_cost_usage_USD_total, _token_usage_tokens_total, _lines_of_code_count_total, _pull_request_count_total, _commit_count_total, _session_count_total.

The other 5 must be left alone: they are on claude_code_tool_result_total, which is connector-derived (a count connector off the logs pipeline, collector-config.yaml), carries no session_id, and is collector-lifetime cumulative. increase() is correct there and max_over_time would be wrong.

operator-usage.json already uses the correct pattern and is the reference.

Known trade-off, accepted

max_over_time attributes a session's entire cumulative value to any window containing its tail, so a session straddling a window boundary is counted wholly in the later window. The error is bounded by one session's length and is far smaller than either mechanism above. This is the standard treatment for per-session cumulative counters and matches operator-usage.json.

Acceptance criteria

Given a short, gapless relay session whose series exposes a nonzero first sample in the range
When a per-ticket cost or token panel is evaluated
Then the panel reports the session's full value, not the value minus its first sample

Given a long session whose series expired and reappeared inside the range
When a total cost panel is evaluated
Then the session contributes its final cumulative value once, not its pre-gap value plus its post-gap climb

Given the five claude_code_tool_result_total expressions
When the change is applied
Then they still use increase() — the connector metric is not per-session

Given a time-series panel whose aggregation becomes a running total
When the change is applied
Then the panel title says "Cumulative", so its semantics are not silently altered

Given the edited dashboard
When every expression (including panels nested inside collapsed rows) is submitted to the live Prometheus datasource
Then all parse without error, and scripts/validate-dashboard.sh reports no new failures