indexer: Flux row conversion is 66% of a telemetry-usage cycle (7m04s of 10m43s) — profile and reduce

[nikw9944]

Summary

66% of a staging telemetry-usage refresh cycle is spent converting Flux rows in-process — more than the InfluxDB read and the ClickHouse insert combined. This is now the dominant term in whether ingest keeps up with real time, and it is a code question rather than a capacity one.

Full breakdown of one staging cycle (2026-08-06 14:35:24 → 14:46:08, mainnet-beta, indexer at 4 CPU):

Phase	Log boundary	Duration	Share
Baseline lookup	queried baselines from clickhouse (cache hit)	~0s (920ns)	0%
InfluxDB Flux read	→ main influxdb query completed (6,899,363 rows)	2m32s	24%
Row conversion	→ queried influxdb (4,611,619 records)	7m04s	66%
ClickHouse insert	→ inserted data to clickhouse	1m07s	10%
Total	refresh completed	10m43s

That middle phase is sort.Slice + convertRowsToUsage inside queryInfluxDB (indexer/pkg/dz/telemetry/usage/view.go:1056, conversion at :1185, ~300 lines): 6.9M raw map[string]any rows are sorted by timestamp, then walked to compute per-key deltas, forward-fill sparse counters, and join link metadata, emitting 4.6M InterfaceUsage records.

Because a capped catch-up cycle advances the watermark by 10 minutes of data, a cycle must finish in under 10 minutes for ingest to converge. At 10m43s staging is at 0.81× real time and diverging — and the conversion phase alone (7m04s) is 71% of that 10-minute budget.

Why this is worth attacking directly

Capacity work has taken this as far as it goes cheaply. infra#2180 (indexer 2 → 4 CPU) produced near-linear gains on the other two phases — Flux read 6m29s → 2m32s, insert 2m6s → 1m07s — but left the cycle above 10 minutes because conversion dominates. infra#2194 (4 → 8 CPU) is in flight and its outcome hinges entirely on whether this phase is CPU-bound:

• If conversion halves: 76s + 212s + 34s ≈ 5m22s → 1.86× real time, comfortable.
• If it does not: 76s + 424s + 34s ≈ 8m54s → 1.12× real time, marginal.

Either way, halving this one phase is worth more than any remaining capacity step, and it is the difference between staging tracking real time with margin and sitting on the edge.

Suggested lines of investigation

1. Profile it first. A pprof CPU profile over one cycle would say whether the cost is JSON/interface decoding, the sort, map allocations, or the delta/forward-fill walk. Nothing below should be built before that.
2. []map[string]any is a likely major cost. 6.9M rows as maps of boxed values means tens of millions of allocations and interface unwraps per cycle. Decoding into a typed struct (or reusing row buffers) would attack allocation and GC pressure directly. Related: indexer: use HTTP/CSV InfluxDB queries #150 proposes switching the transport to HTTP/CSV, which would change the decode path anyway — worth designing these together rather than twice.
3. The sort may be avoidable. sort.Slice over 6.9M rows with a comparator that re-extracts timestamps from map[string]any is expensive. If InfluxDB can return rows already ordered by time (or per-key ordering is sufficient), the sort could shrink or disappear.
4. Streaming instead of materializing. Conversion currently needs the whole window in memory before the insert starts. Processing in ordered chunks would overlap conversion with the Flux read and the insert, cutting wall time even if total CPU is unchanged — though note this interacts with the whole-window delta ordering that the sparse-counter forward-fill depends on, and with the baseline-cache contract (see the durable-partial-progress follow-up on indexer: telemetry-usage catch-up livelocks when a cycle exceeds the activity budget — staging watermark frozen 22.6h across 76 identical windows #740).

Prod relevance

Prod (8 CPU) is healthy at ~6-minute cycles against a 15-minute activity budget, so this is not urgent there. But it is the same code on the same row volumes, and it sets prod's recovery speed after any outage: the faster this phase, the faster prod climbs out of a catch-up backlog before the 24h horizon drops a span. It also directly affects how much headroom #747's adaptive span has to work with.

Context

Measured while running the staging capacity experiments for infra#2106 (infra#2180, infra#2194). Related: #740 (adaptive catch-up span, and its durable-partial-progress follow-up), #150 (HTTP/CSV InfluxDB transport).

[nikw9944]

Update: conversion scales sub-linearly with CPU — evidence it is allocation/GC-bound, not CPU-bound

infra#2194 took the staging indexer from 4 to 8 CPU. Measured across four cycles, with row counts unchanged (~6.9M read / 4.6M written):

Phase	4 CPU	8 CPU	Speedup
InfluxDB Flux read	2m32s–2m42s	1m43s–1m58s	~1.5×
Row conversion	7m04s	4m42s	~1.5×
ClickHouse insert	1m07s–1m18s	23s–30s	~2.8×

Conversion is still 69% of a ~6m50s cycle, so this remains the dominant term. But the interesting part is the shape of the scaling: doubling cores bought only ~1.5× here, while the insert path gained ~2.8× on the same step.

Sub-linear scaling with core count is the signature of allocation and GC pressure rather than CPU-bound compute. That is consistent with the mechanism: 6.9M rows materialized as []map[string]any means tens of millions of heap allocations and interface unwraps per cycle, and GC work does not parallelize away with more cores the way arithmetic does.

Practical implication for the fix ordering in this issue: prioritize reducing allocations over parallelizing the walk. Specifically, option 2 (typed decode / reusable row buffers instead of map[string]any) now looks more promising than throwing goroutines at the existing structure, and a heap profile is probably more informative than a CPU profile — or at least worth capturing alongside it.

For reference, this phase is no longer blocking staging convergence: at 8 CPU the cycle is 6m50s against a 10-minute budget, so staging now runs at 1.18× real time. This is now an efficiency and prod-recovery-speed issue rather than a correctness one.