metrics: collect per-CPU frequency from cpufreq kstats

[citrus-it]

The host OS recently gained per-CPU cpufreq kstats reporting effective and
average CPU frequency, derived from the AMD/Intel APERF/MPERF MSRs. This
change plumbs them into oximeter/ClickHouse alongside the existing per-CPU
microstate data, adding four new timeseries to the existing sled_cpu
target. These are also per-CPU (logical CPU; really CPU thread) metrics so they
fit nicely in there. It also proposes adding a new hertz unit to oximeter's
Units type, with the corresponding Nexus external API version bump.

Example of the kstat for CPU 1 (core 0, thread 1), re-ordered for clarity:

gimlet-sn06 # kstat -p cpufreq:1
cpufreq:1:cpufreq:average_Hz       241024
cpufreq:1:cpufreq:effective_Hz     3689201343
cpufreq:1:cpufreq:aperf_total      8462153906
cpufreq:1:cpufreq:mperf_total      4979542118
cpufreq:1:cpufreq:aperf_delta      334541
cpufreq:1:cpufreq:mperf_delta      181020
cpufreq:1:cpufreq:base_Hz          1996225357
cpufreq:1:cpufreq:class            misc
cpufreq:1:cpufreq:crtime           115.390141468
cpufreq:1:cpufreq:interval_ns      1387994050
cpufreq:1:cpufreq:snapshot_hrtime  429461004909
cpufreq:1:cpufreq:snaptime         429.977975785

The kernel opportunistically samples APERF/MPERF to calculate the frequencies,
approximately once a second. Reading the kstat doesn't trigger a new
computation, it just snapshots the most recently computed values. Therefore
each reading describes a roughly one-second window ending at snapshot_hrtime
with interval_ns recording the actual window length. Given the opportunistic
nature, on a very quiet CPU, or one which is running a single thread and is
isolated from interrupts, the window may stretch.

The two computed statistics are:

• effective_Hz: the frequency at which the CPU ran while it was executing
  instructions, i.e. base_Hz * aperf_delta / mperf_delta. Idle time does
  not contribute.
• average_Hz: APERF cycles over elapsed wall-clock time - the cycles
  actually executed per second of real time, so it measures delivered work
  rather than a clock speed and counts idle as 0 Hz.

We collect four new metrics on the existing sled_cpu target, keyed by
cpu_id (per logical CPU, matching the existing cpu_nsec microstate series),
sampled by the sled-agent's kstat sampler on the existing 10-second CPU
interval:

Timeseries	Type	Units	Meaning
sled_cpu:effective_frequency	gauge u64	hertz	effective_Hz, as computed by the kernel over its ~1s window
sled_cpu:average_frequency	gauge u64	hertz	average_Hz, ditto
sled_cpu:aperf_total	cumulative u64	count	running total of APERF ticks
sled_cpu:mperf_total	cumulative u64	count	running total of MPERF ticks

Sample volume is 4 × n_cpus per sled every 10s, so about 51 samples/s for a
128-thread Gimlet, on top of ~64/s for the existing microstate data.

Samples are timestamped with snapshot_hrtime (the end of the kernel's
averaging window) rather than the kstat read time, since the read can lag
the computation by up to a second or so.

Since the kernel's window is roughly one second and we sample every ten,
each stored gauge value describes one ~1s window and the other ~9 seconds
are unobserved. This is a deliberate trade-off:

• adequate for trend visibility, for example sustained downclocking due to
  thermal/power limits, fleet-wide frequency comparisons, correlation
  with temperature/power over minutes and hours...
• It is lossy for anything sub-10s. Short frequency excursions are
  invisible, and averaging the stored gauges over a long window is not a
  time-weighted average of the CPU's frequency over that window.

APERF and MPERF, on the other hand, are free-running 64-bit MSRs. With these
MSR values stored as oximeter cumulative counters, any query window yields
results that are accurate for the range. The calculations from these are
straightforward if you have base frequency to hand (see below):

• effective frequency while running: base_Hz × Δaperf / Δmperf
• delivered work (the average_Hz analogue): Δaperf / (t1 − t0)
• C0 residency in seconds: Δmperf / base_Hz (cross-checkable against the
  cpu_nsec microstate counters)

We are collecting both forms because the gauges are directly plottable from a
single OxQL query today and match what an operator sees in kstat(8) on the
sled. The totals are the better record for investigations but require some
manipulation outside of OxQL.

Oximeter's Units enum had no frequency unit, so this change also adds
one. That's an API bump and I modelled the change on the recent addition of
"Joules". The alternative would be to use "count" or "none" for these, but
I think it's better to add the new type.

I also chose to add two gauge metrics rather than one with a kind field.
Effective and average are distinct things so separate timeseries seems
clearer.

base_Hz is not collected, and perhaps it should be. It's is constant per
CPU part on oxide hardware. A per-10s timeseries of a constant seems
wasteful, although it is needed for calculations on the other values so
I don't know what's best here. Could we, for example, sample it far less
frequently?

[rmustacc]

It is lossy for anything sub-10s. Short frequency excursions are
invisible, and averaging the stored gauges over a long window is not a
time-weighted average of the CPU's frequency over that window.

I think this makes the stats mostly useless for any investigation especially given these aren't visible in a VM. Short excursions are very real. I think this is going to be a general trend and problem with the 10s sampling window. My main perspective is that one of the things we can do is provide this on behalf of customers and operators to make debugging easier given they don't have a high-fidelity way of getting this.

Longer term we may want to consider the ability to correlate this to a virtual machine like with other things. But that's mostly

[bnaecker]

If there are brief excursions, and those are really valuable to capture, we could consider sampling dynamically based on the shape of the data. The producer is in control as far as actually sampling the data, and the collection interval only controls how often the oximeter collector comes calling for it. As a very simple example, one could sample very finely, and throw away most of them if the derivative is "small".

[bnaecker]

Longer term we may want to consider the ability to correlate this to a virtual machine like with other things

This is interesting. I'd never really thought about how to do this, but it suggests we want Nexus to produce a timeseries that describes the hosting sled for each VM. Cool.