Skip to content

Export per NUMA node local DRAM latency (#599) - #599

Open
abhiShedge wants to merge 3 commits into
facebookincubator:mainfrom
abhiShedge:export-D116837250
Open

Export per NUMA node local DRAM latency (#599)#599
abhiShedge wants to merge 3 commits into
facebookincubator:mainfrom
abhiShedge:export-D116837250

Conversation

@abhiShedge

@abhiShedge abhiShedge commented Aug 20, 2026 •
edited by meta-codesync Bot
Loading

Copy link
Copy Markdown
Contributor

Summary:

Publishes the per NUMA node latency computed in the previous commit. Adds
KernelInfo::memLatLocalNode, forwards it through
KernelMonitor::sumOverHistory, and emits mem_lat_local_node<n>_ns.

Three details differ from the existing per-socket latency keys, because a
socket is not a node here:

  • The array is indexed by NUMA node, not socket. AMD's NPS setting splits one
    socket into several nodes, so the existing memLat*CPU[socket] arrays
    cannot represent it.
  • The forwarding and accumulation loops are bounded by node count, not
    cpuSocketsInt. Reusing the socket loops would silently carry only node 0
    on a single-socket part, which is exactly the target configuration.
  • Emission sits outside the cpuSockets > 1 block. That block exists to skip
    per-socket keys on single-socket hosts, but a single-socket host is precisely
    where this metric is interesting.

Only nodes that actually reported a latency publish a key, so hosts that do not
compute the breakdown add no ODS keys and emission stays limited to the AMD
hosts that do.

The host-wide memLatLocalAvg stays 0 on AMD, matching existing behavior;
that signal already ships as l3_to_{near,far}_dram_lat_avg_ns.

Differential Revision: D116837250

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 20, 2026
@meta-codesync

meta-codesync Bot commented Aug 20, 2026

Copy link
Copy Markdown

@abhiShedge has exported this pull request. If you are a Meta employee, you can view the originating Diff in D116837250.

abhiShedge added a commit to abhiShedge/dynolog that referenced this pull request Aug 21, 2026
@abhiShedge @facebook-github-bot
Export per NUMA node local DRAM latency (facebookincubator#599)
cc4d5e4 
Summary:

Publishes the per NUMA node latency computed in the previous commit. Adds
`KernelInfo::memLatLocalNode`, forwards it through
`KernelMonitor::sumOverHistory`, and emits `mem_lat_local_node<n>_ns`.

Three details differ from the existing per-socket latency keys, because a
socket is not a node here:

- The array is indexed by NUMA node, not socket. AMD's NPS setting splits one
  socket into several nodes, so the existing `memLat*CPU[socket]` arrays
  cannot represent it.
- The forwarding and accumulation loops are bounded by node count, not
  `cpuSocketsInt`. Reusing the socket loops would silently carry only node 0
  on a single-socket part, which is exactly the target configuration.
- Emission sits outside the `cpuSockets > 1` block. That block exists to skip
  per-socket keys on single-socket hosts, but a single-socket host is precisely
  where this metric is interesting.

Only nodes that actually reported a latency publish a key, so hosts that do not
compute the breakdown add no ODS keys and emission stays limited to the AMD
hosts that do.

The host-wide `memLatLocalAvg` stays 0 on AMD, matching existing behavior;
that signal already ships as `l3_to_{near,far}_dram_lat_avg_ns`.

Differential Revision: D116837250
@meta-codesync meta-codesync Bot changed the title Export per NUMA node local DRAM latency Export per NUMA node local DRAM latency (#599) Aug 21, 2026
@abhiShedge @facebook-github-bot
Read the CPU to NUMA node map from sysfs (facebookincubator#597)
a16a9e5 
Summary:

A CPU socket is not always a NUMA node. AMD's NPS BIOS setting splits a single
socket into several NUMA nodes -- on Venice under NPS2 one socket presents two
DRAM nodes -- so a node cannot be derived from `getSocketCoreMapFromSysfs()`.
Reporting per NUMA node memory metrics needs the real node topology.

Add `getCpuToNumaNodeMapFromSysfs()`, which reads
`sys/devices/system/node/node<N>/cpulist`. Two details worth noting:

- `cpulist` uses inclusive ranges (`0-123,248-371`). Parsing it with a plain
  integer conversion would silently keep only the lower bound of each range, so
  ranges are expanded explicitly.
- A memory-only node, such as a CXL node, has an empty `cpulist` and therefore
  contributes no entries. Callers deriving a node count from this map see only
  nodes that actually have CPUs, which is what per-CPU attribution needs.

Also expose `PmuDeviceManager::getRootDir()` so callers that read sysfs
themselves honor the mock tree used by tests.

Differential Revision: D116798426
@abhiShedge @facebook-github-bot
Derive per NUMA node DRAM latency on AMD (facebookincubator#598)
a6cad33 
Summary:

The amd_l3 sampled-latency counters are per CCX, and the Dram_Near data source
already means "this CCX's own NUMA node". The shared uncore path sums every
CCX instance into a single host sample before any average is taken, so the node
breakdown is lost and only a host-wide latency is available.

Read the `L3DramLat` group a second time through the new keyed API, which
preserves the device key. The key carries the servicing CPU, and a CCX belongs
to exactly one NUMA node, so the values can be bucketed per node and published
as `L3DramLat::node<N>` metric frames alongside the untouched host frame.
`calcAvgL3ToNearDramLatNanoSecForNode()` then reads a node's frame with the
existing latency math.

Notes:
- The per node average sums the accumulators and the request counts separately
  before dividing, so busier CCXs weigh proportionally. Averaging the per-CCX
  ratios would be wrong.
- The node topology comes from sysfs once at configure time; it is fixed for
  the lifetime of the boot. A memory-only node (CXL) has no CPUs and so never
  becomes an empty bucket.
- The host path, its metric names, and the perf event count are unchanged, so
  the shipped `l3_to_*_dram_lat_avg_ns` keys are unaffected.
- `saveArchSpecificMetricFrames()` is a no-op hook on the base, so Intel and
  ARM are unaffected.

Differential Revision: D116837219
@abhiShedge @facebook-github-bot
Export per NUMA node local DRAM latency (facebookincubator#599)
5cc2ef4 
Summary:

Publishes the per NUMA node latency computed in the previous commit. Adds
`KernelInfo::memLatLocalNode`, forwards it through
`KernelMonitor::sumOverHistory`, and emits `mem_lat_local_node<n>_ns`.

Three details differ from the existing per-socket latency keys, because a
socket is not a node here:

- The array is indexed by NUMA node, not socket. AMD's NPS setting splits one
  socket into several nodes, so the existing `memLat*CPU[socket]` arrays
  cannot represent it.
- The forwarding and accumulation loops are bounded by node count, not
  `cpuSocketsInt`. Reusing the socket loops would silently carry only node 0
  on a single-socket part, which is exactly the target configuration.
- Emission sits outside the `cpuSockets > 1` block. That block exists to skip
  per-socket keys on single-socket hosts, but a single-socket host is precisely
  where this metric is interesting.

Only nodes that actually reported a latency publish a key, so hosts that do not
compute the breakdown add no ODS keys and emission stays limited to the AMD
hosts that do.

The host-wide `memLatLocalAvg` stays 0 on AMD, matching existing behavior;
that signal already ships as `l3_to_{near,far}_dram_lat_avg_ns`.

Differential Revision: D116837250
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. meta-exported

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@abhiShedge