Skip to content

full_system_profiling.md

Latest commit

 

History

History
89 lines (65 loc) · 3.47 KB

full_system_profiling.md

File metadata and controls

89 lines (65 loc) · 3.47 KB
Raw

full_system_profiling

Full-machine example. Drives ProfilerSuite — the orchestrator over GpuProfiler, SystemProfiler, and DiskProfiler — from a single .pbtxt config file.

Source: examples/full_system_profiling.cu

What it does

Runs the same GEMM + vecAdd workload as gemm_profiling, but with three profilers collecting in parallel:

  • GPU — CUPTI PM sampling at the rate set in the config (default 10 kHz). SM cycles, warp occupancy, DRAM throughput, PCIe read/write, NVLink rx/tx.
  • System — /proc-based CPU + memory sampling (system-wide and per-PID) at a separate rate (default 100 Hz).
  • Disk — /proc/diskstats + /sys/block/*/inflight + /proc/[PID]/io for per-device throughput and per-PID IO (default 100 Hz).

Each profiler runs two threads (sample + flush) and writes its own length-delimited .pb file. Timestamps are anchored to steady_clock with a sync anchor so GPU, CPU, and disk data line up on one timeline.

Profiler setup (external .pbtxt)

The program takes only one flag — where to find the config:

./examples/full_system_profiling                       # uses configs/example.pbtxt
./examples/full_system_profiling -c my_config.pbtxt

The default path is resolved at runtime from __FILE__ (via the SOURCE_FILE_DIR / DEFAULT_CONFIG_PATH macros at the top of examples/full_system_profiling.cu), so running the binary from any working directory picks up the same configs/example.pbtxt shipped with the source tree.

Everything else — device index, metrics, sampling rates, tracked PIDs, tracked disks, output files, output directory — lives in the config file and is changeable without a rebuild. See configs/example.pbtxt.

Sentinel: pids: 0 is resolved to the current process PID at runtime, so you don't need to know it in advance.

Run

cd build
CUDA_VISIBLE_DEVICES=1 ./examples/full_system_profiling

Output: three .pb files (gpu_metrics.pb, system_metrics.pb, disk_metrics.pb) in the current directory, or under output_dir if set in the config.

Visualize

# Static PNG
python tools/visualize_all.py profiling_output/session_metadata.pb \
    -o full_profile.png

# Or the Bokeh interactive viewer
python tools/visualize_interactive.py profiling_output/session_metadata.pb

Both consume the single session_metadata.pb and auto-discover the per-probe .pb files from the manifest. Panel set comes from configs/visualizer_panels.pbtxt; entries that match no series in the trace (e.g. disk panels on a run with no disk probe) are skipped.

Permission notes

  • /proc/[PID]/io needs same-UID or CAP_SYS_PTRACE. The profiler warns once and skips per-process disk IO if unavailable. Fix with sudo setcap cap_sys_ptrace=ep <binary> or sudo sysctl -w kernel.yama.ptrace_scope=0.
  • CUPTI PM sampling typically needs the NVIDIA driver's profiling restriction lifted (nvidia-smi -pm 1 or modprobe nvidia NVreg_RestrictProfilingToAdminUsers=0).

When to use

  • Correlating a GPU workload with the CPU, memory, and disk activity around it.
  • Studying inference/serving pipelines where host-side overheads (tokenization, data loading, PCIe transfers) matter.
  • Giving non-developers a knob (.pbtxt) to change what's collected without rebuilding.

For GPU-only runs with less machinery, use gemm_profiling.