Skip to content

Commit 3c2505d

Browse files
committed
feat: integrate Apple Instruments (xctrace) for Metal GPU profiling
- Add xctrace.py module for wrapping Apple Instruments profiling - Update Python CLI with --xctrace, --xctrace-template, --keep-trace options - Update C++ CLI with xctrace support on macOS - Add comprehensive Python examples for GPU profiling scenarios - Update README with xctrace documentation and usage examples New features: - Real Metal GPU event capture via xctrace (13K+ events) - Automatic event parsing from Instruments trace files - Support for multiple Instruments templates - Perfetto export from xctrace captures New examples: - kernel_timing_stats.py: Kernel execution time statistics - pytorch_profiling.py: PyTorch integration - pytorch_hooks_profiling.py: Layer-by-layer profiling with hooks - memory_profiling.py: GPU memory tracking - multi_gpu_profiling.py: Multi-GPU topology and profiling - realtime_tracing.py: Lock-free real-time tracing - transformers_profiling.py: LLM/Transformer profiling
1 parent 510183c commit 3c2505d

16 files changed

Lines changed: 5481 additions & 60 deletions

README.md

Lines changed: 70 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@
2020
- **High-Performance Event Capture**: Collect 10,000+ GPU instruction-level call stacks without interrupting execution
2121
- **Lock-Free Ring Buffer**: Minimal overhead event collection using SPSC (Single Producer Single Consumer) design
2222
- **SBT Binary Trace Format**: Compact, efficient binary format with string interning and delta timestamp encoding
23-
- **Multi-Platform Support**: NVIDIA CUDA (via CUPTI), AMD ROCm, Apple Metal (planned)
23+
- **Multi-Platform Support**: NVIDIA CUDA (via CUPTI), AMD ROCm, Apple Metal + Instruments (xctrace)
2424
- **Multi-GPU & Multi-Stream**: Full support for complex GPU topologies and async execution
2525
- **Multi-GPU Cluster Profiling** (v0.7.x): GPUTopology discovery, TimeSync (NTP/PTP/CUDA), NCCLTracker for distributed training
2626
- **Perfetto SDK Integration**: Native protobuf export (85% smaller files) + JSON fallback
@@ -54,6 +54,7 @@
5454
|----------|---------|--------|
5555
| NVIDIA | CUPTI SDK | ✅ Production |
5656
| Apple | Metal API | ✅ Production |
57+
| Apple | Instruments (xctrace) | ✅ Production |
5758
| AMD | ROCm | 🔜 Coming Soon |
5859
| Linux | eBPF | ✅ Available |
5960

@@ -275,6 +276,7 @@ TraceSmith provides a comprehensive CLI with ASCII banner and colored output:
275276

276277
| Command | Description |
277278
|---------|-------------|
279+
| `profile` | **Profile a command** (record + execute in one step) |
278280
| `record` | Record GPU events to a trace file |
279281
| `view` | View contents of a trace file |
280282
| `info` | Show detailed information about a trace file |
@@ -288,6 +290,16 @@ TraceSmith provides a comprehensive CLI with ASCII banner and colored output:
288290
**C++ CLI Examples:**
289291

290292
```bash
293+
# Profile a Python script (records GPU events during execution)
294+
./bin/tracesmith profile -- python train.py
295+
./bin/tracesmith profile -o model.sbt -- python train.py --epochs 10
296+
./bin/tracesmith profile --perfetto -- ./my_cuda_app
297+
298+
# Use Apple Instruments (xctrace) for real Metal GPU events on macOS
299+
./bin/tracesmith profile --xctrace -- python train.py
300+
./bin/tracesmith profile --xctrace --keep-trace -- python mps_benchmark.py
301+
./bin/tracesmith profile --xctrace --xctrace-template "GPU Driver" -- ./app
302+
291303
# Record a trace (auto-detect GPU platform)
292304
./bin/tracesmith record -o trace.sbt -d 5
293305

@@ -319,6 +331,15 @@ TraceSmith provides a comprehensive CLI with ASCII banner and colored output:
319331
**Python CLI Examples:**
320332

321333
```bash
334+
# Profile a command (record + execute in one step)
335+
tracesmith-cli profile -- python train.py
336+
tracesmith-cli profile -o model.sbt -- python train.py --epochs 10
337+
tracesmith-cli profile --perfetto -- python inference.py
338+
339+
# Use Apple Instruments (xctrace) for real Metal GPU events on macOS
340+
tracesmith-cli profile --xctrace -- python train.py
341+
tracesmith-cli profile --xctrace --keep-trace -- python mps_benchmark.py
342+
322343
# Show system info
323344
tracesmith-cli info
324345

@@ -341,6 +362,54 @@ tracesmith-cli analyze trace.sbt
341362
tracesmith-cli replay trace.sbt --mode dry-run
342363
```
343364

365+
#### macOS Metal GPU Profiling with xctrace
366+
367+
On macOS, TraceSmith integrates with Apple Instruments (xctrace) for capturing real Metal GPU events. This provides accurate GPU timing and event capture that the Metal Frame Capture API cannot achieve programmatically.
368+
369+
**Why use xctrace?**
370+
- Captures real Metal GPU execution events (kernel launches, command buffer submissions)
371+
- Accurate GPU timing from hardware counters
372+
- Works with any Metal application (PyTorch MPS, TensorFlow Metal, custom Metal apps)
373+
374+
**Usage:**
375+
376+
```bash
377+
# Python CLI (recommended - includes event parsing)
378+
tracesmith-cli profile --xctrace -- python train.py
379+
tracesmith-cli profile --xctrace --keep-trace -o model.sbt -- python inference.py
380+
tracesmith-cli profile --xctrace --perfetto -- python benchmark.py
381+
382+
# C++ CLI (calls xctrace, outputs raw .trace file)
383+
./bin/tracesmith profile --xctrace -- python train.py
384+
./bin/tracesmith profile --xctrace --xctrace-template "GPU Driver" -- ./app
385+
386+
# Python API
387+
from tracesmith.xctrace import XCTraceProfiler, profile_with_xctrace
388+
389+
# Simple usage
390+
events, trace_file = profile_with_xctrace(
391+
["python", "train.py"],
392+
duration=60,
393+
template="Metal System Trace"
394+
)
395+
396+
# Full control
397+
profiler = XCTraceProfiler()
398+
events = profiler.profile_command(["python", "train.py"])
399+
profiler.export_perfetto("metal_trace.json")
400+
```
401+
402+
**Available Templates:**
403+
- `Metal System Trace` - Most detailed Metal profiling (default)
404+
- `GPU Driver` - Driver-level analysis
405+
- `Game Performance` - Frame rate and GPU time
406+
- `Animation Hitches` - Animation performance
407+
408+
**Output:**
409+
- SBT file with parsed GPU events
410+
- Optional: Raw `.trace` file (use `--keep-trace`) for viewing in Instruments
411+
- Optional: Perfetto JSON export (use `--perfetto`)
412+
344413
#### C++ API
345414

346415
```cpp

0 commit comments

Comments
 (0)