Add CUPTI monitor profiler support - #70
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9847c9801b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if trace_path.exists(): | ||
| return |
There was a problem hiding this comment.
When the requested path already exists—for example, when rerunning an experiment with the same output name—the CUPTI exporter still writes the new data to <trace_path>.gz, but this check returns solely because the old target exists. The advertised path therefore retains the stale trace while the current run is left under an unexpected .gz name, silently associating previous profiling data with the new experiment.
Useful? React with 👍 / 👎.
| with gzip.open(monitor_path, "rb") as source, trace_path.open("wb") as destination: | ||
| destination.write(source.read()) |
There was a problem hiding this comment.
When cupti_monitor is combined with trace_format="chrome_json", the monitor backend's exporter produces a gzip-compressed native Perfetto protobuf trace; this branch merely decompresses those bytes rather than converting them to Chrome JSON. The resulting .json file is therefore not JSON, and with gzip_trace=True the uncompressed bytes are additionally written under a .json.gz suffix, so downstream JSON readers fail for every advertised Chrome-JSON monitor export.
Useful? React with 👍 / 👎.
9847c98 to
eb4754a
Compare
## Human Note ## Agent note Add an opt-in CUPTI monitor configuration to the shared profiler wrapper, including PM metric sampling, environment counters, CUDA Graph dependency recording, event-node attribution, and metric discovery. Constructing the profiler now prepares the backend immediately, which lets graph recorders arm before capture while preserving existing context-manager usage. Monitor traces use PyTorch's native Perfetto exporter and normalize its unconditional `.gz` suffix, so callers receive the requested `.pftrace` or JSON path instead of having to discover a differently named artifact. The optional dependency keeps cupti-python out of the default installation. ## Test Plan ```bash ~/.venvs/nightly/bin/python -m pytest test/test_profiler.py test/test_perfetto.py -q uvx ruff check transformer_nuggets/utils/benchmark.py transformer_nuggets/utils/__init__.py test/test_profiler.py uvx ruff format --check transformer_nuggets/utils/benchmark.py transformer_nuggets/utils/__init__.py test/test_profiler.py prek ```
eb4754a to
24d5580
Compare
1 participant
Human Note
Agent note
Add an opt-in CUPTI monitor configuration to the shared profiler wrapper, including PM metric
sampling, environment counters, CUDA Graph dependency recording, event-node attribution, and
metric discovery. Constructing the profiler now prepares the backend immediately, which lets graph
recorders arm before capture while preserving existing context-manager usage.
Monitor traces use PyTorch's native Perfetto exporter and normalize its unconditional
.gzsuffix,so callers receive the requested
.pftraceor JSON path instead of having to discover a differentlynamed artifact. The optional dependency keeps cupti-python out of the default installation.
The monitor path was also exercised on an NVIDIA B200 with native Perfetto output, PM counter
tracks, graph flows, and kernel annotations.
Test Plan
~/.venvs/nightly/bin/python -m pytest test/test_profiler.py test/test_perfetto.py -q uvx ruff check transformer_nuggets/utils/benchmark.py transformer_nuggets/utils/__init__.py test/test_profiler.py uvx ruff format --check transformer_nuggets/utils/benchmark.py transformer_nuggets/utils/__init__.py test/test_profiler.py prek