Merge pull request #3 from peixuana/RenderStage_split_byProcess

Add tools to split RenderStage data by process in Perfetto Trace file

Author: peixuana

.gitignore

@@ -12,3 +12,6 @@ wheels/
 # Input/output data files
 input/
 output/
+
+# Local AI tool state
+.claude/

CLAUDE.md

@@ -0,0 +1,54 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Environment
+
+- Python 3.13+ required. Dependencies managed by [`uv`](https://docs.astral.sh/uv/); `uv sync` creates `.venv` and installs dev deps (pytest).
+- Runtime dependencies list is empty (stdlib only); pytest is the sole dev dependency.
+
+## Common commands
+
+```bash
+uv sync                                   # create venv, install deps
+uv run main.py --help                     # list registered tools
+uv run main.py converter <csv> [-o out.json | --output-dir dir/]
+uv run main.py split-renderstage <input.pftrace> <output.pftrace>
+uv run main.py split-renderstage --debug <input.pftrace>
+
+uv run python -m tools.converter.convert_sdp_csv_to_perfetto_json <csv>   # bypass dispatcher
+uv run python -m tools.converter.split_renderstage_by_process <input.pftrace> <output.pftrace>
+uv run split-renderstage-by-process <input.pftrace> <output.pftrace>      # console script
+
+uv run pytest                             # full suite
+uv run pytest -v
+uv run pytest tests/test_convert_sdp_csv_to_perfetto_json.py::TestExampleCSV -v
+```
+
+Commits must be DCO-signed (`git commit -s`). External PRs are scanned by Semgrep in CI.
+
+On Windows, if `uv run` fails while trying to remove `.venv\lib64` with `Access is denied`, close terminals/processes using `.venv`, remove and recreate `.venv` with `uv sync`, or use direct `python ...` invocation as a stdlib-only workaround for tools without runtime dependencies.
+
+## Architecture
+
+**Dispatcher pattern.** `main.py` is a thin CLI dispatcher. `TOOLS_REGISTRY` maps a tool name (e.g. `converter`) to a dotted module path; the dispatcher imports the module, rewrites `sys.argv` to hide its own name, then calls the module's `main()`. To add a new tool: create `tools/<name>/<module>.py` exposing `main()` that uses argparse, then add one line to `TOOLS_REGISTRY`. Each tool is independently invokable via `python -m ...` and is also registered as a console script in `pyproject.toml` (`[project.scripts]`).
+
+**Converter (`tools/converter/convert_sdp_csv_to_perfetto_json.py`).** The SDP CSV → Perfetto JSON pipeline auto-detects the CSV variant by inspecting the header row:
+
+- **Realtime** header has `Category`, `Metric`, `Timestamp` → `convert_realtime()`. Emits only counter events (`ph: "C"`). Synthesizes PIDs starting at `800000`, one per `(Process, Category)` pair.
+- **Trace** header has `Group`, `Track`, `TimestampStart` → `convert_trace()`. Emits duration slices (`ph: "X"`) and counters (`ph: "C"`). Uses *real* PIDs from the CSV for `Process` rows and synthesizes PIDs starting at `900000` for `System` rows (one per `Group`). Trace-mode conversion does a two-pass read: pass 1 finds main-thread names (rows where `TID == PID`) to label processes; pass 2 emits events.
+
+Counter vs slice disambiguation in trace mode is driven by the `Track` column: tracks containing `CounterValue` are explicit counters; rows with both `TimestampStart` and `TimestampEnd` are slices; rows with only one timestamp are treated as implicit counters.
+
+Output always wraps events as `{"traceEvents": [...]}` with metadata events (`ph: "M"`, `process_name`/`thread_name`) prepended. `_write_perfetto()` is the single write path; `_metadata_events()` is the single metadata builder — reuse these when adding new event sources rather than open-coding JSON structure.
+
+**Batch / glob handling.** `resolve_input_files()` expands glob patterns and deduplicates by normalized path. When multiple inputs are given, `-o` is ignored (warned) and each input is written next to itself (or into `--output-dir`) with the basename + `.json`.
+
+**RenderStage splitter (`tools/converter/split_renderstage_by_process.py`).** This tool reads binary Perfetto `.pftrace` files using stdlib-only protobuf decode/encode helpers. It extracts `GpuRenderStageEvent` packets, resolves each event to a process, groups slices by process, then appends new process-scoped RenderStage tracks to the output trace while preserving the original trace content.
+
+Process ownership resolution is prioritized as: packet `trusted_pid`, stage-spec descriptions like `Process <name> [<pid>]`, Vulkan queue submission IDs, Vulkan debug-utils device context, then GPU context fallback. Debug mode (`--debug <input.pftrace>`) scans packet fields, trusted PIDs, track descriptors, interned names, and sample RenderStage events without writing an output file.
+
+## Tests
+
+- Live in `tests/test_convert_sdp_csv_to_perfetto_json.py`. `test_main.py` is a backward-compat shim that re-exports the same tests so `pytest test_main.py` still works.
+- `tests/__init__.py` and `tools/__init__.py` exist so tests can import `tools.converter...` directly — keep them.

README.md

@@ -26,6 +26,12 @@ This converter supports CSV files exported from both SDP capture modes:
 | System scheduling slices | `"ph": "X"` | Kernel scheduling events (e.g., Sched CPU) |
 | Metadata | `"ph": "M"` | Auto-generated `process_name` and `thread_name` labels |
 
+### RenderStage Splitter
+
+Split GPU RenderStage events in a Perfetto `.pftrace` by process. The tool reads a Perfetto trace, extracts `GpuRenderStageEvent` packets, resolves each event to a process when possible, and writes an augmented `.pftrace` with separate RenderStage tracks per process for easier analysis in [Perfetto UI](https://ui.perfetto.dev/).
+
+The tool also includes a debug mode that scans the input trace and prints packet, process, track, and RenderStage diagnostics.
+
 ## Installation Instructions
 
 ```bash
@@ -50,6 +56,9 @@ uv run main.py --help
 
 # Run a specific tool (e.g., converter)
 uv run main.py converter <tool-specific arguments ...>
+
+# Run the RenderStage splitter
+uv run main.py split-renderstage <input.pftrace> <output.pftrace>
 ```
 
 ### Converter Examples
@@ -77,6 +86,25 @@ You can also invoke the converter directly (bypassing the dispatcher):
 uv run python -m tools.converter.convert_sdp_csv_to_perfetto_json trace1.csv
 ```
 
+### RenderStage Splitter Examples
+
+```bash
+# Split GPU RenderStage events into process-specific tracks
+uv run main.py split-renderstage input.pftrace output.pftrace
+
+# Inspect trace contents without writing an output file
+uv run main.py split-renderstage --debug input.pftrace
+```
+
+You can also invoke the RenderStage splitter directly (bypassing the dispatcher):
+
+```bash
+uv run python -m tools.converter.split_renderstage_by_process input.pftrace output.pftrace
+
+# Or use the uv-installed console script
+uv run split-renderstage-by-process input.pftrace output.pftrace
+```
+
 ## Development
 
 To contribute new features or fixes, please see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines on branching, submitting pull requests, and coding standards.

main.py

@@ -18,6 +18,7 @@
 Examples:
     python main.py converter input.csv -o output.json
     python main.py converter input1.csv input2.csv --output-dir output/
+    python main.py split-renderstage input.pftrace output.pftrace
 """
 
 import importlib
@@ -29,6 +30,7 @@
 # ---------------------------------------------------------------------------
 TOOLS_REGISTRY: dict[str, str] = {
     "converter": "tools.converter.convert_sdp_csv_to_perfetto_json",
+    "split-renderstage": "tools.converter.split_renderstage_by_process",
 }
 
 
@@ -61,6 +63,7 @@ def _print_help():
         print(f"  {name:20s} → {mod}")
     print("\nPass -h/--help after a tool name for tool-specific help.")
     print("Example: python main.py converter --help")
+    print("Example: python main.py split-renderstage --debug input.pftrace")
 
 
 if __name__ == "__main__":

pyproject.toml

@@ -9,6 +9,7 @@ dependencies = []
 [project.scripts]
 snapdragon-profiler-ecosystem = "main:main"
 sdp-csv-to-perfetto-json = "tools.converter.convert_sdp_csv_to_perfetto_json:main"
+split-renderstage-by-process = "tools.converter.split_renderstage_by_process:main"
 
 [dependency-groups]
 dev = ["pytest>=8.0"]