Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
131 commits
Select commit Hold shift + click to select a range
6cddeac
added a new way to trace python ttnn calls
dgomezTT Jan 15, 2026
2744301
added documentation and deleted some testing flags
dgomezTT Jan 15, 2026
50b09d7
addressed feedback
dgomezTT Jan 16, 2026
3c1c8ba
addressed copilot feedback so it stops spamming me
dgomezTT Jan 16, 2026
28f8822
forgot to uncomment some code
dgomezTT Jan 16, 2026
7ddeb66
addressed pr feedback + all the spam from copilot
dgomezTT Jan 21, 2026
598f8ba
addressed PR feedback
dgomezTT Jan 22, 2026
f041633
Merge branch 'main' into dgomeztt/python-tracer
Aswinmcw Jan 23, 2026
319bb14
fixing pipelines
dgomezTT Jan 23, 2026
96d4dc6
Merge branch 'main' into dgomeztt/python-tracer
Aswinmcw Jan 24, 2026
449afe3
Merge branch 'dgomeztt/python-tracer' into Aswinmcw/new_trace_infra
Aswinmcw Jan 24, 2026
536393c
fix
Aswinmcw Jan 25, 2026
762d3a8
fix
Aswinmcw Jan 25, 2026
31e81bb
fix
Aswinmcw Jan 25, 2026
b9b90a9
fix
Aswinmcw Jan 25, 2026
9a4ddac
Merge branch 'main' into dgomeztt/python-tracer
Aswinmcw Jan 25, 2026
de5f188
Merge branch 'dgomeztt/python-tracer' into Aswinmcw/new_trace_infra
Aswinmcw Jan 25, 2026
ee7ba60
fix
Aswinmcw Jan 25, 2026
ef6865f
fix
Aswinmcw Jan 26, 2026
66ba9e8
fix
Aswinmcw Jan 28, 2026
f67616a
Merge branch 'main' into dgomeztt/python-tracer
Aswinmcw Jan 28, 2026
357e0b6
Merge branch 'dgomeztt/python-tracer' into Aswinmcw/new_trace_infra
Aswinmcw Jan 28, 2026
719c923
fix
Aswinmcw Jan 28, 2026
af812c9
fix
Aswinmcw Jan 28, 2026
db85d7f
fix
Aswinmcw Jan 28, 2026
6ef1399
fix
Aswinmcw Jan 28, 2026
5275376
fix
Aswinmcw Jan 28, 2026
1d58910
fix
Aswinmcw Jan 28, 2026
5ee8e54
fix
Aswinmcw Jan 29, 2026
695c223
Merge branch 'main' into dgomeztt/python-tracer
Aswinmcw Jan 29, 2026
abbdede
Merge branch 'dgomeztt/python-tracer' into Aswinmcw/new_trace_infra
Aswinmcw Jan 29, 2026
16f1f64
Merge branch 'main' into dgomeztt/python-tracer
Aswinmcw Jan 29, 2026
46f0edb
fix
Aswinmcw Jan 29, 2026
95a3601
fix
Aswinmcw Jan 29, 2026
1cf410c
fix
Aswinmcw Jan 29, 2026
559b5ef
fix
Aswinmcw Jan 29, 2026
096d24c
fix
Aswinmcw Jan 30, 2026
5eebb60
Merge branch 'main' into dgomeztt/python-tracer
Aswinmcw Jan 31, 2026
5a94c8a
Merge branch 'dgomeztt/python-tracer' into Aswinmcw/new_trace_infra
Aswinmcw Jan 31, 2026
9e0d27f
Merge branch 'main' into dgomeztt/python-tracer
Aswinmcw Feb 2, 2026
948280b
Merge branch 'dgomeztt/python-tracer' into Aswinmcw/new_trace_infra
Aswinmcw Feb 2, 2026
b92e3ec
fix
Aswinmcw Feb 2, 2026
969915a
fix
Aswinmcw Feb 2, 2026
07e3187
fix
Aswinmcw Feb 2, 2026
bc6078c
Merge branch 'main' into dgomeztt/python-tracer
dgomezTT Feb 2, 2026
52751b1
Merge branch 'main' into dgomeztt/python-tracer
Aswinmcw Feb 3, 2026
fc5c5c5
Merge branch 'dgomeztt/python-tracer' into Aswinmcw/new_trace_infra
Aswinmcw Feb 3, 2026
f9cb6b8
fix
Aswinmcw Feb 3, 2026
82ed3fa
fix
Aswinmcw Feb 3, 2026
1d96960
fix
Aswinmcw Feb 3, 2026
0fb696e
fix
Aswinmcw Feb 3, 2026
c5212b6
fix
Aswinmcw Feb 3, 2026
9ef4193
fix
Aswinmcw Feb 3, 2026
0797d27
Merge branch 'main' into dgomeztt/python-tracer
Aswinmcw Feb 3, 2026
a4ab0e0
Merge branch 'dgomeztt/python-tracer' into Aswinmcw/new_trace_infra
Aswinmcw Feb 3, 2026
4a45d8d
fix
Aswinmcw Feb 3, 2026
85a74ad
added a new way to trace python ttnn calls
dgomezTT Jan 15, 2026
57c2e56
added documentation and deleted some testing flags
dgomezTT Jan 15, 2026
85852cd
addressed feedback
dgomezTT Jan 16, 2026
c63d096
addressed copilot feedback so it stops spamming me
dgomezTT Jan 16, 2026
9025d1c
forgot to uncomment some code
dgomezTT Jan 16, 2026
da64627
addressed pr feedback + all the spam from copilot
dgomezTT Jan 21, 2026
0d1aea0
addressed PR feedback
dgomezTT Jan 22, 2026
ac5415b
fixing pipelines
dgomezTT Jan 23, 2026
3a7aa13
Apply suggestion from @Copilot
dgomezTT Feb 3, 2026
219f297
Apply suggestions from code review
dgomezTT Feb 3, 2026
982d85e
fix
Aswinmcw Feb 4, 2026
1d59204
fix
Aswinmcw Feb 4, 2026
5b49ec9
fix
Aswinmcw Feb 4, 2026
5d4bed5
fix
Aswinmcw Feb 4, 2026
56e5221
Addressed all the comments in the PR
dgomezTT Feb 4, 2026
49c93ab
removed unnecessary data from the json files
dgomezTT Feb 4, 2026
d5ac94a
Reverted some code that was no longer modified
dgomezTT Feb 4, 2026
b274dff
Merge branch 'main' into dgomeztt/python-tracer
dgomezTT Feb 4, 2026
cad9144
Potential fix for pull request finding 'Unused local variable'
dgomezTT Feb 4, 2026
d7dabf4
fix
Aswinmcw Feb 5, 2026
c0e53b9
Merge branch 'dgomeztt/python-tracer' into Aswinmcw/new_trace_infra
Aswinmcw Feb 5, 2026
a14116d
fix
Aswinmcw Feb 5, 2026
6fb0519
fix
Aswinmcw Feb 5, 2026
d8d503f
fix
Aswinmcw Feb 6, 2026
ffca1a8
fix
Aswinmcw Feb 6, 2026
62b8146
fix
Aswinmcw Feb 6, 2026
7dcd6c8
fix
Aswinmcw Feb 6, 2026
7eaa343
fix
Aswinmcw Feb 6, 2026
f35cff4
fix
Aswinmcw Feb 6, 2026
4433066
fix
Aswinmcw Feb 6, 2026
5bef0ee
fix
Aswinmcw Feb 6, 2026
7824311
fix
Aswinmcw Feb 6, 2026
8ebfc9a
fix
Aswinmcw Feb 7, 2026
1fcdbad
fix
Aswinmcw Feb 7, 2026
4c15f93
Merge branch 'main' into dgomeztt/python-tracer
Aswinmcw Feb 7, 2026
43c7a93
Merge branch 'dgomeztt/python-tracer' into Aswinmcw/new_trace_infra
Aswinmcw Feb 7, 2026
a48437b
fix
Aswinmcw Feb 7, 2026
875f26c
fix
Aswinmcw Feb 9, 2026
2cbfe21
Merge branch 'main' into dgomeztt/python-tracer
Aswinmcw Feb 9, 2026
7f5daf0
Merge branch 'dgomeztt/python-tracer' into Aswinmcw/new_trace_infra
Aswinmcw Feb 9, 2026
e11567f
fix
Aswinmcw Feb 9, 2026
a0dc94e
Merge branch 'main' into Aswinmcw/new_trace_infra
Aswinmcw Feb 9, 2026
448d5f0
fix
Aswinmcw Feb 9, 2026
c8e00eb
fix
Aswinmcw Feb 9, 2026
7be9ea7
Merge branch 'main' into Aswinmcw/new_trace_infra
Aswinmcw Feb 10, 2026
da0eae5
fix
Aswinmcw Feb 10, 2026
fa92202
fix
Aswinmcw Feb 10, 2026
09a77ff
fix
Aswinmcw Feb 10, 2026
e57c86b
Merge branch 'main' into Aswinmcw/new_trace_infra
Aswinmcw Feb 11, 2026
a71a165
fix
Aswinmcw Feb 11, 2026
b8c8ac8
fix
Aswinmcw Feb 11, 2026
644e127
fix
Aswinmcw Feb 11, 2026
7554a70
fix
Aswinmcw Feb 11, 2026
7cb6b1e
fix
Aswinmcw Feb 11, 2026
4236ffd
Merge branch 'main' into Aswinmcw/new_trace_infra
Aswinmcw Feb 11, 2026
c4486ea
fix
Aswinmcw Feb 11, 2026
ffac818
fix
Aswinmcw Feb 11, 2026
07dde7f
fix
Aswinmcw Feb 11, 2026
4af8e26
fix
Aswinmcw Feb 11, 2026
06ecf61
fix
Aswinmcw Feb 11, 2026
ef9bd79
fix
Aswinmcw Feb 11, 2026
c556393
Merge branch 'main' into Aswinmcw/new_trace_infra
Aswinmcw Feb 11, 2026
76acd9f
fix
Aswinmcw Feb 11, 2026
c4e3ff2
Merge branch 'main' into Aswinmcw/new_trace_infra
Aswinmcw Feb 11, 2026
2310558
Merge branch 'main' into Aswinmcw/new_trace_infra
Aswinmcw Feb 12, 2026
f5ca996
Merge branch 'main' into Aswinmcw/new_trace_infra
Aswinmcw Feb 12, 2026
610cf6f
fix
Aswinmcw Feb 12, 2026
5f249e5
Merge branch 'main' into Aswinmcw/new_trace_infra
Aswinmcw Feb 13, 2026
88ac6fb
fix
Aswinmcw Feb 13, 2026
9bee653
Merge branch 'main' into Aswinmcw/new_trace_infra
Aswinmcw Feb 14, 2026
729351d
fix
Aswinmcw Feb 14, 2026
808de9f
fix
Aswinmcw Feb 14, 2026
1b13c6d
Merge branch 'main' into Aswinmcw/new_trace_infra
Aswinmcw Feb 16, 2026
ae95230
fix
Aswinmcw Feb 16, 2026
f20cb62
fix
Aswinmcw Feb 16, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 18 additions & 5 deletions .github/workflows/ttnn-run-sweeps.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -734,18 +734,31 @@ jobs:
run: |
pip install psycopg2-binary
- name: Generate JSON from database
- name: Generate JSON from database (V1 - legacy)
env:
TTNN_OPS_DATABASE_URL: ${{ secrets.TTNN_OPS_LEAD_MODELS_DATABASE_URL }}
run: |
# Use the centralized reconstruction script
python tests/sweep_framework/load_ttnn_ops_data_v2.py reconstruct model_tracer/traced_operations/ttnn_operations_master_reconstructed.json
# Reconstruct from ttnn_ops schema (V1 format for backward compatibility)
python tests/sweep_framework/load_ttnn_ops_data.py reconstruct \
model_tracer/traced_operations/ttnn_operations_master_reconstructed.json \
ttnn_ops
- name: Upload master JSON artifact
- name: Generate JSON from database (V2 - new format)
env:
TTNN_OPS_DATABASE_URL: ${{ secrets.TTNN_OPS_LEAD_MODELS_DATABASE_URL }}
run: |
# Reconstruct from ttnn_ops_v2 schema (V2 format with per-tensor placement)
python tests/sweep_framework/load_ttnn_ops_data_v2.py reconstruct \
model_tracer/traced_operations/ttnn_operations_master_v2_reconstructed.json \
ttnn_ops_v2
- name: Upload master JSON artifacts
uses: actions/upload-artifact@v4
with:
name: ttnn-operations-master-json
path: model_tracer/traced_operations/ttnn_operations_master_reconstructed.json
path: |
model_tracer/traced_operations/ttnn_operations_master_reconstructed.json
model_tracer/traced_operations/ttnn_operations_master_v2_reconstructed.json
retention-days: 1

ttnn-generate-sweeps:
Expand Down
30 changes: 27 additions & 3 deletions model_tracer/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,19 @@ Automatically extracts real-world operation configurations from model tests and
- ✅ Simple 2-step integration into sweep tests
- ✅ Captures shapes, dtypes, layouts, and exact shard specs

## New Simple Tracer (Updated Jan 2026)

The tracer has been updated to use the new `--trace-params` flag instead of graph tracing. This provides:
- ✅ Simpler implementation (no C++ graph tracing dependencies)
- ✅ Individual JSON files per operation for easier debugging
- ✅ Works with both pytest tests and standalone Python scripts

**How it works:**
1. Runs your test/script with `--trace-params` flag
2. Collects individual operation JSON files from `generated/ttnn/reports/operation_parameters/`
3. Filters valid operations (from `Allops.txt`)
4. Merges into master JSON with deduplication and source tracking

### Currently Traced Models

The following models have been traced and their configurations are available in `ttnn_operations_master.json`:
Expand Down Expand Up @@ -76,7 +89,7 @@ These traced configurations provide real-world operation patterns from productio

### Key Files

- **Tracer**: `model_tracer/generic_ops_tracer.py` - Employs methodology described in the [graph tracing tech report](https://github.com/tenstorrent/tt-metal/blob/main/tech_reports/ttnn/graph-tracing.md)
- **Tracer**: `model_tracer/generic_ops_tracer.py` - Employs methodology described in the [operation tracing tech report](https://github.com/tenstorrent/tt-metal/blob/main/tech_reports/ttnn/operation-tracing.md)
- **Master JSON**: `model_tracer/traced_operations/ttnn_operations_master.json` - Contains all traced configurations
- **Analyzer**: `model_tracer/analyze_operations.py` - Query and view configurations
- **Config Loader**: `tests/sweep_framework/master_config_loader.py` - Converts JSON configs to sweep test parameters
Expand Down Expand Up @@ -237,6 +250,17 @@ python model_tracer/generic_ops_tracer.py models/experimental/some_model/run_inf

# Keep trace files (default: auto-deleted after adding to master)
python model_tracer/generic_ops_tracer.py <test_path> --store

# Import traces from another machine
# 1. On Machine A: Trace with --store to keep JSON files
python model_tracer/generic_ops_tracer.py test.py --store
# This creates: generated/ttnn/reports/operation_parameters/<test_name>_<timestamp>/

# 2. Copy trace directory to Machine B
# scp -r generated/ttnn/reports/operation_parameters/<test_name>_<timestamp>/ machine_b:/tmp/traces/

# 3. On Machine B: Import the traces
python model_tracer/generic_ops_tracer.py --from-trace-dir /tmp/traces/<test_name>_<timestamp>
```

**Detection Logic:**
Expand Down Expand Up @@ -287,11 +311,11 @@ python model_tracer/analyze_operations.py sigmoid_accurate
```bash
# Generate test vectors
python3 tests/sweep_framework/sweeps_parameter_generator.py \
--module-name model_traced.pad_model_traced
--module-name model_traced.add_model_traced

# Run model_traced suite
python3 tests/sweep_framework/sweeps_runner.py \
--module-name model_traced.pad_model_traced \
--module-name model_traced.add_model_traced \
--suite model_traced \
--vector-source vectors_export \
--result-dest results_export
Expand Down
Loading
Loading