Skip to content

optimizer details (draft) #2754

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

optimizer details (draft) #2754

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions docs/src/optimizer.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# Using the AI Compiler Optimizer with TT-Forge-FE

TT-Forge-FE uses the TT-MLIR AI compiler optimizer. It is responsible for optimizing how operations run on a Tenstorrent chip for maximum performance. Its key purposes are optimizing tensor memory layouts and selecting optimal operation configurations. This page provides resources for using it in code for deploying a trained model with TT-Forge-FE.

## Prerequisites

To use the AI compiler optimizer with TT-Forge-FE, you need the following:
* TT-Forge set up, using the [Getting Started with Building TT-Forge-FE From Source](getting_started_build_from_source.md) instructions
Copy link
Contributor

@odjuricicTT odjuricicTT Aug 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think a build from source is needed just for setting this config.

* A physical Tenstorrent device

You can review the [Optimizer page for TT-MLIR](https://docs.tenstorrent.com/tt-mlir/optimizer.html) for further details about how it works with TT-MLIR.

## Configuring the AI Compiler Optimizer for TT-Forge-FE
This section covers how to use the AI Compiler optimizer with TT-Forge-FE. The [**resnet_hf.py**](https://github.com/tenstorrent/tt-forge-fe/blob/odjuricic/simple-perf-test/forge/test/benchmark/benchmark/models/resnet_hf.py) benchmarking sample is used.

The key part of the code for the optimizer is this:

```python
# Turn on MLIR optimizations
compiler_cfg.mlir_config = (
MLIRConfig()
.set_enable_optimizer(True)
.set_enable_fusing(True)
Copy link
Contributor

@odjuricicTT odjuricicTT Aug 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is on by default and is no longer needed.

.set_enable_fusing_conv2d_with_multiply_pattern(True)
.set_enable_memory_layout_analysis(False)
Comment on lines +22 to +25
Copy link
Contributor

@svuckovicTT svuckovicTT Aug 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider using ANCHOR links.

Example:
doc: https://github.com/tenstorrent/tt-mlir/blob/05c95b344a6f8622b357c6f7208297acf5dc447f/docs/src/adding-an-op.md?plain=1#L272
code: https://github.com/tenstorrent/tt-mlir/blob/05c95b344a6f8622b357c6f7208297acf5dc447f/include/ttmlir/Target/TTNN/CMakeLists.txt#L17-L19

Copy link
Contributor

@svuckovicTT svuckovicTT Aug 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Context: this way, we wouldn't need to update the docs each time the options change - they would be read directly from code. Ideally, we also communicate default values by reading them from code.

)
```

After you configure your compiler, you can call `compiler_cfg.mlir_config` to set up the optimizer. By default, the optimizer is off. In this example, the following configuration options are used:
* `.set_enable_optimizer(True)` - Enables the optimizer for use.
* `.set_enable_fusing(True)` - Enables general operator fusion. This reduces kernel launches, lowers memory reads/writes, and improves cache usage.
* `.set_enable_fusing_conv2d_with_multiply_pattern(True)` - This specifically looks for `Conv2D` followed by `Multiply` and tries to fuse them into a single kernel.
* `.set_enable_memory_layout_analysis(False)` - This disables the compiler from optimizing or transforming how tensors are laid out in memory.

## Running resnet_hf.py
Loading