Skip to content

Commit a79b53a

Browse files
chandrasekaranpradeepjmcgrathTT
authored andcommitted
Add nearest-interpolation decomposition for floating scalar factors, downsampling, and mixed sampling (resize1d & resize2d) (#2885)
Fixes #2820 Fixes #2017 This PR adds decomposition support for **nearest** interpolation in: - `resize2d` — supports **floating scalar factors**, **pure downsampling**, and **mixed sampling** (upsample in one axis, downsample in the other). - `resize1d` — supports **floating scalar factors** and **downsampling**. The decomposition rewrites nearest-neighbor into two primitive operations: 1. **Upsampling** via `repeat_interleave` (integer repetition). 2. **Downsampling** via **depthwise `conv2d`** with a pick-first kernel (select the first element in each stride window). Nearest-neighbor scaling with non-integer (floating) scale factors can be represented exactly by factoring input and output sizes using their greatest common divisor (gcd). This produces integer `up` and `down` factors for each axis: - `g = gcd(out_size, in_size)` - `up = out_size / g` → how many times each source element must be repeated - `down = in_size / g` → how many source elements are collapsed into one output Algorithm (applied per spatial axis): 1. Compute `g = gcd(out_size, in_size)`. 2. `up = out_size / g` — if `up > 1`, call `repeat_interleave(..., repeats=up, dim=axis)` to expand each element `up` times on that axis. 3. `down = in_size / g` — if `down != 1`, apply a **depthwise `conv2d`** with: - a kernel shaped to pick the first element in the conv window, - stride = `down`, - no padding, no dilation, - `groups = C` (depthwise conv). Why this reproduces nearest-neighbor: - `repeat_interleave` expands each input element into an integer block of identical values (upsample). - The depthwise conv kernel contains zeros except a single `1` at its top-left (first) position; with stride=`down`, the conv selects the first element from each `down`-sized block (pick-first semantics). Together, expand → pick-first collapses create the exact top-left variant of nearest-neighbor for any rational scale factor. Implementation notes: - For 2D: - Upsampling applied independently on H and W (code applies repeat H then repeat W). - Width-downsample kernel: `(C, 1, 1, down_w)`, stride `(1, down_w)`. - Height-downsample kernel: `(C, 1, down_h, 1)`, stride `(down_h, 1)`. - For 1D: - Temporarily `unsqueeze` `(N, C, W)` → `(N, C, 1, W)` to reuse the same conv2d path, then `squeeze` back to 3D. **Input:** `(N=1, C=3, H_in=5, W_in=7)` **Target:** `size = (H_out=9, W_out=5)` Per-dim gcd decomposition: - H: `g_h = gcd(9, 5) = 1` → `up_h = 9`, `down_h = 5` - W: `g_w = gcd(5, 7) = 1` → `up_w = 5`, `down_w = 7` ``` Input: (1, 3, 5, 7) │ ▼ repeat_interleave(dim=H, repeats=up_h=9) ▼ Shape: (1, 3, 45, 7) # each row repeated 9x │ ▼ repeat_interleave(dim=W, repeats=up_w=5) ▼ Shape: (1, 3, 45, 35) # each column repeated 5x │ ▼ conv2d(depthwise, kernel=(C,1,1,down_w=7), stride=(1,7)) → picks first column in each block of width 7 ▼ Shape: (1, 3, 45, 5) # width reduced to W_out=5 │ ▼ conv2d(depthwise, kernel=(C,1,down_h=5,1), stride=(5,1)) → picks first row in each block of height 5 ▼ Output: (1, 3, 9, 5) # height reduced to H_out=9 ``` **Input:** `(1, 3, W_in=9)` **Target:** `size = (W_out=7,)` Compute: - `g = gcd(9, 7) = 1` → `up_w = 7`, `down_w = 9` ``` Input: (1, 3, 9) │ ▼ repeat_interleave(dim=W, repeats=up_w=7) ▼ Shape: (1, 3, 63) # every element repeated 7x │ ▼ unsqueeze to 4D for conv2d: Temp: (1, 3, 1, 63) │ ▼ conv2d(depthwise, kernel=(C,1,1,down_w=9), stride=(1,9)) → picks first element in each block of width 9 ▼ Shape: (1, 3, 1, 7) │ ▼ squeeze back to 3D: Output: (1, 3, 7) ``` RuntimeError: TT_ASSERT @ /__w/tt-forge-fe/tt-forge-fe/forge/csrc/ops/op_resize_2d.cpp:104: (input_h % size_h == 0) && (input_w % size_w == 0) info: info: This occurred because the original implementation only supported integer downsampling ratios. - **GHOSTNET** - PyTorch: `forge/test/models/pytorch/vision/ghostnet/test_ghostnet.py::test_ghostnet_timm[ghostnetv2_100.in1k]` - **DETR** - ONNX: `forge/test/models/onnx/vision/detr/test_detr.py::test_detr_segmentation_onnx[facebook/detr-resnet-50-panoptic]` - PyTorch: `forge/test/models/pytorch/vision/detr/test_detr.py::test_detr_segmentation[resnet_50_panoptic]` `forge/test/models/pytorch/vision/detr/test_detr.py::test_detr_detection[resnet_50]` - **SSD300_VGG16** - ONNX: `forge/test/models/onnx/vision/ssd300_vgg16/test_ssd300_vgg16_onnx.py::test_ssd300_vgg16[ssd300_vgg16]` - PyTorch: `forge/test/models/pytorch/vision/ssd300_vgg16/test_ssd300_vgg16.py::test_ssd300_vgg16[ssd300_vgg16]` - **GHOSTNET**: Passing end-to-end, `xfail` markers removed. - **DETR** & **SSD300_VGG16**: Resize issue resolved, but failing with **Out of Memory (OOM)** at runtime. - Additional fix: Corrected DETR input data format issues in `tt_forge_models` repo. [test_detr.log](https://github.com/user-attachments/files/22041939/test_detr.log) [test_ghostnet_timm.log](https://github.com/user-attachments/files/22041942/test_ghostnet_timm.log) [test_resize2d_nearest_interpolation.log](https://github.com/user-attachments/files/22041943/test_resize2d_nearest_interpolation.log) [test_ssd300_vgg16_onnx.log](https://github.com/user-attachments/files/22041944/test_ssd300_vgg16_onnx.log) [test_ssd300_vgg16.log](https://github.com/user-attachments/files/22041945/test_ssd300_vgg16.log) Add Xfail Marker (#2942) [Issue_1](#2940) [Issue_2](#2888) Facing Crash issue in nightly pipeline Added Xfail Marker cc: @nvukobratTT Fix the albert & distilbert postprocessing and invalid config issue (#2858) [Issue](#2870) Due to the framework_models variable which is reused for the storing onnx model which causes the config issue and updated the correct logic for the albert and distilbert postprocessing and modified the regnet and dla to follow the export script based on the tt_forge_models The testscripts of the dla,regnet,albert,distilbert has been modified. - [x] New/Existing tests provide coverage for changes Fix model analysis failures (#2937) This PR fixes multiple issues in **model analysis** and **models ops tests**: - Resolves configuration extraction failure for the `speecht5` model. - Fixes failures in `cognito`, `segformer`, and `whisper` models. - Adds generated models ops tests for `speecht5`, `cognito`, and `unet`. - Adds `xfail` markers for known failing models ops tests. - Updates `resize2d` models ops tests to align with the [nearest-interpolation decomposition uplift](#2885). 1. **Speecht5 Config Extraction Fix** - Issue: While extracting unique ops config for `speecht5`, the `multiply` op had an empty tuple `()` as input shape for the parameter node type. - Root Cause: The input shape in the TVM JSON graph was empty because it represents a scalar tensor. - Fix: Updated the ops config extraction function to use the actual parameter tensor when the node shape is an empty tuple. 2. **Cognito Model Fix** - Issue: Failure with ``` huggingface_hub.errors.HFValidationError: Repo id must use alphanumeric chars or '-', '_', '.', '--' and '..' are forbidden, '-' and '.' cannot start or end the name, max length is 96 ``` - Root Cause: After porting model/tokenizer loading to `tt_forge_models`, the model was also being passed into `CogitoWrapper`, causing duplicate loading and invalid repo IDs. - Fix: - Removed redundant `model_utils` file for model/input sample loading. - Removed `CogitoWrapper` from test cases. - Applied similar fixes for the Cognito ONNX model. 3. **Whisper Model Fix** - Issue: ``` framework_model = Wrapper(framework_model) UnboundLocalError: local variable 'framework_model' referenced before assignment ``` - Fix: Corrected initialization by passing the model variable directly into the wrapper. 4. **Generated Models Ops Tests** - Added ops tests for: - `speecht5` - `cognito` - `unet` - Since config extraction issues are now resolved. - Added `xfail` markers for known failing models ops tests. 5. **Resize2D Models Ops Tests Update** - Updated skipped tests with `xfail` markers for failing cases. - Aligned with the uplift of [nearest-interpolation decomposition patch](#2885). Fix group and priority property recording (#2941) This PR fixes incorrect or missing **model group** and **priority** assignments by updating the `ModelGroup` and `ModelPriority` properties for Qwen v3, MobileNet v2, Swin, and Falcon models. - **From `GENERALITY → RED`** - **From `P2 → P1`** - `Qwen/Qwen3-Embedding-8B` - `Qwen/Qwen3-Embedding-4B` - `swin_v2_s` - `Falcon_1B` - `Falcon_3B` - `Falcon_7B` - `Falcon_10B` - `mobilnet_v2` [Deps] Upgrades Jax to 0.7 and python to 3.11 (#2918) Fixes : [Link to Github Issue](#2914) - [x] Upgrade Jax to version 0.7. - [x] Upgrade Python to version 3.11. - [x] includes mlir uplift to latest main - [x] New/Existing tests provide coverage for changes
1 parent bc10933 commit a79b53a

File tree

75 files changed

+3536
-307
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

75 files changed

+3536
-307
lines changed

.github/Dockerfile.base

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,8 +8,6 @@ ENV DEBIAN_FRONTEND=noninteractive
88
RUN apt-get update && apt-get install -y \
99
software-properties-common \
1010
build-essential \
11-
python3-dev \
12-
python3-venv \
1311
python3-pip \
1412
git \
1513
git-lfs \
@@ -61,6 +59,15 @@ RUN set -eux; \
6159
apt-get install -f -y "$TMP_DIR/ompi.deb" && \
6260
rm -rf "$TMP_DIR"
6361

62+
# Install deadsnakes PPA for latest Python versions
63+
RUN add-apt-repository ppa:deadsnakes/ppa && \
64+
apt-get update && \
65+
apt-get install -y \
66+
python3.11 \
67+
python3.11-dev \
68+
python3.11-venv \
69+
python3-pip
70+
6471
# Install python packages
6572
RUN pip install cmake \
6673
pytest

.github/Dockerfile.ci

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ SHELL ["/bin/bash", "-c"]
77
ENV PROJECT_NAME=tt-forge-fe
88
ENV TTMLIR_TOOLCHAIN_DIR=/opt/ttmlir-toolchain
99
ENV TTFORGE_TOOLCHAIN_DIR=/opt/ttforge-toolchain
10+
ENV TTMLIR_PYTHON_VERSION=python3.11
1011

1112
# Create a directory for the build and toolchain
1213
ARG BUILD_DIR=/home/build

.github/Dockerfile.ird

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,9 @@ RUN mkdir -p $TTFORGE_TOOLCHAIN_DIR && \
3131

3232
# Install GDB 14.2
3333
RUN apt install libmpfr-dev -y && \
34-
wget https://ftp.gnu.org/gnu/gdb/gdb-14.2.tar.gz && \
34+
# wget https://ftp.gnu.org/gnu/gdb/gdb-14.2.tar.gz && \
35+
# Mirror for ftp.gnu.org which is not reliable
36+
wget https://mirror.csclub.uwaterloo.ca/gnu/gdb/gdb-14.2.tar.gz && \
3537
tar -xvf gdb-14.2.tar.gz && \
3638
cd gdb-14.2 && \
3739
./configure && \

.github/workflows/codeql.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -58,7 +58,7 @@ jobs:
5858
- name: Set up Python
5959
uses: actions/setup-python@v5
6060
with:
61-
python-version: '3.10' # Specify the Python version you want to use
61+
python-version: '3.11' # Specify the Python version you want to use
6262
- name: Install dependencies
6363
shell: bash
6464
run: pip install flake8 flake8-sarif

.github/workflows/pre-commit.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,6 @@ jobs:
1212
- uses: actions/checkout@v4
1313
- uses: actions/setup-python@v5
1414
with:
15-
python-version: '3.10'
15+
python-version: '3.11'
1616
cache: 'pip'
1717
- uses: pre-commit/[email protected]

.github/workflows/update-test-durations.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ jobs:
2222
# ref: main
2323
- uses: actions/setup-python@v5
2424
with:
25-
python-version: '3.10'
25+
python-version: '3.11'
2626
- name: Process nightly test durations
2727
id: get-test-durations
2828
env:

.pre-commit-config.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ repos:
1313
rev: 22.3.0
1414
hooks:
1515
- id: black
16-
language_version: python3.10
16+
language_version: python3.11
1717
args: ["--line-length=120","-W 4"]
1818
- repo: https://github.com/pre-commit/mirrors-clang-format
1919
rev: v18.1.7

CMakeLists.txt

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -68,7 +68,7 @@ add_subdirectory(docs)
6868
set(CMAKE_INSTALL_RPATH "$ORIGIN")
6969
### Install _C.so directly into the install directory (not ${CMAKE_INSTALL_PREFIX}/lib/)
7070
### so that the python can find it. When we package this in the wheel, and it is installed
71-
### the ${CMAKE_INSTALL_PREFIX} directory will end up in `venv/lib/python3.10/site-packages/forge`.
71+
### the ${CMAKE_INSTALL_PREFIX} directory will end up in `venv/lib/python3.11/site-packages/forge`.
7272
install(TARGETS ttforge_csrc LIBRARY DESTINATION ${CMAKE_INSTALL_PREFIX})
7373

7474
### Generate stubs for ttforge

compile_flags.txt

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88
-Irbg
99
-I.
1010
-Ithird_party/pybind11/include
11-
-I/usr/include/python3.8
11+
-I/usr/include/python3.11
1212
-Igui_lib
1313
-Ithird_party/json
1414
-Iforge/csrc

docs/src/getting_started_build_from_source.md

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,7 @@ The prerequisites for building TT-Forge-FE from souce are:
3838
* Clang 17
3939
* Ninja
4040
* CMake (latest)
41-
* Python 3.10
41+
* Python 3.11
4242

4343
On Ubuntu 22.04 systems, you can install these dependencies using the following commands:
4444

@@ -84,16 +84,16 @@ sudo apt install ninja-build
8484
```
8585

8686
### Checking Python Version
87-
Make sure you have Python 3.10 installed:
87+
Make sure you have Python 3.11 installed:
8888

8989
```bash
9090
python3 --version
9191
```
9292

93-
If you do not have Python 3.10 installed:
93+
If you do not have Python 3.11 installed:
9494

9595
```bash
96-
sudo apt install python3.10
96+
sudo apt install python3.11
9797
```
9898

9999
### Installing CMake
@@ -280,4 +280,4 @@ This section goes over some useful environment variables for use with the [Build
280280
* `TTMLIR_VENV_DIR` - Specifies the virtual environment directory for TTMLIR. Defaults to `/opt/ttmlir-toolchain/venv` if not defined.
281281
* `TTFORGE_TOOLCHAIN_DIR` - Specifies the directory where tt-forge dependencies will be installed. Defaults to `/opt/ttforge-toolchain` if not defined.
282282
* `TTFORGE_VENV_DIR` - Specifies the virtual environment directory for tt-forge. Defaults to `/opt/ttforge-toolchain/venv` if not defined.
283-
* `TTFORGE_PYTHON_VERSION` - Specifies the Python version to use. Defaults to `python3.10` if not defined.
283+
* `TTFORGE_PYTHON_VERSION` - Specifies the Python version to use. Defaults to `python3.11` if not defined.

0 commit comments

Comments
 (0)