Skip to content

[GPU][TRANSFORMATIONS] Support RMS Normalization Fusion without Learnable Affine Parameters#33861

Merged
e-ddykim merged 13 commits intoopenvinotoolkit:masterfrom
andrew-k-park:ltx_video_rms
Feb 11, 2026
Merged

[GPU][TRANSFORMATIONS] Support RMS Normalization Fusion without Learnable Affine Parameters#33861
e-ddykim merged 13 commits intoopenvinotoolkit:masterfrom
andrew-k-park:ltx_video_rms

Conversation

@andrew-k-park
Copy link
Contributor

@andrew-k-park andrew-k-park commented Jan 29, 2026

Details:

  • This PR enhances RMS normalization fusion to support pattern without learnable affine parameter (gamma), enabling optimization of transformer architecture like LTX-Video
  • The existing RMS fusion pass only supported pattern with constant gamma parameter. However, some transformer model (e.g., LTX-Video's attention layers) use RMS normalization followed by dynamic scaling operation where the scale factor is non-constant. These pattern was previously unfused, missing optimization opportunity
  • When elementwise_affine=False (equivalent to Pytorch RMS's attribute), RMS normalization does not include learnable gamma parameters. The gamma is implicitly fixed to ones, reducing the decomposed graph pattern
    from:
    x → Power(2) → ReduceMean → Add(eps) → Sqrt → Divide(1/√) → Multiply(x, 1/√) → Multiply(gamma)
    to:
    x → Power(2) → ReduceMean → Add(eps) → Sqrt → Divide(1/√) → Multiply(x, 1/√) [NO gamma multiplication]
image-2026-01-26-22-53-05-973

Tickets:

@andrew-k-park andrew-k-park requested review from a team as code owners January 29, 2026 04:20
@github-actions github-actions bot added category: GPU OpenVINO GPU plugin category: transformations OpenVINO Runtime library - Transformations labels Jan 29, 2026
@andrew-k-park andrew-k-park requested a review from a team as a code owner January 29, 2026 12:46
@github-actions github-actions bot added the category: TEMPLATE OpenVINO Template plugin label Jan 29, 2026
@andrew-k-park andrew-k-park added this to the 2026.1 milestone Jan 29, 2026
e-ddykim
e-ddykim reviewed Jan 29, 2026
View reviewed changes
e-ddykim
e-ddykim approved these changes Jan 30, 2026
View reviewed changes
Copy link
Contributor

@e-ddykim e-ddykim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me for the GPU part

@andrew-k-park
Copy link
Contributor Author

andrew-k-park commented Jan 30, 2026

@CuriousPanCake Could you review this PR?

CuriousPanCake
CuriousPanCake requested changes Jan 30, 2026
View reviewed changes
Copy link
Contributor

@CuriousPanCake CuriousPanCake left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please, review the comments for the Transformations part

@andrew-k-park andrew-k-park force-pushed the ltx_video_rms branch 4 times, most recently from c499127 to e5ebd49 Compare February 2, 2026 11:28
@andrew-k-park
Copy link
Contributor Author

andrew-k-park commented Feb 4, 2026

@dmatveev, this PR is breaking some NPUW func test. Please have a look.
In my opinion, this NPU test appears to be related to the CPU plugin’s transformation pipeline. In the CPU plugin, I have made a change so that the relevant pattern is no longer matched in RMSFusion.

@andrew-k-park andrew-k-park force-pushed the ltx_video_rms branch 2 times, most recently from c9733df to c284fb2 Compare February 5, 2026 00:29
@andrew-k-park
Copy link
Contributor Author

andrew-k-park commented Feb 5, 2026

@CuriousPanCake Could you please review the updated code that apply your comments again?

CuriousPanCake
CuriousPanCake approved these changes Feb 9, 2026
View reviewed changes
@CuriousPanCake
Copy link
Contributor

CuriousPanCake commented Feb 9, 2026

LGTM from the transformations side

@e-ddykim e-ddykim enabled auto-merge February 10, 2026 00:29
@andrew-k-park andrew-k-park changed the title [GPU] Support RMS Normalization Fusion without Learnable Affine Parameters [GPU][TRANSFORMATIONS] Support RMS Normalization Fusion without Learnable Affine Parameters Feb 10, 2026
@andrew-k-park andrew-k-park force-pushed the ltx_video_rms branch 3 times, most recently from 18199cc to 67c9a7e Compare February 10, 2026 23:43
@e-ddykim e-ddykim added this pull request to the merge queue Feb 11, 2026
Merged via the queue into openvinotoolkit:master with commit 6b80671 Feb 11, 2026
216 checks passed
@andrew-k-park andrew-k-park deleted the ltx_video_rms branch February 11, 2026 07:43
Naseer-010 pushed a commit to Naseer-010/openvino that referenced this pull request Feb 18, 2026
@andrew-k-park @Naseer-010
[GPU][TRANSFORMATIONS] Support RMS Normalization Fusion without Learn…
c48df79 
…able Affine Parameters (openvinotoolkit#33861)

### Details:
- This PR enhances RMS normalization fusion to support pattern without
learnable affine parameter (gamma), enabling optimization of transformer
architecture like LTX-Video
- The existing RMS fusion pass only supported pattern with constant
gamma parameter. However, some transformer model (e.g., LTX-Video's
attention layers) use RMS normalization followed by dynamic scaling
operation where the scale factor is non-constant. These pattern was
previously unfused, missing optimization opportunity
- When `elementwise_affine=False` (equivalent to [Pytorch RMS's
attribute](https://docs.pytorch.org/docs/stable/generated/torch.nn.modules.normalization.RMSNorm.html)),
RMS normalization does not include learnable gamma parameters. The gamma
is implicitly fixed to ones, reducing the decomposed graph pattern
from:
`x → Power(2) → ReduceMean → Add(eps) → Sqrt → Divide(1/√) → Multiply(x,
1/√) → Multiply(gamma) `
to:
`x → Power(2) → ReduceMean → Add(eps) → Sqrt → Divide(1/√) → Multiply(x,
1/√) [NO gamma multiplication] `
<img width="648" height="924" alt="image-2026-01-26-22-53-05-973"
src="https://github.com/user-attachments/assets/02b4580f-bbce-43ea-affd-438f0a5f4ea7"
/>

### Tickets:
 - [CVS-179953](https://jira.devtools.intel.com/browse/CVS-179953)

---------

Signed-off-by: Andrew Park <andrew.park@intel.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@e-ddykim e-ddykim e-ddykim approved these changes

@CuriousPanCake CuriousPanCake CuriousPanCake approved these changes

Assignees

No one assigned

Labels

category: GPU OpenVINO GPU plugin category: TEMPLATE OpenVINO Template plugin category: transformations OpenVINO Runtime library - Transformations

Projects

None yet

Milestone

2026.1

Development

Successfully merging this pull request may close these issues.

4 participants

@andrew-k-park @pereanub @CuriousPanCake @e-ddykim