[NPUW] Replacing longrope pattern with precalculated values#33011
Merged
dmatveev merged 7 commits intoopenvinotoolkit:masterfrom Mar 4, 2026
Merged
[NPUW] Replacing longrope pattern with precalculated values#33011dmatveev merged 7 commits intoopenvinotoolkit:masterfrom
dmatveev merged 7 commits intoopenvinotoolkit:masterfrom
Conversation
github-actions
bot
added
category: NPU
| auto* pos_ids_data = position_ids->data<int64_t>(); | ||
| // assuming position_ids are constantly non-deacreasing. | ||
| // this potentially could be not true. Alternative is to find max value in position_ids | ||
| auto max_pos_id = pos_ids_data[position_ids->get_size() - 1]; |
Contributor
Author
There was a problem hiding this comment.
Check if attention mask is right padded
| const std::shared_ptr<ov::Model>& model) { | ||
| auto rpe = std::make_shared<RopePatternLLama2>(); | ||
| ov::npuw::patterns::pre_compute::LongRopePatternPhi::LongRopePatternPhi() : matcher("sin-cos-matcher") { | ||
| auto MakeConstant = []() { |
Contributor
Author
There was a problem hiding this comment.
Run pass conditionally
AsyaPronina
approved these changes
Mar 4, 2026
| // Step 2: Apply Gather for cos and sin | ||
| auto gather_cos = std::make_shared<ov::op::v8::Gather>(cache[0], rpe->matched_position_ids, axis); | ||
| auto gather_sin = std::make_shared<ov::op::v8::Gather>(cache[1], rpe->matched_position_ids, axis); | ||
| LOG_VERB("Created gather op facilitate LUT search: " << gather_cos->get_name() << ", " << gather_cos->get_shape()); |
Contributor
There was a problem hiding this comment.
Do we need verbose for the sin also?
| LOG_VERB("Rope sin detected at: " << rpe->matched_sin->get_name() << ", replacing by cache node: " | ||
| << gather_sin->get_name() << ", " << gather_sin->get_shape()); | ||
|
|
||
| // replacing sin with gather op |
| ov::replace_node(rpe->matched_cos, squeeze_cos); | ||
| ov::replace_node(rpe->matched_sin, squeeze_sin); | ||
|
|
||
| // disconnecting gather from rest or subgraph started from concat_1 |
Contributor
There was a problem hiding this comment.
Minor comment: typo in of
| return {Cos, Sin}; | ||
| } | ||
|
|
||
| void replaceSinCosByCache(int max_prompt_len, const ov::OutputVector& cache, const pre_compute::RopePatternDesc* rpe) { |
Contributor
There was a problem hiding this comment.
Might be to align with makeCosSinCache we need to swap Sin and Cos in name by places?
mlukasze
pushed a commit
to mlukasze/openvino
that referenced
this pull request
Mar 5, 2026
…toolkit#33011) Depends on huggingface/optimum-intel#1504 Searching pattern would be updated to the actual pattern which would be generated after huggingface/optimum-intel#1504 get merged.
Nishant-ZFYII
pushed a commit
to Nishant-ZFYII/openvino
that referenced
this pull request
Mar 5, 2026
…toolkit#33011) Depends on huggingface/optimum-intel#1504 Searching pattern would be updated to the actual pattern which would be generated after huggingface/optimum-intel#1504 get merged.
atamas19
pushed a commit
to atamas19/openvino
that referenced
this pull request
Mar 6, 2026
…toolkit#33011) Depends on huggingface/optimum-intel#1504 Searching pattern would be updated to the actual pattern which would be generated after huggingface/optimum-intel#1504 get merged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Depends on huggingface/optimum-intel#1504
Searching pattern would be updated to the actual pattern which would be generated after huggingface/optimum-intel#1504 get merged.