Skip to content

[Testing] Support Qwen3-VL + LoRA for VLM + PA accuracy fix (M-RoPE, Qwen VL models)#3409

Closed
yatarkan wants to merge 65 commits intoopenvinotoolkit:masterfrom
yatarkan:yt/testing/qwen3-vl
Closed

[Testing] Support Qwen3-VL + LoRA for VLM + PA accuracy fix (M-RoPE, Qwen VL models)#3409
yatarkan wants to merge 65 commits intoopenvinotoolkit:masterfrom
yatarkan:yt/testing/qwen3-vl

Conversation

@yatarkan
Copy link
Contributor

@yatarkan yatarkan commented Feb 27, 2026

Description

This PR enables Qwen3-VL model in GenAI VLM pipeline for end-to-end testing by aggregating related changes from different PRs.
The PR is based on #3253 and also includes changes from #3402 and #3400 (which also requires OpenVINO PR).

For model exporting latest Optimum Intel from master and transformers>=4.57.0 are required.

CVS-175825
CVS-180081
CVS-180080

Checklist:

  • This PR follows GenAI Contributing guidelines.
  • Tests have been updated or added to cover the new code.
  • This PR fully addresses the ticket.
  • I have made corresponding changes to the documentation.

yatarkan and others added 30 commits January 27, 2026 11:44
@yatarkan
Add qwen3_vl to vlm model type enum
07f5441
@yatarkan
Expose merge_text_and_video_image_embeddings qwen2vl utils function
2732e8d
@yatarkan
Enable nested json param reading for vector type
a5b870b
@yatarkan
Add qwen3_vl specific vlm config
7e654b3
@yatarkan
Add qwen3_vl params to processor config
75be9d2
@yatarkan
Add classes for qwen3_vl
7a4f8b9
@yatarkan
Add qwen3_vl classes to inputs embedder and vision encoder
f44f6c2
@yatarkan
Enable extra inputs in lm_encoding
49e31ac
@yatarkan
Propagate lm extra inputs in stateful vlm pipeline
f023416
@yatarkan
Merge branch 'master' into yt/qwen3-vl
b94d9fc
@yatarkan
Fix signature after merge
2dc6698
@yatarkan
Add qwen3vl extra inputs to sequence group
a7b8818
@yatarkan
Use qwen3vl extra inputs in CB model runner
e974a35
@yatarkan
Propogate lm extra inputs to CB pipelines generate and add_request me…
d116d1d 
…thods
@yatarkan
Update generate and add_request signatures of CB inherited classes
7bbe27c
@yatarkan
Add tiny-random-qwen3-vl model to python tests
8a73df1
@yatarkan
Add min tf version check for vlm pipeline tests
52bee9e
@yatarkan
Add steps for running vlm tests with qwen3-vl in CI
eef0d57
@yatarkan
Check sequence group type in forward for deepstack inputs aggregated …
01bb64b 
…values
@yatarkan
Merge branch 'master' into yt/qwen3-vl
528d4ef
@likholat
lora for vlm
3c7aefd
@likholat
hf vlm with lora
afe925c
@yatarkan
Add optimum intel installation from master
03dd46f
@yatarkan
Update qwen3_vl default resolution in python tests, add skip for fail…
6bba7f0 
…ing 350x350 image
@yatarkan
Disable cleanup tokenization spaces for genai vs optimum python test
196a135
@likholat
python sample
5d1a42a
@likholat
Merge remote-tracking branch 'origin/master' into lora_for_vlm
de0fe24
@likholat
enable Lora for VLM CB wwb case
5c9abc0
@CuriousPanCake
wip
1705946
@yatarkan
Fix deepstack vision inputs for chunked prefill case in CB
a7b1597
yatarkan added 26 commits March 5, 2026 20:33
@yatarkan
Add Qwen3-VL to supported models in docs
7345d1a
@yatarkan
Add qwen3-vl to llm_bench visual_text_gen use case
07b37c8
@yatarkan
Fix trailing comma
ff91f93
@yatarkan
Fix typo in var name
611033c
@yatarkan
Fix copilot review comments
95d5e53
@yatarkan
Pass video metadata struct to calculate_timestamps function, fix revi…
d12544b 
…ew comments
@yatarkan
Safe get prompt_ids optional tensor in CB
fb28c75
@yatarkan
Add lm extra inputs to add_request with video inputs
2c1de78
@yatarkan
Fix copyright year
3cc9ac0
@yatarkan
Move deep copy tensors map to utility function
88d36a5
@yatarkan
Make variables const
3f1f74b
@yatarkan
Remove ununsed var
e0e1253
@yatarkan
Move deepstack data aggregation and filling tensor to struct with fun…
1393641 
…ctions
@yatarkan
Add default initializers for DeepstackGroupData, remove unused var
2b08b71
@yatarkan
Change return type to const ref
68a4973
@yatarkan
Add qwen3-vl image and video tags to docs
a1202f9
@yatarkan
Fix name
ee2ed7b
@yatarkan
Pin optimum-intel commit
e389293
@yatarkan
Fix docstring
acb70db
@yatarkan
Fix review comment
831244c
@yatarkan
Move qwen3vl utils to .cpp anonymous namespace, change fps type to fl…
fe6ade2 
…oat, fix review comments
@yatarkan
Fix default value
8a9d7b6
@yatarkan
Add comment for video processor config fps
0bfacd4
@yatarkan
Add qwen3-vl tests skip for resolutions corner cases
3152801
@yatarkan
Merge branch 'master' into yt/qwen3-vl
05933bf
@yatarkan
Merge branch 'yt/qwen3-vl' into yt/testing/qwen3-vl
045b12b
@github-actions github-actions bot added category: llm_bench Label for tool/llm_bench folder category: GH Pages Docs Github Pages documentation labels Mar 11, 2026
@yatarkan
Copy link
Contributor Author

yatarkan commented Mar 16, 2026

Closing as all aggregated PRs are merged to master

@yatarkan yatarkan closed this Mar 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

@sbalandi sbalandi Awaiting requested review from sbalandi sbalandi will be requested when the pull request is marked ready for review sbalandi is a code owner

@sgonorov sgonorov Awaiting requested review from sgonorov sgonorov will be requested when the pull request is marked ready for review sgonorov is a code owner

@likholat likholat Awaiting requested review from likholat likholat will be requested when the pull request is marked ready for review likholat is a code owner

@popovaan popovaan Awaiting requested review from popovaan popovaan will be requested when the pull request is marked ready for review popovaan is a code owner

@akashchi akashchi Awaiting requested review from akashchi akashchi will be requested when the pull request is marked ready for review akashchi is a code owner

@Wovchena Wovchena Awaiting requested review from Wovchena Wovchena will be requested when the pull request is marked ready for review Wovchena is a code owner

@as-suvorov as-suvorov Awaiting requested review from as-suvorov as-suvorov will be requested when the pull request is marked ready for review as-suvorov is a code owner

Assignees

No one assigned

Labels

category: cmake / build Cmake scripts category: continuous batching Continuous batching category: GGUF GGUF file reader category: GH Pages Docs Github Pages documentation category: GHA CI based on Github actions category: llm_bench Label for tool/llm_bench folder category: LLM LLM pipeline (stateful, static) category: LoRA Low rank adapters category: prompt lookup Prompt look-up decoding category: speculative decoding Speculative decoding category: visual language Visual language pipeline category: VLM samples GenAI VLM samples category: WWB PR changes WWB no-match-files

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@yatarkan @likholat @CuriousPanCake