[Feature] Add Qwen3.5 vision model support by gnguralnick · Pull Request #3474 · mlc-ai/mlc-llm

gnguralnick · 2026-04-02T17:41:05Z

Summary

Add Qwen3.5 vision model support with custom ViT encoder (2D RoPE, spatial patch merging)
Vision-aware conversation protocol handling for Qwen3.5V image tokens
Image embedding size computation for Qwen3.5V

Stacked on #3473 and #3429 .

Test plan

Unit tests for Qwen3.5V model config, vision encoder, and weight loading (test_qwen35v.py)
End-to-end: compile and chat with a Qwen3.5V model with image input

gemini-code-assist

Code Review

This pull request introduces support for Gemma 3 Vision and Qwen 3.5 Vision models. It includes the implementation of the SigLIP vision encoder, model architectures, weight loaders, and conversation templates for these VLMs. Additionally, it updates image processing utilities to handle model-specific normalization and tokenization requirements. Feedback focuses on correcting API usage in the Qwen 3.5 vision encoder, fixing a missing import in the attention operator, and improving the dynamic calculation of image embedding sizes in the serving layer.

gemini-code-assist · 2026-04-02T17:46:27Z

+        cos = nn.wrap_nested(cos, "rope_cos")
+        sin = nn.wrap_nested(sin, "rope_sin")


The nn module (imported as tvm.relax.frontend.nn) does not have a wrap_nested attribute. This function is located within the nn.op module. Since you have already imported op from nn, you should use op.wrap_nested instead.

Suggested change

cos = nn.wrap_nested(cos, "rope_cos")

sin = nn.wrap_nested(sin, "rope_sin")

cos = op.wrap_nested(cos, "rope_cos")

sin = op.wrap_nested(sin, "rope_sin")

gemini-code-assist · 2026-04-02T17:46:27Z

+
+    # rotate_half: [-x2, x1] where x1, x2 are first/second half of last dim
+    x1, x2 = op.split(x, 2, axis=-1)
+    rotated = op.concat([op.negative(x2), x1], dim=-1)


The op.concat function in the nn.op module uses the keyword argument axis, not dim. Using dim will result in a TypeError at runtime.

Suggested change

rotated = op.concat([op.negative(x2), x1], dim=-1)

rotated = op.concat([op.negative(x2), x1], axis=-1)

gemini-code-assist · 2026-04-02T17:46:28Z

            v = v.repeat(h_q // h_kv, axis=2)

-        target = tvm.target.Target("cuda")
+        target = _extern.get_store().target or tvm.target.Target("cuda")


The variable _extern is used here but it is not imported in the file's header. This will cause a NameError when the _fallback path is executed. Please ensure _extern is correctly imported from the local module.

Suggested change

target = _extern.get_store().target or tvm.target.Target("cuda")

target = nn.op.extern.get_store().target or tvm.target.Target("cuda")

gemini-code-assist · 2026-04-02T17:46:28Z

+        if model_type == "gemma3_v":
+            # fixed to 256 per image
+            return 256
+
+        if model_type == "qwen3_5_vision":
+            # (image_size / patch_size / spatial_merge_size)^2 = (448/16/2)^2 = 196
+            return 196


The image embedding sizes for gemma3_v and qwen3_5_vision are currently hardcoded. This prevents the engine from correctly handling images if the model is compiled with a different resolution or patch configuration. It is better to compute these values dynamically using the provided config.

Suggested change

if model_type == "gemma3_v":

# fixed to 256 per image

return 256

if model_type == "qwen3_5_vision":

# (image_size / patch_size / spatial_merge_size)^2 = (448/16/2)^2 = 196

return 196

if model_type == "gemma3_v":

# (image_size // patch_size // 4)^2 = (896 // 14 // 4)^2 = 256

vision_cfg = config["model_config"]["vision_config"]

image_size = vision_cfg.get("image_size", 896)

patch_size = vision_cfg.get("patch_size", 14)

return (image_size // patch_size // 4) ** 2

if model_type == "qwen3_5_vision":

# (image_size / patch_size / spatial_merge_size)^2 = (448/16/2)^2 = 196

vision_cfg = config["model_config"]["vision_config"]

image_size = config["model_config"].get("image_size", 448)

patch_size = vision_cfg.get("patch_size", 16)

merge_size = vision_cfg.get("spatial_merge_size", 2)

return (image_size // patch_size // merge_size) ** 2

Add dedicated conversation templates for Qwen3.5 models instead of using the generic ChatML format. Includes two variants: - qwen3_5: thinking enabled (assistant opens a <think> block) - qwen3_5_nothink: thinking disabled (empty <think> block prefix) Sculptor: gabeguralnick

… config

…ix type hint - Use self.config.vision_config.image_size instead of hardcoded 896 in gemma3v image_preprocess - Refactor normalize/normalize_siglip into shared _normalize_impl to reduce duplication in ImageProcessor - Fix SigLIPEncoder.forward return type hint (Tensor -> Tuple[Tensor, ...])

…igLIP - Move trailing \n into else branch so gemma3_v doesn't get extra newline after EOI - Remove unused image_token_index from Gemma3VConfig - Fix misleading comment on mm_soft_emb_norm (Gemma +1 is fused during weight loading) - Remove redundant gemma3_vision_instruction template (identical to gemma3_instruction) - Simplify SigLIP encoder: remove tuple wrapping, state accumulation, unused logger

Adapts to the quantization refactoring on main that replaced per-model quantization files with make_quantization_functions.

- test_gemma3v.py: model registration, TVM IR export with VLM composition (vision_tower + language_model + projector), config validation - test_siglip_vision.py: SigLIP vision encoder export with expected parameter components - test_attention_fallback.py: correctness test for naive matmul+softmax fallback when head_dim % 16 != 0 (e.g. SigLIP head_dim=72), compared against numpy reference

- Update gemma3v_model.py to use tirx namespace (matching upstream migration) - Remove head_dim%16 attention fallback from op/attention.py (TVM bug is fixed) - Keep Optional type hints and target detection fix in attention.py - Remove test_attention_fallback.py (no longer needed)

Use the standard loader for gate_up concatenation and 1:1 param fallback, matching the pattern established in gemma3_loader.py. Only vision-specific overrides (mm_input_projection transpose, norm +1) remain manual.

TVM FFI objects don't support setting arbitrary Python attributes. Expose embed_size via a new ImageDataGetEmbedSize FFI function instead of storing it as a Python attribute.

Add vision encoder, VLM wrapper, and weight loader for Qwen3.5 VLM (model type: qwen3_5_vision). The vision encoder uses a custom ViT with 2D RoPE and 2x2 spatial patch merging, producing 196 image tokens at 448x448 resolution. Also adds qwen3_5 and qwen3_5_nothink conversation templates with correct Qwen3.5 stop token IDs (248046/248044) and thinking support. Tested with Qwen3.5-4B (q4f16_1, Metal) for both text and vision.

Raise explicit error when vision_config is None instead of silently leaving it unset (which causes AttributeError later in grid_h/grid_w). Remove spurious blank line in conversation_template imports.

Tests model registration, TVM IR export, config validation, and standalone vision encoder creation following existing patterns (test_gemma3v.py, test_siglip_vision.py).

gnguralnick mentioned this pull request Apr 2, 2026

[Feature] Add Qwen3.5 vision support + conversation templates #3471

Closed

6 tasks

gemini-code-assist Bot reviewed Apr 2, 2026

View reviewed changes

gnguralnick changed the title ~~[Feature] Add Qwen3.5 vision and Gemma3 vision model support~~ [Feature] Add Qwen3.5 vision-language model support Apr 2, 2026

gnguralnick changed the title ~~[Feature] Add Qwen3.5 vision-language model support~~ [Feature] Add Qwen3.5 vision model support Apr 2, 2026

Gabriel Guralnick added 19 commits April 2, 2026 10:50

implementation of gemma 3 with vision

0a3289d

make attention bug workaround more general

c5778e6

fix image dtype for webgpu

4a410f9

allow quantization of vision encoder

9918f53

fix lint: isort, black, mypy, and pylint issues

4b3e6be

address code review: use op.clip, op.nn.avg_pool2d, read BOI/EOI from…

207526e

… config

fix op.clip and op.nn.avg_pool2d: use relax.op with wrap_nested

05ae01a

fix typing issues

1488bfe

remove gemma3v_quantization.py: use centralized factory instead

dd5a1ef

Adapts to the quantization refactoring on main that replaced per-model quantization files with make_quantization_functions.

refactor gemma3v_loader to use make_standard_hf_loader

b505426

Use the standard loader for gate_up concatenation and 1:1 param fallback, matching the pattern established in gemma3_loader.py. Only vision-specific overrides (mm_input_projection transpose, norm +1) remain manual.

fix ImageData embed_size attribute error on TVM FFI objects

513aff9

TVM FFI objects don't support setting arbitrary Python attributes. Expose embed_size via a new ImageDataGetEmbedSize FFI function instead of storing it as a Python attribute.

[Fix] Qwen3.5V: require vision_config and fix import formatting

905b21c

Raise explicit error when vision_config is None instead of silently leaving it unset (which causes AttributeError later in grid_h/grid_w). Remove spurious blank line in conversation_template imports.

[Test] Add unit tests for Qwen3.5 vision-language model

4f97a4e

Tests model registration, TVM IR export, config validation, and standalone vision encoder creation following existing patterns (test_gemma3v.py, test_siglip_vision.py).

gnguralnick force-pushed the qwen35-vision-pr branch from 01faf04 to 4f97a4e Compare April 2, 2026 17:50

gnguralnick marked this pull request as ready for review April 2, 2026 17:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Feature] Add Qwen3.5 vision model support - #3474

[Feature] Add Qwen3.5 vision model support#3474
gnguralnick wants to merge 19 commits into
mlc-ai:mainfrom
gnguralnick:qwen35-vision-pr

gnguralnick commented Apr 2, 2026 •

edited

Loading

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

gemini-code-assist Bot Apr 2, 2026

Uh oh!

gemini-code-assist Bot Apr 2, 2026

Uh oh!

gemini-code-assist Bot Apr 2, 2026

Uh oh!

gemini-code-assist Bot Apr 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

		cos = nn.wrap_nested(cos, "rope_cos")
		sin = nn.wrap_nested(sin, "rope_sin")

	rotated = op.concat([op.negative(x2), x1], dim=-1)
	rotated = op.concat([op.negative(x2), x1], axis=-1)

	target = _extern.get_store().target or tvm.target.Target("cuda")
	target = nn.op.extern.get_store().target or tvm.target.Target("cuda")

-        if model_type == "gemma3_v":
-            # fixed to 256 per image
-            return 256
-        if model_type == "qwen3_5_vision":
-            # (image_size / patch_size / spatial_merge_size)^2 = (448/16/2)^2 = 196
-            return 196
+        if model_type == "gemma3_v":
+            # (image_size // patch_size // 4)^2 = (896 // 14 // 4)^2 = 256
+            vision_cfg = config["model_config"]["vision_config"]
+            image_size = vision_cfg.get("image_size", 896)
+            patch_size = vision_cfg.get("patch_size", 14)
+            return (image_size // patch_size // 4) ** 2
+        if model_type == "qwen3_5_vision":
+            # (image_size / patch_size / spatial_merge_size)^2 = (448/16/2)^2 = 196
+            vision_cfg = config["model_config"]["vision_config"]
+            image_size = config["model_config"].get("image_size", 448)
+            patch_size = vision_cfg.get("patch_size", 16)
+            merge_size = vision_cfg.get("spatial_merge_size", 2)
+            return (image_size // patch_size // merge_size) ** 2

Uh oh!

Conversation

gnguralnick commented Apr 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

gnguralnick commented Apr 2, 2026 •

edited

Loading