Add qwen3vl support - #3463
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces support for the Qwen3-VL model, including its configuration, conversation template, and architecture, while refactoring the model registry to use explicit quantization mappings. The reviewer noted that the implementation is currently incomplete, as several critical methods in the Qwen3-VL model and text-processing components are stubs or raise NotImplementedError. Key feedback includes correcting multiple "CasualLM" typos, fixing the gorilla module name, updating incorrect model references in error messages, and addressing issues with commented-out code, type hints, and indentation.
| def forward( | ||
| self, | ||
| input_ids: Tensor = None, | ||
| attention_mask: Optional[Tensor] = None, | ||
| position_ids: Optional[Tensor] = None, | ||
| past_key_values: Optional["Cache"] = None, | ||
| inputs_embeds: Optional[Tensor] = None, | ||
| pixel_values: Optional[Tensor] = None, | ||
| pixel_values_videos: Optional[Tensor] = None, | ||
| image_grid_thw: Optional[Tensor] = None, | ||
| video_grid_thw: Optional[Tensor] = None, | ||
| cache_position: Optional[Tensor] = None, | ||
| **kwargs, | ||
| ): | ||
| r""" | ||
| image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`, *optional*): | ||
| The temporal, height and width of feature shape of each image in LLM. | ||
| video_grid_thw (`torch.LongTensor` of shape `(num_videos, 3)`, *optional*): | ||
| The temporal, height and width of feature shape of each video in LLM. | ||
| """ | ||
| raise NotImplementedError |
There was a problem hiding this comment.
The forward method of Qwen3VLModel raises NotImplementedError and contains a large block of commented-out PyTorch code. This is a critical part of the model implementation that is missing. Please implement the forward pass for the Qwen3VLModel in TVM/Relax.
| def forward(self, x, position_ids): | ||
| # TODO: translate from pytorch to tvm | ||
| raise NotImplementedError |
There was a problem hiding this comment.
The forward method in LlamaRotaryEmbedding raises NotImplementedError. Multiple other methods in this file (Qwen3VLTextRotaryEmbedding.forward, Qwen3VLTextAttention.forward, Qwen3VLTextDecoderLayer.forward, Qwen3VLTextModel.forward) are also unimplemented skeletons. This makes the text-processing part of the model non-functional.
This is a critical issue. Please implement the logic for these methods.
| def prefill(self, input_embed: Tensor, paged_kv_cache: PagedKVCache): | ||
| b, s, d = input_embed.shape | ||
| return op.zeros((b, s, self.config.text_config.vocab_size), dtype="float32"), paged_kv_cache | ||
|
|
||
| def decode(self, input_embed: Tensor, paged_kv_cache: PagedKVCache): | ||
| b, s, d = input_embed.shape | ||
| return op.zeros((b, s, self.config.text_config.vocab_size), dtype="float32"), paged_kv_cache |
There was a problem hiding this comment.
The prefill and decode methods in Qwen3VLForConditionalGeneration are stubs that return tensors of zeros. This makes the model non-functional as it will not produce any meaningful output. Please implement the correct logic for these methods.
| "mistral": Model( | ||
| name="mistral", | ||
| model=mistral_model.MistralForCausalLM, | ||
| model=mistral_model.MistralForCasualLM, |
There was a problem hiding this comment.
There's a typo in the class name. It should be MistralForCausalLM, not MistralForCasualLM. This "Casual" typo appears for several other models in this file:
mixtral(line 209)rwkv5(line 480)orion(line 494)llava(line 507)rwkv6(line 522)eagle(line 548)
Please correct all occurrences.
| model=mistral_model.MistralForCasualLM, | |
| model=mistral_model.MistralForCausalLM, |
| ) | ||
| else: | ||
| raise ValueError( | ||
| "Invalid DeepSeek model quantization config: unrecognized quantization config: " |
There was a problem hiding this comment.
The error message refers to "DeepSeek model". It should be "Qwen3-VL model" to avoid confusion.
| "Invalid DeepSeek model quantization config: unrecognized quantization config: " | |
| "Invalid Qwen3-VL model quantization config: unrecognized quantization config: " |
|
|
||
| return patch_pos_embeds | ||
|
|
||
| def forward(self, hidden_states: Tensor, grid_thw: Tensor, **kwargs) -> Tensor: |
There was a problem hiding this comment.
The return type hint is -> Tensor, but the function can return a list of tensors. Please update the type hint to -> "Union[Tensor, List[Tensor]]" and import Union and List from typing if not already present. Using a string for the type hint can avoid import issues.
| def forward(self, hidden_states: Tensor, grid_thw: Tensor, **kwargs) -> Tensor: | |
| def forward(self, hidden_states: Tensor, grid_thw: Tensor, **kwargs) -> "Union[Tensor, List[Tensor]]": |
| accepts_loss_kwargs = False | ||
|
|
||
| def __init__(self, config): | ||
| super().__init__(config) No newline at end of file |
There was a problem hiding this comment.
The file ends without a newline character. Please add one for consistency.
| @@ -0,0 +1,66 @@ | |||
|
|
|||
There was a problem hiding this comment.
This file starts with a blank line, which is a minor style issue. Please remove it.
| k_embed = (k * cos) + (rotate_half(k) * sin) | ||
| q_embed = q_embed.to(orig_q_dtype) | ||
| k_embed = k_embed.to(orig_k_dtype) | ||
| return q_embed, k_embed No newline at end of file |
There was a problem hiding this comment.
The file ends without a newline character. Please add one for consistency.
| hidden_states, | ||
| (-1, self.in_channels, self.temporal_patch_size, self.patch_size, self.patch_size) | ||
| ) | ||
|
|
There was a problem hiding this comment.
This line has an extra level of indentation.
| hidden_states = self.proj(hidden_states) |
1 participant
No description provided.