Conversation
Summary of ChangesHello @samudraneel05, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly expands Keras Hub's capabilities by integrating the Qwen2-VL multimodal model. It provides a complete framework for handling both visual and textual inputs, allowing for advanced vision-language tasks. The changes encompass the core model architecture, data preprocessing, and tools for converting existing models, making it easier for users to leverage this powerful new model. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Changelog
Activity
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces the Qwen2-VL multimodal model, including its backbone, causal LM task, preprocessor, image converter, and tokenizer. The implementation is comprehensive and well-structured, with thorough testing and a detailed checkpoint conversion script. The code largely adheres to the repository's style guide, particularly in its backend-agnostic implementation and modular design. I have two main suggestions for improvement: one is to populate the presets file to enable from_preset() functionality and testing, and the other is to refactor some duplicated code for scattering vision embeddings to improve maintainability. Overall, this is a high-quality contribution.
| # Scatter vision features into image placeholder positions. | ||
| if img_embeddings is not None: | ||
| image_mask = ops.equal( | ||
| token_ids, | ||
| ops.cast(self.backbone.image_token_id, token_ids.dtype), | ||
| ) | ||
| batch_size = ops.shape(x)[0] | ||
| seq_len = ops.shape(x)[1] | ||
| x_flat = ops.reshape(x, (-1, self.backbone.hidden_dim)) | ||
| mask_flat = ops.reshape(image_mask, (-1,)) | ||
| vision_indices = ops.where(mask_flat) | ||
| if isinstance(vision_indices, (list, tuple)): | ||
| vision_indices = vision_indices[0] | ||
| vision_indices = ops.reshape(vision_indices, (-1, 1)) | ||
| vision_indices = ops.cast(vision_indices, "int32") | ||
| x_flat = ops.scatter_update(x_flat, vision_indices, img_embeddings) | ||
| x = ops.reshape( | ||
| x_flat, (batch_size, seq_len, self.backbone.hidden_dim) | ||
| ) |
There was a problem hiding this comment.
The logic for scattering vision features into the text embeddings is duplicated between this method (call_with_cache) and Qwen2VLBackbone.call. To improve maintainability and adhere to the principles of modularity and reusability, consider refactoring this logic into a helper method within the Qwen2VLBackbone class. This helper could take the text embeddings, token IDs, and vision features as input and return the updated text embeddings. Both Qwen2VLBackbone.call and Qwen2VLCausalLM.call_with_cache could then call this shared method.
References
- The style guide emphasizes modularity and reusability. Refactoring duplicated code into a shared helper method aligns with these key principles. (link)
|
My bad, I got confused with the github handle names in the comment I mentioned about qwen2-VL. Sorry again for the confusion and inconvenience. |
|
i've reached out to the original issue assignee to see if we can do a best of both worlds model addition. i'll be closing this PR. The omni model PR is and has been ready for review for a while! |
2 participants
Description of the change
Added Qwen 2 VL, with notebooks documenting output matching and numerics verification on the 2B parameter version. Open for review and feedback!
Reference
Fixes #2323
Hugging Face Link: Link
Colab Notebook
Overall check with numerics here
Tokenizer comparison with hf here
Preprocessor comparison here
Checklist