Commit 7c19bb0
committed
Thread image_grid_thw through Qwen3-VL, add smart-resize + multi-image
Per-image grid dimensions are now a real model input rather than being
inferred from sqrt(num_patches). This fixes correctness for non-square
images and lets one forward pass handle multiple images of different
sizes in the same prompt (concatenated patch sequence + per-image
grid_thw).
Featurizer changes:
- smart_resize mirrors qwen-vl-utils: preserves aspect ratio, rounds to
a multiple of patch_size * merge_size, clamps total pixels between
min_pixels and max_pixels
- :quality preset (:low / :medium / :high) over explicit pixel caps
- accepts a list of images of different sizes; concatenates patches,
returns image_grid_thw of shape {num_images, 3}
- patches are emitted in windowed order so the merger reshape stays
shape-agnostic (4 consecutive patches = one 2x2 merge block)
Vision encoder:
- image_grid_thw declared as a model input
- per-patch (row, col, grid_h, grid_w, image_id) derived from grid_thw
via Nx ops; drives bilinear pos-embed interpolation and 2D rotary
- block-diagonal attention mask (image_id == image_id) so patches from
one image cannot attend to patches from another
- patch merger and deepstack merger reshapes no longer assume a square
single-image grid
Multimodal:
- image_grid_thw plumbed as an optional model input and forwarded into
the vision sub-model
Validation:
- 13 new tests covering smart_resize aspect ratio, quality presets,
multi-image concat, windowed-layout invariant, single+multi-image
end-to-end with the tiny model
- Full fast suite: 285 passed, 0 regressions
- Real Qwen3-VL-2B-Instruct on a 640x480 COCO image: 8/10 top-token
agreement vs HuggingFace transformers reference; top-3 identical and
in same order
Refs #442.1 parent 8548ee3 commit 7c19bb0
6 files changed
Lines changed: 797 additions & 441 deletions
File tree
- lib/bumblebee
- multimodal
- vision
- notebooks
- test/bumblebee
- multimodal
- vision
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
33 | 33 | | |
34 | 34 | | |
35 | 35 | | |
36 | | - | |
37 | | - | |
38 | | - | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
39 | 45 | | |
40 | 46 | | |
41 | 47 | | |
| |||
92 | 98 | | |
93 | 99 | | |
94 | 100 | | |
| 101 | + | |
95 | 102 | | |
96 | 103 | | |
97 | 104 | | |
| |||
114 | 121 | | |
115 | 122 | | |
116 | 123 | | |
117 | | - | |
| 124 | + | |
| 125 | + | |
118 | 126 | | |
119 | 127 | | |
120 | 128 | | |
| |||
194 | 202 | | |
195 | 203 | | |
196 | 204 | | |
| 205 | + | |
197 | 206 | | |
198 | 207 | | |
199 | 208 | | |
| |||
0 commit comments