Commit 7108c2c
authored
feat: add LLaVA-OneVision2 chat model wrapper (#1337)
Register llava_onevision2_chat (key: llava_onevision2) targeting the
released checkpoint lmms-lab-encoder/LLaVA-OneVision2-8B-Instruct.
The wrapper loads via AutoModelForImageTextToText with trust_remote_code
so the bundled processing pipeline (patch_positions, RoPE block layout,
frame sampling + smart_resize, per-frame timestamp expansion) is used
exactly as during training.
- New: lmms_eval/models/chat/llava_onevision2.py
- Register in lmms_eval/models/__init__.py
- Example launch script: examples/models/llava_onevision2.sh
- Documented under docs/advanced/throughput_metrics.md as a backend
that logs throughput via log_metrics().
Co-authored-by: yiyexy <yiyexy@users.noreply.github.com>1 parent 9c7a55a commit 7108c2c
4 files changed
Lines changed: 438 additions & 1 deletion
File tree
- docs/advanced
- examples/models
- lmms_eval/models
- chat
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
57 | 57 | | |
58 | 58 | | |
59 | 59 | | |
| 60 | + | |
60 | 61 | | |
61 | 62 | | |
62 | 63 | | |
63 | 64 | | |
64 | 65 | | |
65 | | - | |
| 66 | + | |
66 | 67 | | |
67 | 68 | | |
68 | 69 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
133 | 133 | | |
134 | 134 | | |
135 | 135 | | |
| 136 | + | |
136 | 137 | | |
137 | 138 | | |
138 | 139 | | |
| |||
0 commit comments