Skip to content

[Feature] Add multi-turn chat for Gemma 3, Mantis, and Pixtral - #1637

Open
reacher-z wants to merge 1 commit into
open-compass:mainfrom
reacher-z:feat/chat-inner-gemma3-mantis-pixtral
Open

[Feature] Add multi-turn chat for Gemma 3, Mantis, and Pixtral#1637
reacher-z wants to merge 1 commit into
open-compass:mainfrom
reacher-z:feat/chat-inner-gemma3-mantis-pixtral

Conversation

@reacher-z

Copy link
Copy Markdown

Summary

  • add chat_inner for Gemma 3 while preserving complete user/assistant history and images in both Transformers and vLLM backends
  • add chat_inner for Mantis using its native conversation template, with a separate Idefics chat-template path
  • add chat_inner for Pixtral using native Mistral UserMessage, AssistantMessage, and SystemMessage requests
  • keep each model's existing single-turn generate_inner path working through shared generation helpers
  • reject images in assistant turns instead of silently constructing unsupported histories

This addresses another group of three models requested in #323. Their upstream documentation confirms the required capabilities:

Validation

  • python -m unittest -v tests/test_vlm_chat_inner_gemma_mantis_pixtral.py (8 tests passed)
  • python -m py_compile vlmeval/vlm/gemma.py vlmeval/vlm/mantis.py vlmeval/vlm/pixtral.py tests/test_vlm_chat_inner_gemma_mantis_pixtral.py
  • pre-commit run --files vlmeval/vlm/gemma.py vlmeval/vlm/mantis.py vlmeval/vlm/pixtral.py tests/test_vlm_chat_inner_gemma_mantis_pixtral.py

The tests exercise Gemma's Transformers and vLLM paths, Mantis's native and Idefics paths, Pixtral's Mistral request construction, image ordering, role preservation, and single-turn regression behavior without downloading model weights.

AI assistance

OpenAI Codex assisted with implementation and testing. I reviewed the model capability sources, code paths, and test outputs before submission.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant