Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-training

[📜 Paper] [⭐️Project Page] [🤗 Model] [📝 Chinese Post]

📰 News

2025.3: We release the SFT code on LLaVA-v1.5-mix665k dataset. We also release the 258M synthetic data used in S1.2 to boost future research.
2025.2: 🎉🎉 Mono-InternVL is accepted by CVPR 2025. Also check out our SynerGen-VL (CVPR 2025) that extends the monolithic structure to unified image generation and multimodal understanding, which will be open-sourced soon.
2024.11: Mono-InternVL is supported by lmdeploy.
2024.11: Mono-InternVL is supported by vllm.

⭐️ Introduction

We release Mono-InternVL, a monolithic multimodal large language model (MLLM) that integrates visual encoding and textual decoding into a single LLM. In Mono-InternVL, a set of visual experts is embedded into the pre-trained LLM via a mixture-of-experts (MoE) mechanism. By freezing the LLM, Mono-InternVL ensures that visual capabilities are optimized without compromising the pre-trained language knowledge. Based on this structure, an innovative Endogenous Visual Pretraining (EViP) is introduced to realize coarse-to-fine visual learning.

Mono-InternVL achieves superior performance compared to state-of-the-art MLLM Mini-InternVL-2B-1.5 and significantly outperforms other monolithic MLLMs, as shown in the radar chart above. Meanwhile, it achieves better deployment efficiency, with first token latency reduced by up to 67%.

For more details, please refer to our paper.

📊 Performance

Benchmark	Chameleon-7B	EVE-7B (HD)	Emu3	Mini-InternVL-2B-1-5	Mono-InternVL-2B
Type	Monolithic	Monolithic	Monolithic	Modular	Monolithic
#Activated Params	7B	7B	8B	2.2B	1.8B

MMVet	8.3	25.7	37.2	39.3	40.1
MMMU_val	25.4	32.6	31.6	34.6	33.7
MME_sum	170	1628	—	1902	1875
MMBench-EN_test	31.1	52.3	58.5	70.9	65.5
MathVista_testmini	22.3	34.2	—	41.1	45.7
SEED-Image	30.6	64.6	68.2	69.8	67.4
OCRBench	7	398	687	654	767
Hallusion-Bench	17.1	26.4	—	37.5	34.8
CCBench_dev	3.5	16.3	—	63.5	66.3
Avg_multimodal	16.1	38.9	—	54.4	55.2

TextVQA_val	4.8	56.8	64.7	70.5	72.6
SQA-I_test	47.2	64.9	89.2	84.9	93.6
GQA_test	—	62.6	60.3	61.6	59.5
DocVQA_test	1.5	53.0	76.3	85.0	80.0
AI2D_test	46.0	61.0	70.0	69.8	68.6
ChartQA_test	2.9	59.1	68.6	74.8	73.7
InfoVQA_test	5.0	25.0	43.8	55.4	43.0
Avg_VQA	17.9	54.6	67.6	71.7	70.1

Sources of the results include the original papers, our evaluation with VLMEvalKit, and OpenCompass.

Average scores are computed by normalizing each metric to a range between 0 and 100.

Please note that evaluating the same model using different testing toolkits can result in slight differences, which is normal. Updates to code versions and variations in environment and hardware can also cause minor discrepancies in results.

🚀 Inference

We provide an example code to run Mono-InternVL-2B inference using transformers.

Please use transformers==4.37.2 to ensure the model works normally.

Inference with Transformers (click to expand)

import numpy as np
import torch
import torchvision.transforms as T
from decord import VideoReader, cpu
from PIL import Image
from torchvision.transforms.functional import InterpolationMode
from transformers import AutoModel, AutoTokenizer

IMAGENET_MEAN = (0.485, 0.456, 0.406)
IMAGENET_STD = (0.229, 0.224, 0.225)

def build_transform(input_size):
    MEAN, STD = IMAGENET_MEAN, IMAGENET_STD
    transform = T.Compose([
        T.Lambda(lambda img: img.convert('RGB') if img.mode != 'RGB' else img),
        T.Resize((input_size, input_size), interpolation=InterpolationMode.BICUBIC),
        T.ToTensor(),
        T.Normalize(mean=MEAN, std=STD)
    ])
    return transform

def find_closest_aspect_ratio(aspect_ratio, target_ratios, width, height, image_size):
    best_ratio_diff = float('inf')
    best_ratio = (1, 1)
    area = width * height
    for ratio in target_ratios:
        target_aspect_ratio = ratio[0] / ratio[1]
        ratio_diff = abs(aspect_ratio - target_aspect_ratio)
        if ratio_diff < best_ratio_diff:
            best_ratio_diff = ratio_diff
            best_ratio = ratio
        elif ratio_diff == best_ratio_diff:
            if area > 0.5 * image_size * image_size * ratio[0] * ratio[1]:
                best_ratio = ratio
    return best_ratio

def dynamic_preprocess(image, min_num=1, max_num=12, image_size=448, use_thumbnail=False):
    orig_width, orig_height = image.size
    aspect_ratio = orig_width / orig_height

    # calculate the existing image aspect ratio
    target_ratios = set(
        (i, j) for n in range(min_num, max_num + 1) for i in range(1, n + 1) for j in range(1, n + 1) if
        i * j <= max_num and i * j >= min_num)
    target_ratios = sorted(target_ratios, key=lambda x: x[0] * x[1])

    # find the closest aspect ratio to the target
    target_aspect_ratio = find_closest_aspect_ratio(
        aspect_ratio, target_ratios, orig_width, orig_height, image_size)

    # calculate the target width and height
    target_width = image_size * target_aspect_ratio[0]
    target_height = image_size * target_aspect_ratio[1]
    blocks = target_aspect_ratio[0] * target_aspect_ratio[1]

    # resize the image
    resized_img = image.resize((target_width, target_height))
    processed_images = []
    for i in range(blocks):
        box = (
            (i % (target_width // image_size)) * image_size,
            (i // (target_width // image_size)) * image_size,
            ((i % (target_width // image_size)) + 1) * image_size,
            ((i // (target_width // image_size)) + 1) * image_size
        )
        # split the image
        split_img = resized_img.crop(box)
        processed_images.append(split_img)
    assert len(processed_images) == blocks
    if use_thumbnail and len(processed_images) != 1:
        thumbnail_img = image.resize((image_size, image_size))
        processed_images.append(thumbnail_img)
    return processed_images

def load_image(image_file, input_size=448, max_num=12):
    image = Image.open(image_file).convert('RGB')
    transform = build_transform(input_size=input_size)
    images = dynamic_preprocess(image, image_size=input_size, use_thumbnail=True, max_num=max_num)
    pixel_values = [transform(image) for image in images]
    pixel_values = torch.stack(pixel_values)
    return pixel_values


path = 'OpenGVLab/Mono-InternVL-2B'
model = AutoModel.from_pretrained(
    path,
    torch_dtype=torch.bfloat16,
    low_cpu_mem_usage=True,
    trust_remote_code=True).eval().cuda()
tokenizer = AutoTokenizer.from_pretrained(path, trust_remote_code=True, use_fast=False)

# set the max number of tiles in `max_num`
pixel_values = load_image('./examples/image1.jpg', max_num=12).to(torch.bfloat16).cuda()
generation_config = dict(max_new_tokens=1024, do_sample=True)

# pure-text conversation
question = 'Hello, who are you?'
response, history = model.chat(tokenizer, None, question, generation_config, history=None, return_history=True)
print(f'User: {question}\nAssistant: {response}')

question = 'Can you tell me a story?'
response, history = model.chat(tokenizer, None, question, generation_config, history=history, return_history=True)
print(f'User: {question}\nAssistant: {response}')

# single-image single-round conversation
question = '<image>\nPlease describe the image shortly.'
response = model.chat(tokenizer, pixel_values, question, generation_config)
print(f'User: {question}\nAssistant: {response}')

# single-image multi-round conversation
question = '<image>\nPlease describe the image in detail.'
response, history = model.chat(tokenizer, pixel_values, question, generation_config, history=None, return_history=True)
print(f'User: {question}\nAssistant: {response}')

question = 'Please write a poem according to the image.'
response, history = model.chat(tokenizer, pixel_values, question, generation_config, history=history, return_history=True)
print(f'User: {question}\nAssistant: {response}')

Inference with LMDeploy

Please install lmdeploy>=0.6.3 for Mono-InternVL support.

from lmdeploy import pipeline
from lmdeploy.vl import load_image

image = load_image('./examples/image1.jpg')
pipe = pipeline('OpenGVLab/Mono-InternVL-2B')
response = pipe(('Please describe the image shortly.', image))
print(response.text)

🔥 Supervised Finetuning

Currently we provide the supervised finetuning (S2 instruction tuning) code on the LLaVA-v1.5-mix665k dataset. For details on the dataset, please refer to LLaVA-v1.5.

Installation

Clone this repository:

git clone https://github.com/OpenGVLab/Mono-InternVL.git

Create a conda virtual environment and activate it:

conda create -n monointernvl python=3.9 -y
conda activate monointernvl

Install dependencies using requirements.txt:
```
pip install -r requirements.txt
```

Additional: Install flash-attn==2.5.6:

pip install flash-attn==2.5.6 --no-build-isolation

Alternatively you can compile from source:

git clone https://github.com/Dao-AILab/flash-attention.git
cd flash-attention
git checkout v2.5.6
python setup.py install

Dataset Preparation

LLaVA-v1.5-mix665k Dataset

Download the instruction tuning data:

mkdir playground
wget https://huggingface.co/datasets/liuhaotian/LLaVA-Instruct-150K/resolve/main/llava_v1_5_mix665k.json -P playground/

Download image datasets:

COCO: train2017
GQA: images
OCR-VQA: download script
TextVQA: train_val_images
VisualGenome: part1, part2

Organize data as follows:

playground/
├── data/
│   ├── coco/train2017/
│   ├── gqa/images/
│   ├── ocr_vqa/images/
│   ├── textvqa/train_images/
│   └── vg/
│       ├── VG_100K/
│       └── VG_100K_2/
└── llava_v1_5_mix665k.json

Custom Dataset

For custom dataset, format your data in to a JSONL file, where each entry is a dictionary organized in the following format (similar to llava_v1_5_mix665k.json):

{
    "id": "000000120375",
    "image": "coco/train2017/000000120375.jpg",
    "conversations": [
        {
            "from": "human",
            "value": "<image>\nWhat type of vehicle is driving down the street in the image?"
        },
        {
            "from": "gpt",
            "value": "A red sports utility vehicle (SUV) is driving down the street in the image."
        },
        {
            "from": "human",
            "value": "Is the street crowded with people?"
        },
        {
            "from": "gpt",
            "value": "Yes, the street is filled with a considerable number of people, which indicates that the area is busy."
        }
        # (more turns ...)
    ]
}

Then modify the metadata file shell/data_llava_finetune.json:

{
  "name of your dataset": {
    "root": "playground/data/", # combination of "root" and "image" in the JSONL gives the complete image path
    "annotation": "path to your JSONL",
    "data_augment": false,
    "repeat_time": 1,
    "length": 12345 # change to the actual number of samples in your dataset
  }
}

Model Preparation

We provide pretrained models of different stages (S1.1 concept learning, S1.2 semantic learning, S1.3 alignment learning). Choose from the following models and download the weights to workdirs/ folder.

model name	download	size
Mono-InternVL-2B-S1-1	🤗 HF link	6.2 GB
Mono-InternVL-2B-S1-2	🤗 HF link	6.2 GB
Mono-InternVL-2B-S1-3	🤗 HF link	6.2 GB

mkdir workdirs
cd workdirs/
# pip install -U huggingface_hub
huggingface-cli download --resume-download --local-dir-use-symlinks False OpenGVLab/Mono-InternVL-2B-S1-1 --local-dir Mono-InternVL-2B-S1-1

The directory structure is:

workdirs/
├── Mono-InternVL-2B-S1-1/
├── Mono-InternVL-2B-S1-2/
└── Mono-InternVL-2B-S1-3/

Training

Finetuning takes around 12 hours on 8x A100 (80G) GPUs.

Single Node Multi-GPU

MODEL="./workdirs/Mono-InternVL-2B-S1-3" OUTPUT_DIR="./workdirs/mono_internvl_llava_sft" sh shell/mono_internvl_finetune_llava_torchrun.sh

Slurm Cluster

PARTITION="your partition" MODEL="./workdirs/Mono-InternVL-2B-S1-3" OUTPUT_DIR="./workdirs/mono_internvl_llava_sft" sh shell/mono_internvl_finetune_llava_slurm.sh

🎫 License

This project is released under the MIT License.

🖊️ Citation

If you find this work helpful in your research, please consider giving this repo a star ⭐ and citing our paper:

@article{luo2024mono,
  title={Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-training},
  author={Luo, Gen and Yang, Xue and Dou, Wenhan and Wang, Zhaokai and Liu, Jiawen and Dai, Jifeng and Qiao, Yu and Zhu, Xizhou},
  journal={arXiv preprint arXiv:2410.08202},
  year={2024}
}

Name		Name	Last commit message	Last commit date
Latest commit History 9 Commits
examples		examples
images		images
internvl		internvl
shell		shell
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
requirements.txt		requirements.txt

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-training

📰 News

⭐️ Introduction

📊 Performance

🚀 Inference

🔥 Supervised Finetuning

LLaVA-v1.5-mix665k Dataset

Custom Dataset

Single Node Multi-GPU

Slurm Cluster

🎫 License

🖊️ Citation

About

Languages

License

OpenGVLab/Mono-InternVL

Folders and files

Latest commit

History

Repository files navigation

Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-training

📰 News

⭐️ Introduction

📊 Performance

🚀 Inference

🔥 Supervised Finetuning

LLaVA-v1.5-mix665k Dataset

Custom Dataset

Single Node Multi-GPU

Slurm Cluster

🎫 License

🖊️ Citation

About

Resources

License

Stars

Watchers

Forks

Languages