Abnormal GPU Utilization When Evaluating Qwen 2.5 VL 72B

[LiamLian0727]

Low GPU Utilization and Errors During Qwen 2.5 VL 72B Evaluation

First, I'd like to commend the lmms-eval team for developing such a robust evaluation framework - it's been incredibly valuable for multimodal research.

Problem Description

When evaluating Qwen 2.5 VL 72B on 8x H100 80G GPUs using the following command:

accelerate launch --num_processes=1 --main_process_port=12346 -m lmms_eval \
  --model qwen2_5_vl \
  --model_args="pretrained=/mnt/mypath/models/Qwen2.5-VL-72B-Instruct,max_pixels=12845056,attn_implementation=flash_attention_2,interleave_visuals=False,device_map=auto" \
  --tasks super_clevr \
  --output_path /mnt/mypath/qwen72b \
  --batch_size 1 \
  --verbosity DEBUG

My GPU utilization only reaches a maximum of 10%.

Attempted Solutions and Errors

1 Increasing num_processes to 8: Results in OOM (Out of Memory) errors despite using H100 GPUs 80G

This is most likely because the code in this ignores the device_map passed in by the user when num_processes is greater than 1 and uses f “cuda:{accelerator.local_process_index}”, which results in the model not being sharded correctly

accelerator = Accelerator()
if accelerator.num_processes > 1:
    self._device = torch.device(f"cuda:{accelerator.local_process_index}")
    self.device_map = f"cuda:{accelerator.local_process_index}"
else:
    self._device = torch.device(device)
    self.device_map = device_map if device_map else device

2 Increasing batch_size > 1: Throws ValueError:

ValueError: You are attempting to perform batched generation with padding_side='right' this may lead to unexpected behaviour for Flash Attention version of Qwen2_5_VL. 
Make sure to call `tokenizer.padding_side = 'left'` before tokenizing the input

After that, I tried to change code :self._tokenizer = AutoTokenizer.from_pretrained(pretrained) in qwen2_5_vl.py to self._tokenizer = AutoTokenizer.from_pretrained(pretrained, padding_side = 'left') but it didn't work.

Request for Assistance

How should I properly configure either:

1. Multi-GPU evaluation to avoid OOM, or
2. Batched evaluation with correct padding configuration?
3. Any other recommended parameters for efficient evaluation of this large vision-language model?

We'd be grateful for any insights from the team or community members who have successfully evaluated similar large multimodal models.

[kcz358]

I would recommend you to use vllm or sglang to evaluate the 72B models. You can either serve it or serve it as an api server and then use openai to evaluate it

[LiamLian0727]

I tried using vllm, but depending on performance with the use of accelerate launch --num_processes=1 --main_process_port=12346 -m lmms_eval --model qwen2_5_vl is different (4-5% lower accuracy for vllm), this is my vllm script, is there something wrong with my setup?

set -x

export VLLM_WORKER_MULTIPROC_METHOD=spawn
export NCCL_BLOCKING_WAIT=1
export NCCL_TIMEOUT=18000000
export NCCL_DEBUG=DEBUG
export TORCH_CUDA_ARCH_LIST="9.0"

python3 -m lmms_eval \
    --model vllm \
    --model_args model_version=${model_path},tensor_parallel_size=4,gpu_memory_utilization=0.9 \
    --tasks my_task \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix vllm \
    --output_path ./logs \
    --verbosity DEBUG

Because I noticed that a parameter like max_pixels=12845056 doesn't exist in the vllm model code implemented by lmms-eval. I'd appreciate it if you could guide me.

[kcz358]

I am a bit unsure about whether it is possible to override image processor config with vllm commands directly. The most easy way is that you modify your local image processor json and then start server using that path

[LiamLian0727]

Hi @kcz358
I think I know what the problem is, it's because of a bug in the vllm backend in the lmms-eval code

Specifically, the encode_video function in the code returns a list in the call

def encode_video(self, video_path):
    vr = VideoReader(video_path, ctx=cpu(0))
    total_frame_num = len(vr)
    uniform_sampled_frames = np.linspace(0, total_frame_num - 1, self.max_frame_num, dtype=int)

    # Ensure the last frame is included
    if total_frame_num - 1 not in uniform_sampled_frames:
        uniform_sampled_frames = np.append(uniform_sampled_frames, total_frame_num - 1)

    frame_idx = uniform_sampled_frames.tolist()
    frames = vr.get_batch(frame_idx).asnumpy()

    base64_frames = []
    for frame in frames:
        img = Image.fromarray(frame)
        output_buffer = BytesIO()
        img.save(output_buffer, format="PNG")
        byte_data = output_buffer.getvalue()
        base64_str = base64.b64encode(byte_data).decode("utf-8")
        base64_frames.append(base64_str)
     return base64_frames

This list will be appended to all_tasks and eventually aggregated to imgs, using the following code, where the img in the f-string expression may be a list, not a str：

.......
    imgs = []  # multiple images or frames for video
    all_tasks = []
    with ThreadPoolExecutor(max_workers=self.threads) as executor:
        for visual in visuals:
            if isinstance(visual, str) and (".mp4" in visual or ".avi" in visual or ".mov" in visual or ".flv" in visual or ".wmv" in visual):
                all_tasks.append(executor.submit(self.encode_video, visual))
.......
        for task in all_tasks:
            imgs.append(task.result()) # base64_frames list be append in imgs

messages = [{"role": "user", "content": []}]
messages[0]["content"].append({"type": "text", "text": contexts})
for img in imgs:
    messages[0]["content"].append({"type": "image_url", "image_url": {"url": f"data:image/png;base64,{img}"}})

As a result, the model receives invalid URLs such as data:image/png;base64,[‘AAA’,'BBB',...] and will not be able to get any video information.

So when I use the vllm backend instead of the qwen25vl, the model never gets it right when it encounters the video, which results in a performance drop.

I'd be happy to submit a pr to fix it if you're sure it's a bug!

[kcz358]

Yeah I think you are correct. I saw you are PR about this. Do you mind also give me a reproduce script so I can test the fix is correct? After testing and reviewing, if the performance drop is fixed, I will merge the PR. Thanks!

Hi, I want to know what's the version of your Transformers during evaluating Qwen?

[LiamLian0727]

Ok, my dataset is not open source yet, but it can be reviewed on other video datasets supported on lmms-eval, such as vsi-bench

set -x

export VLLM_WORKER_MULTIPROC_METHOD=spawn
export NCCL_BLOCKING_WAIT=1
export NCCL_TIMEOUT=18000000
export NCCL_DEBUG=DEBUG

model_path=/mnt/your_models_path/Qwen2.5-VL-3B-Instruct

python3 -m lmms_eval \
    --model vllm \
    --model_args model_version=${model_path},tensor_parallel_size=1,gpu_memory_utilization=0.9 \
    --tasks vsibench \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix vllm \
    --output_path ./logs \
    --verbosity DEBUG

You can try this script

[LiamLian0727]

Hi @tianjiedai， this is my version of transformers and qwen-vl-utils

[Fu-Fu-Fu-Fu]

Hello, I'm also encountering this issue. Using VLLM to test qwen2.5vl is indeed fast, but the accuracy is 2-5% lower than using accelerate, and even more than 5% lower on videomme. However, using accelerate results in extremely low GPU utilization (less than 10%). Has this bug not been fixed? Are there any solutions? I look forward to your reply, thank you! @kcz358

[Luodian]

no activity since july - closing this out. if you're still seeing the list encoding bug with vLLM, reopen or file a fresh one